Miyoshi CREST

File I/O for Big data

"Innovating Big Data Assimilation technology for revolutionizing very-short-range severe weather prediction"

An innovative 30-second super-rapid update numerical weather prediction system for 30-minute/1-hour severe weather forecasting will be developed, aiding disaster prevention and mitigation, as well as bringing a scientific breakthrough in meteorology.

File I/O patterns in Ensemble simulation and data assimilation

  • Every 30 second
    • Each weather simulation generates 17 GB data. 1.7 TB in total for 100 cases
    • Data assimilation code reads all results (17GB) and observed data (-- 1GB), and outputs assimilated data for 100 cases. That is, 1.7 TB in total for 100cases

Approach: I/O Arbitrator

  • Keeping the file I/O API's (netCDF), data is transferred from the ensemble simulation jobs to the data assimilation job without storing/loading data to/from the file system
    • Application programs are not modified

Note: the current prototype system extends netCDF