A central repository for gridded data in the MeteoSwiss Data Warehouse, Zürich M2: Data Rescue management, quality and homogenization September 16th, 2010 Data Coordination, MeteoSwiss 1
Agenda Short introduction into Data Warehouse at MeteoSwiss Objectives of the MeteoSwiss Data Warehouse Integration of new data sources Conceptual architecture (CIF) Evaluation of solution Description of the chosen solution Achievements Conclusions 2
Objectives of the MeteoSwiss Data Warehouse Implementation of a central, integrated data platform for the collection, storage and processing of meteorological data and its meta data. Storage of meta data in one central meta data repository Subdivision in a processing and an application area Creating a basis for a customized access to data respecting official standards for internal and external clients 3
Integration of new data sources 2000 2001 2002 Start Project MeteoSwiss Data Warehouse Surface data (Rain gauge measurements, AWS data, observations) 2003 Phenology & Pollen 2004 Aviation messages 2005 Lightning data 2006 International surface data (SYNOP) 2007 National sounding data 2008 Homogenized data 2009 National & International Upper Air data 2010 Gridded datasets 4
Conceptual Architecture (CIF) Data sources Staging area Data storage Metadata Management Tools Kontextd Context Data aten Data analysis Climate Information Service Model Validation Research Climatology Data for the Public Internet Durable data archive Weather Forecast Data delivery and products (on-/offline) Data analysis & Interfaces (API, DB Tools, Web access, GIS, R ) Database for point data analysis T Database for point data processing (incl. raw data repository) QC Geodatabase (raster data,...) L Calc, Check, Data Transformation & Integration Observations Partner Observ. systems Autom. Observ. systems Aviation data Upper Air data Lightning data NWP VERA Radar Clima Analysis Tools CM SAF 5
Evaluation of solution Originally six possible solutions 2 were thoroughly evaluated (THREDDS, database solution) 55 criterias based on users requirements were applied in the categories : Import of different formats such as NetCDF, AscII, GIS Export of different formats (see Import) Possibilities for visualization Technical requirements Decision for database oriented solution 6
Description of the chosen solution R Data streaming service NetCDF NetCDF Climate analysis Radar information GIF ASCII Spatial data transformation Geodata Mgmt Geodatabase GIF ASCII Products combining different data sources Diverse Analysis etc. GIS,... All raster data in data base format GIS,... 7
Achievements April 2009: Proof of concept to check if the database solution meets the critical requirements June 2009 now: Test phase delivering precipitation grids in NetCDF format to 5 to 10 test clients Mid 2009 Mid 2010: Further developments ( Raster datastreaming services for R; GIF Import/Export; ASCII Import; Spatial Aggregation) September 2010: Installation of Release 7 in deployment Environment 4. Q 2010: Get ready for operational mode 8
Conclusions Proof of concept was a good indicator to see if the solution can fulfill the most important user requests but it didn t prevent us from unexpected problems Testphase was a big help to gain knowledge about possible problems during operational service to learn more about users needs The development of a first version helped to get a more detailed specification of the users needs. Next versions can profit of experiences made during test phases. 9
Questions? Thank you very much for your attention 10