You are here: Home » Archive » Phase 1 » Project » Description of work » JRA4/WP10 - ESM Data: Enhancement of Management Protocols and Software Infrastructures

JRA4/WP10 - ESM Data: Enhancement of Management Protocols and Software Infrastructures



JRA4 will deliver software infrastructure and a management framework to:

  • create a distributed archive capability and
  • enhance the services beyond those provided by existing archives.

The distributed archive will benefit from the resources of diverse institutions but will provide a service which is transparent to the user.

Management protocols and software developed in JRA4 will facilitate the provision of transparent and efficient data services from distributed Earth System Archives using independently funded and maintained hardware. These services will complement and work with the Common Information Model and single sign-on packages delivered by METAFOR and build them into a comprehensive resource management system, with the necessary workflows at all levels of operation, from the institutional level, ensuring that all resources which are available can be exploited without compromising the integrity of the archive, down to the `bit-level' ensuring that data is secure and data movements are managed efficiently. The package will also add higher level services to the basic geospatial processing provided by METAFOR. METAFOR will provide a structure for the archive catalogue and software to allow users to browse through that catalogue. Once the user has identified the data they need, they will click on a link. What happens after that click is the business of this WP.

The utilisation of distributed resources avoids the massive expense of a centralised archive and utilises the distributed national expertise in running data archives. The work package will deliver a suite of management protocols and software tested on a prototype distributed archive. A smooth transition between the development work here and the service provision in SA2 the software will be packaged into alpha, beta and production versions.

The objectives of this package will be shaped by the challenge of providing a distributed archive for the IPCC 5th Assessment Report (AR5) climate projections, but the software and protocols will be designed to be flexible and extensible for use with other European and international Earth System Model data archives. The functionality of the system will also be tested on the CCMVal project. The AR5 archive development will be led by PCMDI (USA), but in a departure from previous IPCC reports, this will be a collaborative effort with European Data Centres providing an essential contribution to the archive. JRA4 will coordinate European input into the archive development of the protocols and standards. JRA4 will ensure that European Data Centres are in a position to meet the technical requirements and will provide additional support for modelling groups which intend to exceed the AR5 requirements by, for instance, providing sufficient data to drive regional models for downscaling experiments. The GO-ESSP framework will be used to ensure that developments within this package exploit synergies with developments outside Europe.

See the full description in the DoW



The partners of JRA4 have been working intensively to deliver infrastructure capable of supporting the CMIP5 (Climate Model Inter-comparison Project, phase 5). The software infrastructure for the CMIP5 is being developed jointly by the Earth System Grid Federation. Key contributions from the JRA4 activity are the security infrastructure which will allow the distributed archive to appear as a single entity from the user perspective and quality control procedures and software which will facilitate the timely distribution of reliable data.

Task 1: Archive Management and Core Services:

The global consortium which will deliver the CMIP5 distributed archive has been formalised as the “Earth System Grid Federation” (ESGF), with IS-ENES partners participating.

Task 2: Common Information Model (CIM) and Standard Names:

A report on  dependencies of distributed archive services on the Common Information Model (CIM) being developed in the FP7 project METAFOR has been completed (D10.1). 

Task 3: Development of the Operational Services Package:

The operational services are embedded in the Earth System Grid data nodes (details are given in the M10.3 report). Effort has been devoted to security infrastructure and the data reference syntax. A grid service, using the Grid Relational Catalog Project software (GrelC), has been deployed at the CMCC site and it is available for testing activity. Moreover an XML-based back-end library has been designed to address the IS-ENES needs for flexible access to the detailed documentation of climate models being collected within the FP7 METAFOR project (M10.4). A GRIDSs testbed has been deployed and used to evaluate benefits of parallelization for evaluation of multi-model ensemble means.

Task 4: Development of the User Services Package:

The security architecture supports the use of wget to retrieve data with a certificate. A certificate service has been developed to allow users easy access to certificate from a script environment.

Through discussions with UNIDATA, the security model is supported by the latest NetCDF library and will enable secure OpenDAP (Open-source Project for a Network Data Access Protocol) access.




D10.1 Report on CIM dependencies


D10.2 Data policy and software review




© Copyright ENES Portal 2011



IS-ENES3 has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 824084