Collaborative Climate Community Data and Processing Grid (C3Grid) T2.1: Grid Information Service Architecture and Specication

Transkript

1 Collaborative Climate Community Data and Processing Grid (C3Grid) T2.1: Grid Information Service Architecture and Specication Work Package: AP 2: Grid Information Service Author(s): Tobias Langhammer, Florian Schintke Version Idenitier: C3Grid-T Draft Publication Date: March 2006 Work Package Coordinator: Zuse Institute Berlin (ZIB) Partners: ZIB, GKSS, CEI, Uni Kln, FUB Contact: Tobias Langhammer, Florian Schintke

2 Contents Contents 2 1. Introduction Notation y Common Terms y The C3Grid Project y 7 3. Global C3Grid Architecture y 8 4. The C3Grid Grid Information Service Common Requirements Requirements on the Resource Information Service Requirements on the Data Information Service Grid Tools and Solutions in this Area Globus Toolkit 4.0 Monitoring & Discovery Services (MDS) OGSA-DAI Storage Resource Broker (SRB) Open Challenges Architecture of the Resource Information Service Design Use Cases Monitoring Scheduled Jobs Serving the Workow Scheduler Recovery of the Data Information Service or the Grid Data Management System Interfaces of the Resource Information Service Architecture of the Data Information Service Design Use Cases User's Search for Base Data and Metadata Caching of Discovery Metadata Registration of New Data y Interfaces of the Data Information Service Interface to the Portal Datatypes Operations of the DataInfoService Interface of the Primary Metadata Provider Harvesting Discovery Metadata Operations of the UseMetaAccess / 51 C3Grid-T Draft

3 9. Authorization y Conclusion 33 A. Interface Spezication Syntax y 34 B. Questionnaire for Users and Data Providers 35 B.1. Important Aspects B.2. Miscellaneous B.3. Aspects of Data Management B.4. Aspects of Metadata Management C. Quotes from the Development Discussion y 44 C C.2. C3Grid-Wiki C.2.1. WF im Portal (Stand ) C.2.2. Metadaten im WF (Stand ) C.2.3. AG-Metadaten (Auszug, Stand ) C.2.4. AG-Metadaten-Meeting1 (Stand ) C.2.5. Metadaten-Meeting-Marum (Stand ) C.3. Mailing Lists Bibliography 51 C3Grid-T Draft 3 / 51

4 1. Introduction This document describes the results of the discussion on the architecture, interfaces, and functionality of the C3Grid Information Service. The document is structured as follows Notation y The y-sign The two documents [LS06a, LS06b], which describe the C3Grid Grid Information Service and the C3Grid Grid Data Management System are related to each other. To make each document self contained, some sections are shared between both documents and printed in the exactly same way in both documents. Such chapters, sections, gures, etc. are marked by the y -sign. Interface Denitions For the specication of interfaces, this document uses a notation, which is described in the following. Datatypes used are common basic datatypes (like int, double, string) and constructed types. Constructed types can be lists, pairs, records, enumerations or variants. They can be bound to new names by a type denition. For example, type foo = (int,double) list denes a new type foo which is a list of (int,double) pairs. For dening a record with an int and a oat entry we write type foobar = f foo : int, bar : double g For dening enumerations we write type color = ( Red j Green j Blue j Yellow ) Variants are a generalization of enumerations, where each handle can be assigned a type type color = Name string j RGB (int, int, int) 1 Service Interfaces are decribed in sections called 'Operations of the FooService'. Such section lists the respective operations of the interface. Callers can use the functionality of the interfaces by invoking the listed operations remotely (using Web Services). Remote operations are given by their signature (consisting of the name, parameter list and return type), a short functional description and a detailed list describing the parameters, the return value and exceptions. The following example demonstrates this notation. 1 Note that a variant data type can be mapped to a C union. 4 / 51 C3Grid-T Draft

5 foo (in huey : double, inout dewey : bool, out louie : string ) : int Example operation with some special functionality. Parameters: in huey Example input argument which is a oating point value inout dewey Example boolean argument which is modied by the operation. out louie Example string argument which is set by the operation. Returns: Example integer return value. For a detailed BNF-style description of the interface denition syntax see Appendix A Common Terms y The terms must, must not, should, should not and may are used in accordance to RFC 2119 [Bra97]. This specication also uses the following terms. base data: data which is not metadata, i.e. actual climate data les. grid sites: sites within the C3Grid providing data capacity (workspaces) for processing and storing les temporarily. metadata: Data that describes other data. In the C3Grid we distinguish metadata by their origin and purpose. primary metadata: metadata generated and stored at primary data sites. Though parts of this metadata may be cached by the Data Information System, the original copy always remains out of reach for C3Grid middleware. grid metadata: metadata generated in the grid workspaces and managed by the Data Information Service. This metadata also describes data only stored in the grid workspaces. discovery metadata: data necessary to nd climate data in the C3Grid. use metadata: metadata describing how the processing tools can access a data object. le metadata: metadata used by the Grid Data Management System in order to track the location of replicated les. Note, that the set of primary metadata objects and the set of grid metadata objects are mutually exclusive. Discovery and use metadata are either primary or grid metadata, le metadata are always grid metadata. primary data: Read-only data at a primary site. The creation of primary data is outside the scope of the C3Grid. primary site: Organisation providing a large data repository which allows read-only access (DBMS, archive,... ) process: In terms of workows, an atomic processing unit. In terms of operating system, a running program. workspace: Local (to a grid site) disk space, where data is made available for processing. C3Grid-T Draft 5 / 51

6 DOI: Digital Object Identier [DOI]. A system for identifying objects on the Internet. The DOI of a data object is unique and does not assign it to a certain location like a URI. DOIs provide a persistent way of accessing data objects by a mapping service. In the C3Grid DOIs are used for identifying single datasets. The data service provides DOIs to identify search results. These DOIs can then be used for requesting primary data from the data repositories. 6 / 51 C3Grid-T Draft

7 2. The C3Grid Project y The aim of the Collaborative Climate Community Data and Processing Grid is to develop a productive, grid-based environment for the German earth system science, that supports eective, distributed data processing and the interchange of high volume datasets from models and observations between participating institutions. The collaborative system as a whole will provide unied and transparent access to the geographically distributed archives of dierent institutions and simplify the processing of these data for the individual researcher. Specic challenges are to provide a collaborative platform for accessing and processing huge amounts of data, as well as the eective management of community specic meta information. Earth system research studies the system of earth and the processes within, and wants to develop models to predict its climate changes. The C3Grid will provide the infrastructure to share data and processing power across institutional boundaries. C3Grid-T Draft 7 / 51

8 3. Global C3Grid Architecture y The C3Grid provides an infrastructure which is designed to the specic needs of the community of climate researchers. It provides a means of collaboration, integrating the heterogeneous data sources of the participating organizations and oering easy access by a common users interface. The main application of the C3Grid is the analysis of huge amounts of data. To increase the eciency the system provides automatic reduction or pre-processing operations and exploits data locality, replication and caching. user Userinterfaces API GUI Nutzer Grid data management Grid scheduling Distributed Grid infrastructure Grid information service data-transfer service Local interfaces to institutional resources userrmodules pre-processing data jobs Existing resources of the institutes metadata data archive access to archive distributed distributed data archives data archives distributed distributed computeresources computeresources Figure 3.1.: Architecture of the C3Grid. The architecture overview of Figure 3.1 shows the major components of the C3Grid in their operational context. Existing data repositories are integrated by dedicated interfaces for providing primary base data and respective metadata to the middleware. Pre-processing capabilities allow a rst on-site data reduction. A distributed grid middleware consists of an information service, data management and transfer services, and a scheduler. Actual jobs are executed on data which was extracted (staged) from primary repositories and made available on a local disk share. Each processing not only produces new base data, but also generates new metadata from metadata describing the original input. The information services provides a comprehensive search for resources, primary data available in local repositories, and data created at local grid shares. The Portal uses these services to assist users in their data analysis workows. 8 / 51 C3Grid-T Draft

9 4. The C3Grid Grid Information Service The Grid Information Service is split into two seperate components. The Resource Information Service, which is responsible for information on compute ressources and the Data Information Service, which is responsible for information on data Common Requirements The following requirements for the Grid Information System were specied in the C3Grid project proposal [DG]. They are common for both, the Resource Information Service and the Data Information Service. (R1) The system must provide an information service for compute resources and data. (R2) The system must provide a virtual integration of the metadata of several existing metadata and archive systems. (R3) The system should be a decentralized service, which is loosely coupled with the existing data repositories (during the second phase of the project). (R4) The system must support ecient search and insertion. The user should receive a useful or successful response within seconds. (R5) The system must support users in automated creation and management of metadata Requirements on the Resource Information Service The following requirements are specic to the service providing resource information. They are based on the architecture described in the C3Grid project proposal [DG]. (R6) The system must provide host and network information to the Portal and Workow Scheduler. (R7) The system must provide queuing information from the Workow Scheduler to the Portal. (R8) The Resource Information Service should manage the following specic information. Host information { Static host information (host name, OS name and version) { CPU information (type, load) { Memory statistics (total, free) { File system (size, occupation) { Compute capabilities (presence of software tools,...) Queuing information { Static information (available queues) { Job information (scheduled jobs, state, resources occupied, estimated running time) Network information { Current transfers { Network linkage C3Grid-T Draft 9 / 51

10 (R9) This information must be kept up-to-date by obtaining it from the resources periodically or when it is created Requirements on the Data Information Service For the management of metadata about base data, the project proposal denes additional special requirements. (R10) The system must be aware of the specic base data of the climate and earth system science community. (R11) The search must be provided for geographic section dened by latitude, longitude and altitude. (R12) The search may also provide temporal cuts. (R13) The access to data must be restrictable to a subset of authorized users. (R14) The system must support a common subset of the ISO standard for the description of geographic information, which is used throughout the C3Grid. Special requirements derive from the answers by the C3Grid users and data providers given in the questionnaire of Appendix B. (R15) The system must access existing metadata at le and databases (e.g. CERA [LTTR98]) repositories. (R16) The system may not care about the deletion of metadata from primary data providers, as this is hardly the case. (R17) Due to the size and period of new metadata published by the primary providers, the system may have an interval of up to several hours between polls. (R18) The system must handle a total of several gigabytes of metadata. The following list of requirements is derived from the discussion with the users. (R19) Depending on its purpose, metadata is divided conceptually into the following types. 1 details need to be discussed with metadata group Discovery metadata: contains scientic and geographic information needed to nd base data, like project and experiment information, citations, climate parameters, geographical space or time ranges. 2 Use metadata: contains information specic to the production environment, describing how data can be made available in the grid and for the processing tools. For example, it contains the location of data and how the information is organized in les. The Grid Data Management System may treat both types of metadata independently. (R20) As attributes for the search, the system must at least provide the subset metadata which is required to satisfy the input requirements of the predened workows. 3 (R21) In order to reduce the amount of data cached in the Data Information Service it must not cache the complete set of metadata. Therefore, use metadata is stored by the data providers and oered to the grid service via an interface. 1 See also the quote from the C3Grid wiki in Section C See also the quote from the C3Grid wiki in Section C See also the use case discussed in the in Section C / 51 C3Grid-T Draft

11 (R22) Some primary data providers, especially databases, may generate use metadata dynamically for a given pre-processing specication. Therefore, the Data Information Service must provide to the Portal the optional passing of special pre-processing arguments. Note that if the data repository provides the generation of dynamic metadata, it must also provide it independent from the actual generation of the base data. Further discussion among the workow and metadata developers yielded the following requirements. (R23) The system must handle metadata les and integrate their information in its catalog. (R24) The system must pass pre-processing specications to the primary database repositories. Pre-processing results must be registered as new data objects Grid Tools and Solutions in this Area Globus Toolkit 4.0 Monitoring & Discovery Services (MDS) The MDS4 [SDM + 05] information services of Globus Toolkit 4 oer a suite of services for the monitoring and discovery of grid resources. MDS4, also called WS MDS, is based on web services. For legacy reasons, it still includes the pre-web-service MDS2 components. Features MDS is mostly used in the context of resource selection. It aids the user or agent to identify the hosts on which to run a specic application. The MDS architecture basically consists of three layers. Information providers. These are resources like hosts, services or tools. MDS4 services. Currently, MDS4 provides an Index Service and a Trigger Service. WebMDS. This is a web-based interface for the Index Service. For collecting data from the information providers, the MDS4 services are built on a more general framework, the Aggregator Framework. An MDS4 Service acts as aggregator service to collect data from the providers which are accessed through aggregator sources. Information may be collected from the providers in dierent ways: by accessing them through WSRF [CFF + 04] services or by running an external, administrator-supplied executable. The WSRF services provide either a periodical polling of information or a subscription/notication mechanism, allowing information to be updated when information changes. GT4 also provides implemented aggregator sources, like the Grid Resource Allocation and Management (WS GRAM). The MDS Index Service can be compared to UDDI but is more exible in some respect. In addition to the dierent schemes for collecting data, it can act as information provider as well, allowing a hierarchical structuring of Index Services. Also it provides a self-cleaning mechanism of old information because each entry has a specied lifetime. WebMDS is a web interface to access the MDS Index Service. It uses XSL transforms to generate the respective web representation from the MDS index information. Applicability to the Grid Information Service MDS4 Index suites well to the requirements of the resource information service in the C3Grid. WebMDS provides a high-level, transparent access to the Portal. C3Grid-T Draft 11 / 51

12 The Workow Scheduler scheduler can obtain host information via WSRF or may use the WS GRAM interface. MDS4 is fault tolerant as it can always re-collect information from the data sources. MDS4 can collect information from any type of resource, especially host, network and queuing information. The aggregator framework provides a subscribe/notication based collection of information, which is needed for le-system and queuing information. The aggregator framework provides a periodical polling of information, which can be used, e.g., for system- and network-specic information. The special requirements of the C3Grid rule out the use MDS4 for the Data Information Service. It is not aware of the special types of geographic metadata (e.g. the ISO19115 scheme). It is not designed to provide specialized search queries, like range searches. It is not capable of managing the amount of metadata on a gigabyte scale OGSA-DAI OGSA-DAI [OGS06] is a grid middleware for accessing dierent data resources, like relational databases, XML databases and le systems, in a unied way. It was developed mainly at the EPCC, Univerity of Edinburgh, Scotland and at IBM Hursley. OGSA-DAI provides an abstraction from the dierent database management systems (relational, XML based,...), query languages and database drivers. The user of OGSA-DAI can access these resources in a generic way via a common web service interface. As it is integrated in the Open Grid Service Architecture (OGSA), other OGSA services can build on OGSA-DAI to provide higher level services like data federation or distributed queries. Querying and updating data, as well as data transformations and data passing, is provided via a number of protocols. In addition, metadata can be requested { about data of a particular resource, about the resource itsself or about the functionality provided to access this resource. OGSA-DAI is designed to work with the Web Service Resource Framework (WSRF) of Globus Toolkit 4. There also exists an OGSA-DAI distribution conforming to the Web Service Interoperability (WS-I) standard used in the UK OMII web services infrastructure. The web services for integrating resources also allow extensions, so the user can integrate additional access functionalities. Architecture of OGSA-DAI The OGSA-DAI architecture is based based on the Web Service Architecture [BHM + 04], consisting of three main components. GDS (GridDataService) is a transient web service, acting as access point to a specic data resources. GDSF (GridDataServiceFactory) is a persistent web service providing meta-information about a specic resource and establishing instances of GDS services. DAISGR (DAIServiceGroupRegistry) is a persistent web service providing the discovery of GDFS services. For accessing a data resource, the user queries a DAIASGR, which returns the the respective GDSF the user was looking for. This service generates an instance of GDS as access point for the user. The user prepares a suitable perform document which contains the SQL query to be submited to the data 12 / 51 C3Grid-T Draft

13 resource. The GDS accepts this document and passes is to the database. The result of this request is passed back to the user. According to the developers, OGSA-DAI supports the following database systems. relational databases: MySQL or above IBM DB2 (no delation or creation of databases under OGSA-DAI) Microsoft SQL Server (no support of the SQL Server IMAGE column type under OGSA-DAI) Oracle (no creation or deletion of databases under OGSA-DAI, no support for addition of BLOBs to tables in Oracle 9 databases) PostgreSQL HSQL (tested but not ocial supported) XML databases: Apache Xindice exist 1.0 beta2 (tested but not ocial supported) File Systems: several Applicability to the Grid Information Services OGSA-DAI is especially suited for providing generic access to dierent types of data resources. Therefore, its deployment in C3Grid could provide high-level access to the dierent repositories of the C3Grid organizations. Neverteless, the requirements of the data and information management in the C3Grid are very specic. to do Storage Resource Broker (SRB) The Storage Resource Broker (SRB) is a data management system which has been developed by the San Diego Supercomputer Center (SDSC). In the beginning, this development was motivated by the need for a uniform access to the data of the SDSC. Today it is deployed in many dierent projects worldwide. The SDSC managed projects alone use SRB for a total of 626 TB, 99 million les and 5000 users. SRB provides uniform access to heterogeneous data resources, like le systems and relational databases. Uniformity is reached by a logical namespace and a common metadata and user management. SRB also provides the creation and management of le replicas. Components of the SRB y An SRB deployment is structured in three layers. The physical layer consists of the dierent types of data repositories SRB provides access to. There are two abstractions providing either a database or storage repository style access. Table 4.1 gives an overview of the repositories supported so far. The central layer of an SRB deployment consists of the SRB middleware, which provides the main functionalities { logical namespace, latency management, data transport and metadata transport. The functionalities are used by a common consistency and metadata management. The middleware also provides authorization, authentication and audit. The application layer consists of a collection of tools and interfaces providing high-level access. The following list gives an uncompleted overview. C3Grid-T Draft 13 / 51

14 Table 4.1.: Repository types supported by SRB. y Abstraction Database Storage Repository Systems IBM DB2, Oracle, Sybase, PostgreSQL, Informix Archives { Tape, SAM-QFS, HPSS, ADSM, UniTree, ADS ORB File systems { UNIX, NT, Mac OSX Databases { DB2, Oracle, PostgreSQL, mysql, Informix APIs for C, Java, Python, Perl,... Unix shell commands. Graphical user clients (NT Browser, mysrb...). Access via HTTP, DSpace, OpenDAP, GridFTP Webservice-based access via OAI, WSDL, WSRF. Federated Zones y The typical distributed architecture of the SRB middleware is distributed in administrative units, called SRB zones. A zone consists and one or many SRB servers. One server also keeps the metadata catalog (MCAT) of the SRB zone. The MCAT contains information like user information, access rights and resource information. It is deployed in a relational DBMS. Each SRB servers can provide access to several resources. For the connection of dierent zones, SRB provides the federated zones mechanism. It allows mutual access of resources, data and metadata of several zones. Though, federation user accounts of each local zone are made available in all remote zones. Still, these zones remain independent administrative domains. Synchronization of the MCATs of federated zones is reached by the periodical execution of a provided script. Metadata y Metadata stored in the MCAT can be classied by their purpose. administrative and system metadata allow the mapping from logical to physical le names and to authenticate users. user metadata provide a means of describing data objects by attribute values. extensible schema metadata use the facility SRB provides to integrate external metadata schemas into the MCAT. SRB provides a detailed search on its MCAT metadata, e.g., for le creation times or user-given attributes. Replica Management y Replication is used by client or server side strategies which aim to enhance the access time by providing a le closer to the location it is needed. Replicas can be created in dierent ways: by copying manually, implicitly by logical resources or by user registration. Replicas are synchronized by a call of a special command. 14 / 51 C3Grid-T Draft

15 Rights Management y Users of an SRB Zone must be registered in the respective MCAT. For Authentication several techniques are provided (password, GSI, etc.), though, they are not supported by all clients. The denition of user groups is possible, as well. The SRB server provides ne-grain authorized access on its object by access control lists (ACL). An additional concept of authorization is via tickets. Users which access the SRB with tickets need not be registered in the MCAT. Scalability by Master and Slave MCATs y In order to enhance the response time of MCAT operations, the MCAT can be replicated to several SRB servers in a single zone. These MCAT replicas are able to share the load of requests. Reading requests are done on one of the many slave MCAT. Modifying access is handled by a dedicated master MCAT. To be able to replicate MCATs the underlying DBMS must support database replication. Logical Resources y Another feature of the SRB server is to provide a single logical view for a number of physical resources. A special application for this is the automatic replication of objects to several servers. For example, two servers may each have a local physical resource and share both as single logical resource. By setting the number of replicas to 2, a replication on both physical resources is guaranteed. Other Features y The SRB comprises the following additional features. Extraction of data can be combined with preprocessing, e.g. to generate previews of images. Many small les can be combined in containers to prevent their fragmentation over several tape archives. Support of parallel I/O operations. Bulk operations, which enhance the transport of many small les. SDSC SRB vs. Nirvana SRB y Currently, there are two development branches of the Storage Resource Broker. SDSC maintains the open source branch of the SRB development, whereas Nirvana oers a commercial version of SRB. Though Nirvana SRB originates from the SDSC branch, both versions are mutually incompatible. The following lists gives a brief overview of the dierences between both versions. Nirvana SRB does not support federated zones Nirvana SRB uses system daemons to guarantee the synchronicity of replicas and the global namespace. SDSC SRB uses a rsync-like mechanism, which has to be called externally. Nirvana SRB supports drivers for a wider range of repository types The licence of SDSC is restricted to non-prot use. The short-term use of the C3Grid would meet this criteria. Nirvana provides commercial support. SDSC provides free community support. C3Grid-T Draft 15 / 51

16 Applicability to the Grid Information Services Especially the ability to manage meta information about les makes SRB an interesting candidate for the Data Information Service. The SRB MCAT keeps replication information and metadata specic for logical objects (les, collections, etc.). The latter information is saved in user-given attribute value pairs, which may suite well for most applications but is not aware of the special scheme of C3Grid metadata (ISO 19115). By extensible schema metadata it is possible to integrate it in SRB but this also requires extra coding. Also the managment of replica information does not fully comply the requirements expecially of AP5 [LS06a] The metadata search would also need a wrapping to provide the Portal a coherent serach for the dierent types of metadata. Note, e.g., that use metadata may be generated dynamically on request. Noch zu untersuchen: DIS: OGSA-DAI SRB MCAT- foderierte Zonen (SDSC ja/ Nirvana nein) (dcache) nur files? EGEE glite? 4.5. Open Challenges Timeconstraints; coupling of AP2, 5, and 6; ISO / 51 C3Grid-T Draft

17 5. Architecture of the Resource Information Service As self-contained component, the Resource Information Service (RIS) provides an information service for resources available on the C3Grid Design portal distributed grid Infrastructure WebMDS (Tomcat Servlet) MDSIndex xml data xml transform aggregator service host information: processor information memory size cpu load queueing information memory statistics local resources and interfaces at the institutes collect generic: local or remote program WSRF: push or pull based Frontend GT4 Frontend GT4 Frontend GT4 register information provider data flow direction of call Figure 5.1.: Architecture overview of the Resource Information Service For the implementation of the RIS, the Monitoring & Discovery Service (MDS) [SDM + 05] of the Globus Toolkit 4 (GT4) is used. As depicted in Figure 7.1, it provides information about the grid state by keeping track of the grid host systems and by holding the gathered information in a global index. Basically, MDS consists of an architecture of three layers. A web interface to the portal oering access to the information system An index service, which is aggregating resource information in a catalog. Information providers, which are are GT4 host systems oering information about their resources. The resource information is collected by the index service either by querying the WSRF services running on the hosts or by using dedicated tools supplied by the system administrator. For the portal, MDS provides the WebMDS service, which is a web-based user interface. WebMDS uses XSL transforms to generate a user-friendly frontend to the index service, allowing property requests and displaying the results in various formats. The following components of the C3Grid act as information providers to the Resource Information Service. The Workow Scheduler, providing queueing information The compute resources, reporting system information C3Grid-T Draft 17 / 51

18 The following components of the C3Grid act as clients to the RIS. The Portal, for system monitoring and queuing information The Workow Scheduler, requesting host and compute capabilities The Data Information Service, asking for available data providers and workspaces 5.2. Use Cases Monitoring Scheduled Jobs RIS Workflow Scheduler Portal 1. register 2. submit job description time index update 3. notify 4. poll queuing/job state user request Figure 5.2.: Sequence diagram: submission and monitoring of workow jobs Resource Information Service. Figure 5.2 shows a typical use of the Resource Information Service for monitoring scheduled workow jobs. 1. At startup time, the Workow Scheduler registers to the RIS. From now on, the RIS knows the Workow Scheduler and can collect queuing information from it. 2. The portal submits a workow specication to be scheduled and executed on the grid. 3. The Workow Scheduler noties the Resource Information Service as soon as the state of any job changes. 4. On a user's request, the portal collects the current queuing information stored in the RIS index. These steps can be run concurrently provided that the Workow Scheduler is registered before it sends notications. The portal is interested in the following information stored in the RIS. queue state (load,...) 18 / 51 C3Grid-T Draft

19 job state (queued, running, nished,...) estimated remaining run-time of a job state of available compute nodes Serving the Workow Scheduler Workflow Scheduler RIS Processing Host 1. register 2. collect time node information 3. collect node information Figure 5.3.: Sequence diagram: Workow Scheduler collecting processing capabilities from the Resource Information Service. Workow specications commonly contain resource prerequisites needed to run the respective processing. For example, some user-provided tool may need a special operating system or processing tools provided in the grid, which are available only on certain hosts. In order to match these prerequisites with the resources available, the Workow Scheduler needs to communicate with the RIS. Figure 5.3 illustrates the necessary operations. In addition to the availability of hosts, the following host specic information is needed by the Workow Scheduler. CPU type Operating system Special tool T is available Note that le-system-specic information is not provided by the Resource Information Service but by the Data Information Service (see Chapter 7) Recovery of the Data Information Service or the Grid Data Management System If the DIS is being restarted, it has to gather all information necessary to resume its service. The rst step is to query the RIS for hosts providing primary base data, primary metadata and/or processing workspace (see Figure 5.4). Furthermore, the DIS needs the following host-specic information. URI of the metadata information service URI of the data staging service The root path of the local processing workspace The second recovery step for the DIS is to ll its cache with discovery metadata. This step is part of a separate use case described in Section C3Grid-T Draft 19 / 51

20 DIS RIS Processing Host 1. register 2. collect time node information 3. collect node information Figure 5.4.: Sequence diagram: Data Information Service collecting host information from the Resource Information Service. 20 / 51 C3Grid-T Draft

21 6. Interfaces of the Resource Information Service As stated before, the Resource Information Service will be a deployment of the Monitoring & Discovery System (MDS) [SDM + 05] of the Globus Toolkit 4. This System already denes interfaces for the web frontend (WebMDS). For the acquisition of information providers, MDS is build on the WS MDS Aggregator Framework. this framework denes aggregator sources, which include modules that (a) query resource properties, (b) acquire data through subscription/notication, and (c) execute programs to generate data. All three types of aggregator sources may be used in the C3Grid. Type (a) is used for information, which can be collected periodically, like host or network state information. Type (b) is used when information changes non-regularly, like queuing information. Type (b) is the preferred type where scripting based information collection is simple to maintain or can build upon existing tools. C3Grid-T Draft 21 / 51

22 7. Architecture of the Data Information Service The second component of the Grid Information Service is the Data Information Service (DIS). This service manages all information about data present in the C3Grid. This chapter provides an overview of the architecture of the DIS and describes how it satises the special requirements of the C3Grid Design distributed grid Infrastructure global DIS global metadata discovery metadata disciplin keywords some disciplin some keywords some disciplin other keywords some disciplin more keywords parameter humidity temparature air pressure describes primary and grid data region time interval OID some range 10/ /2004 ABC another range 01/ /3002 ADX a range 01/ /2004 VDK use metadata describes data available in the grid local DIS local DIS local DIS filename duffy bugs elmar format grib CF CF other X Y Z OID ID ABC 23 ADX 42 ADX 33 local resources and interfaces at the institutes primary primary metadata primary metadata metadata request of use metadata polling of discovery metadata use metadata filename format duffy grib bugs CF elmar CF variable X Y Z describes data at primary sites Figure 7.1.: Architecture overview of the data information service Figure 7.1 illustrates how the DIS meets the above requirements. In order to achieve fault tolerance and fast access, the DIS does not store all metadata at a single location, but acts as caching proxy to metadata available at primary data providers or in the grid workspaces. For this purpose it keeps a central catalog, where new metadata are inserted as soon as they are available in the grid. The requirement of providing large amounts of metadata is met by caching only discovery metadata and delegating individual requests for use metadata to the respective data source. This partial caching reduces the total amount of metadata to be kept in the information service. A fast response time for search requests is retained by guiding the user to perform a two-step search. The rst query is on discovery metadata only, yielding metadata sets and respective object identiers. With these object identiers, the user can submit a second query for use metadata on a limited selection of hosts, yielding information about how to access the respective base data. Discovery metadata cached in the global DIS catalog needs to be updated periodically by polling the providers of primary metadata. Because metadata generated at the primary sources is unlikely to be removed and new metadata is produced rather infrequently, an update needs to be performed only few times a day and inconsistency is not expected to be a critical issue. Also the amount of new metadata transferred each update is expected to be rather small. The DIS is closely coupled with the Grid Data Management System (DMS) [LS06b] and also shares some of its interfaces with it. This coupling is motivated mainly by the many tasks of the DMS which involve also the management of metadata. On the one hand, the DMS needs to keep track of the location of replicated data on several workspaces. On the other hand, also metadata generated by grid processes are les which must be handled by the DMS. 22 / 51 C3Grid-T Draft

23 Data Information System Data Management System ISO OID (0,1) keeps meta (0,1) host model disc. meta (1,*) (1,1) describes (1,*) (1,1) file is located replica project search attributes (1,1) refers to (1,1) URI logical path primary path min life max life pin OID format Cera use meta Pangaea use meta... use meta Grid Workspace use meta stored/generated at local providers Figure 7.2.: Entity-relationship diagram of the common data modle of the Data Information Service and the Grid Data Management System. y Note that the DMS also acts as data provider for data generated or used by grid jobs. Because respective use metadata are created, updated and deleted frequently, they are managed directly by the DIS. The entity-relationship diagram 1 in Figure 7.2 shows the common data model of both the DIS and the DMS. it is implemented by a metadata catalog which is shared by both services (see Figure 7.3). This allows a simpler, more coherent handling of the dierent types of metadata Use Cases Below, we outline the functionality of the DIS by typical use scenarios User's Search for Base Data and Metadata Figure 7.4 shows a typical use of the Data Information Service. A user searches for information about available data and prepares the access to the input for the processing tasks he is planing to perform in the grid. 1. The user initiates a search for climate data available in the grid or at local data providers. Input: search values can be given for all the following attributes. 1 An entity-relationship diagram uses two kinds of concepts: entities (boxes) and relations between entities (diamonds connecting boxes). Both kinds of concepts can have attributes (ovals). Key attributes (underlined attribute name) uniquely identify instances of the respective entity. Relations are annotated with (min; max) information. A tuple (n; m) between an entity E and a relation R indicates that an instance of E participates in R at least n and at most m times. C3Grid-T Draft 23 / 51

24 portal GUI/API distributed grid Infrastructure AP2 A global DIS B global DMS AP5 AP6 workflow scheduler catalog local DIS local DMS process execution local resources and interfaces at the institutes C primary metadata D primary data preproc. metadata data workspace DIS: Data Information Service DMS: Data Management Service internal data flow X direction of request external interface Figure 7.3.: Combined Data Information Service (DIS) and Grid Data Management System (DMS). y project (e.g., IPCC, WOCE,... ) model (ECHAM, OPA, MPI-OM,... ) or observation compartment (atmosphere, ocean, lithosphere) provider (WDCs,... ) time and space bounds all attributes needed to meet the requirements of any predened workow Also constraints for the returned result set can be specied: The number of results per page The number of the rst result (defaults to 0) Output: result of the search. I.e., list of results total number of results A single result consists of a set of a minimum set if information the user and the system needs to identify data. object identier project name model/observation compartment source (primary source or grid) Exception: Search which would result in too much data is forbidden. (E.g., search without any attribute specied) 2. The user is interested in detailed information about one of the data sets returned in the previous search. For this data set he requests all use metadata, i.e., an ISO XML document. 24 / 51 C3Grid-T Draft

25 Portal 1. search data search results Data Information System lookup in local catalog Primary Metadata 2. get metadata ISO document lookup in local catalog time 3. get use metadata data is at primary provider 3.1 get use metadata access information 4. get use metadata access information data is in grid: lookup in local catalog Figure 7.4.: Sequence diagram: user's search for base data and metadata via the portal. Input: object identier Output: ISO document 3. In order to complete the workow specication, the portal requests information about how to access data (i.e., use metadata). For data oered by primary data providers, the Data Information Service needs to request the use metadata from the respective local site (3.1) and passes them to the portal. Input: Preferences how the data should be accessible. object identier le format (GRIB, LOLA, CF) variable/parameter spatial cut time cut other site-specic pre-processing Except for the identier, these arguments are all optional and given as attribute-value pairs. Output: Data access information, i.e., host name list of le names metadata le name C3Grid-T Draft 25 / 51

26 le format (GRIB, LOLA, CF) Exceptions: The data source may not support cuts according variable, space, time 4. If the user requests use metadata for base data available in the grid, the Data Information Service looks up this information in its catalog directly Caching of Discovery Metadata Data Information System Primary Metadata 1. get metadata diff initialization all discovery metadata time 2. get metadata diff since T periodic update difference since last call Figure 7.5.: Sequence diagram: caching of discovery metadata in the Data Information Service. The catalog of the Data Information Service caches discovery metadata in order to accelerate the search in the portal. The cache is kept up-to-date by polling the primary data provider regularly (harvesting). (See Figure 7.5) 1. In order to initialize the catalog, all discovery metadata is retrieved from the primary metadata providers. Input: Time of the last call { must be set to undened in the case of an initialization. Output: A metadata di. This is actually a list of records which contain the following information. An object identier. A timestamp of the last creation, update or delete operation. The set of discovery metadata. A status ag indicating a delete operation. 2. Periodically, the Data Information Service polls all metadata providers for new use metadata. Input: Time of the last call Output: A metadata di of the format described in step 1. For the implementation of this harvesting scheme, the OAI-PMH [OAI02] standard will be used. 26 / 51 C3Grid-T Draft

27 Data Information System DIS/DMS Interface Workflow Scheduler 1.1 register metadata 1. register base and meta file register discovery and use metadata time Data Management System 1.2 register base and metafile register files and local replicas object identifier Figure 7.6.: Sequence diagram: registration of new base data and metadata produced in the grid. y Registration of New Data y New data produced as output of a workow execution usually goes along with new metadata describing it. Actually, a workow can produce a series of output les which are described semantically by a single metadata le. All of these generated les must be registered at both the Grid Data Management System and the Data Information Service in order to make them available in the grid. For this task, as illustrated in Figure 7.6, the Workow Scheduler calls the common interface of both services. 1. The Workow Scheduler calls the common interface to register all les. Input: new base data les and single metadata le Output: void 1.1 The Workow Scheduler calls the Data Information Service to register the new meta information Input: discovery and use metadata Output: void 1.2 The Workow Scheduler calls the Grid Data Management System to register the new les. Input: base and meta les Output: void C3Grid-T Draft 27 / 51

28 8. Interfaces of the Data Information Service 8.1. Interface to the Portal This interface provides search functionality implemented in the Data Information Service. The grid Portal uses this interface to provide the user a guided search for any climate metadata oered through the grid. As mentioned before, the Data Information Service distinguishes metadata according to its purpose (discovery or use) and to the location of its corresponding data (primary source or grid workspace). In the interface for the Portal, the rst dierentiation is reected by several operations for accessing metadata { two for searching and accessing discovery metadata, one for requesting use metadata. The second dierentiation, whether data has to be fetched from a primary data source or is available somewhere in the shared grid workspaces, is returned as information in the respective search result set Datatypes Type oid is used for unique identiers of metadata sets. Type discoverymeta describes the main information of a set of discovery metadata. The content of this datatype is the rst information the user gets as result of a metadata search. type discoverymeta = f identier : oid, project : string, model observation : string, compartment : string, source = ( Primary Workspace ), g Instances of type discoveryresult contain results from a search for discovery metadata. type discoveryresult = f object : discoverymeta list, size : int g Type isometa is a string conforming to the C3Grid discovery metadata schema (ISO 19115/ISO 19139). type isometa = string Use metadata is stored in an instance of type usemeta. In addition to the identier of the meteorologic data set, this type contains a list of le information 28 / 51 C3Grid-T Draft

29 type usemeta = f parent : oid, object : f url : string, format : ( Grib NetCDF LOLA ), dimensions: string list, size : int, sourcetype : ( Primary Workspace ), creationduration : int g list g Operations of the DataInfoService The operations of the remote Data Information Service are dened as follows. searchmeta (in arguments : (attribute,value) list) : discoveryresult Queries the information system for discovery metadata. The values bound to the search attributes are used to either match a respective metadata attribute or to dene a bound in a range query (e.g., timemax, timemin, latmin, latmax, longmin, longmax,...). Parameters: arguments list of attribute-value-pairs to be used for search. possible search attributes are: project (e.g., IPCC, WOCE,... ) model (ECHAM, OPA, MPI-OM,... ) or observation compartment (atmosphere, ocean, lithosphere) provider (WDCs,... ) timemax, timemin, latmin, latmax, longmin, longmax,... all attributes needed to meet the requirements of any standard workow Returns: search results getdiscoverymeta (in identier : oid) : isometa Requests the complete set of discovery metadata for a given climate object. Parameters: identier Returns: object identier discovery metadata as ISO 19115/ISO document. getusemeta (in host : string, in objs :in oid list, in constraints :(attribute, value) list ) : usemeta list Queries the information system for use metadata. Each call of this operation is specic to a single host. This restriction is due to the fact that the Data Information Service does not handle the request C3Grid-T Draft 29 / 51