Collaborative Climate Community Data and Processing Grid (C3Grid) T5.1: Grid Data Management Architecture and Specication

Transkript

1 Collaborative Climate Community Data and Processing Grid (C3Grid) T5.1: Grid Data Management Architecture and Specication Work Package: AP 5: Grid Data Management Author(s): Tobias Langhammer, Florian Schintke Version Idenitier: C3Grid-T Draft Publication Date: March 2006 Work Package Coordinator: Zuse Institute Berlin (ZIB) Partners: ZIB, GKSS, Uni Kln FUB, DLR, DWD Contact: Tobias Langhammer, Florian Schintke

2 Contents Contents 2 1. Introduction Notation y Common Terms y The C3Grid Project y 7 3. Global C3Grid Architecture y 8 4. The C3Grid Grid Data Management System Requirements Available Tools and Solutions dcache Storage Resource Broker (SRB) gridftp Chimera Open Challenges Grid State Prediction ISO19115 as Scheme for Climate Metadata Architecture of the Grid Data Management System Design AP5 Use Cases Querying the Current State Querying State Predictions Agreements Staging and Transferring Files Registration of Generated Files High-level File Operations Interfaces of the Grid Data Management System Interface for the Workow Scheduler Exceptions Datatypes Operations of the DataService Interface of the Primary Data Provider for the Data Management Service Operations of the DatabaseAccess Operations of the FlatFileAccess Authorization y 29 2 / 48 C3Grid-T Draft

3 9. Conclusion 30 A. Interface Spezication Syntax y 31 B. Questionnaire for Users and Data Providers 32 B.1. Important Aspects B.2. Miscellaneous B.3. Aspects of Data Management B.4. Aspects of Metadata Management C. Quotes from the Development Discussion y 41 C C.2. C3Grid-Wiki C.2.1. WF im Portal (Stand ) C.2.2. Metadaten im WF (Stand ) C.2.3. AG-Metadaten (Auszug, Stand ) C.2.4. AG-Metadaten-Meeting1 (Stand ) C.2.5. Metadaten-Meeting-Marum (Stand ) C.3. Mailing Lists Bibliography 48 C3Grid-T Draft 3 / 48

4 1. Introduction This document describes the results of the discussion on the architecture, interfaces, and functionality of the C3Grid Grid Data Management. The document is structured as follows Notation y The y-sign The two documents [LS06a, LS06b], which describe the C3Grid Grid Information Service and the C3Grid Grid Data Management System are related to each other. To make each document self contained, some sections are shared between both documents and printed in the exactly same way in both documents. Such chapters, sections, gures, etc. are marked by the y -sign. Interface Denitions For the specication of interfaces, this document uses a notation, which is described in the following. Datatypes used are common basic datatypes (like int, double, string) and constructed types. Constructed types can be lists, pairs, records, enumerations or variants. They can be bound to new names by a type denition. For example, type foo = (int,double) list denes a new type foo which is a list of (int,double) pairs. For dening a record with an int and a oat entry we write type foobar = f foo : int, bar : double g For dening enumerations we write type color = ( Red j Green j Blue j Yellow ) Variants are a generalization of enumerations, where each handle can be assigned a type type color = Name string j RGB (int, int, int) 1 Service Interfaces are decribed in sections called 'Operations of the FooService'. Such section lists the respective operations of the interface. Callers can use the functionality of the interfaces by invoking the listed operations remotely (using Web Services). Remote operations are given by their signature (consisting of the name, parameter list and return type), a short functional description and a detailed list describing the parameters, the return value and exceptions. The following example demonstrates this notation. 1 Note that a variant data type can be mapped to a C union. 4 / 48 C3Grid-T Draft

5 foo (in huey : double, inout dewey : bool, out louie : string ) : int Example operation with some special functionality. Parameters: in huey Example input argument which is a oating point value inout dewey Example boolean argument which is modied by the operation. out louie Example string argument which is set by the operation. Returns: Example integer return value. For a detailed BNF-style description of the interface denition syntax see Appendix A Common Terms y The terms must, must not, should, should not and may are used in accordance to RFC 2119 [Bra97]. This specication also uses the following terms. base data: data which is not metadata, i.e. actual climate data les. grid sites: sites within the C3Grid providing data capacity (workspaces) for processing and storing les temporarily. metadata: Data that describes other data. In the C3Grid we distinguish metadata by their origin and purpose. primary metadata: metadata generated and stored at primary data sites. Though parts of this metadata may be cached by the Data Information System, the original copy always remains out of reach for C3Grid middleware. grid metadata: metadata generated in the grid workspaces and managed by the Data Information Service. This metadata also describes data only stored in the grid workspaces. discovery metadata: data necessary to nd climate data in the C3Grid. use metadata: metadata describing how the processing tools can access a data object. le metadata: metadata used by the Grid Data Management System in order to track the location of replicated les. Note, that the set of primary metadata objects and the set of grid metadata objects are mutually exclusive. Discovery and use metadata are either primary or grid metadata, le metadata are always grid metadata. primary data: Read-only data at a primary site. The creation of primary data is outside the scope of the C3Grid. primary site: Organisation providing a large data repository which allows read-only access (DBMS, archive,... ) process: In terms of workows, an atomic processing unit. In terms of operating system, a running program. workspace: Local (to a grid site) disk space, where data is made available for processing. C3Grid-T Draft 5 / 48

6 DOI: Digital Object Identier [DOI]. A system for identifying objects on the Internet. The DOI of a data object is unique and does not assign it to a certain location like a URI. DOIs provide a persistent way of accessing data objects by a mapping service. In the C3Grid DOIs are used for identifying single datasets. The data service provides DOIs to identify search results. These DOIs can then be used for requesting primary data from the data repositories. 6 / 48 C3Grid-T Draft

7 2. The C3Grid Project y The aim of the Collaborative Climate Community Data and Processing Grid is to develop a productive, grid-based environment for the German earth system science, that supports eective, distributed data processing and the interchange of high volume datasets from models and observations between participating institutions. The collaborative system as a whole will provide unied and transparent access to the geographically distributed archives of dierent institutions and simplify the processing of these data for the individual researcher. Specic challenges are to provide a collaborative platform for accessing and processing huge amounts of data, as well as the eective management of community specic meta information. Earth system research studies the system of earth and the processes within, and wants to develop models to predict its climate changes. The C3Grid will provide the infrastructure to share data and processing power across institutional boundaries. C3Grid-T Draft 7 / 48

8 3. Global C3Grid Architecture y The C3Grid provides an infrastructure which is designed to the specic needs of the community of climate researchers. It provides a means of collaboration, integrating the heterogeneous data sources of the participating organizations and oering easy access by a common users interface. The main application of the C3Grid is the analysis of huge amounts of data. To increase the eciency the system provides automatic reduction or pre-processing operations and exploits data locality, replication and caching. user Userinterfaces API GUI Nutzer Grid data management Grid scheduling Distributed Grid infrastructure Grid information service data-transfer service Local interfaces to institutional resources userrmodules pre-processing data jobs Existing resources of the institutes metadata data archive access to archive distributed distributed data archives data archives distributed distributed computeresources computeresources Figure 3.1.: Architecture of the C3Grid. The architecture overview of Figure 3.1 shows the major components of the C3Grid in their operational context. Existing data repositories are integrated by dedicated interfaces for providing primary base data and respective metadata to the middleware. Pre-processing capabilities allow a rst on-site data reduction. A distributed grid middleware consists of an information service, data management and transfer services, and a scheduler. Actual jobs are executed on data which was extracted (staged) from primary repositories and made available on a local disk share. Each processing not only produces new base data, but also generates new metadata from metadata describing the original input. The information services provides a comprehensive search for resources, primary data available in local repositories, and data created at local grid shares. The Portal uses these services to assist users in their data analysis workows. 8 / 48 C3Grid-T Draft

9 4. The C3Grid Grid Data Management System Still to be written: Responsibilities of the Service Requirements The following requirements for the Grid Data Management System are specied in the C3Grid project proposal [DG] (R1) The system must coordinate the access and transfer of data in the grid. Files must not be created or transferred unless it is necessary. The system must be economic in handling huge amount of data. (R2) The system must provide consistent access to all data provided by primary repositories. (R3) On request, the system must provide replicas at processing resources. (R4) The system must provide information about transfer times by referring to knowledge about network topologies and bandwidths. (R5) The system must provide information about replicas. (R6) The system must be tightly coordinated with the Workow Scheduler to support it in providing an ecient data and process scheduling. (R7) The system must be fault tolerant, i.e., in the case of a breakdown it should recover automatically. Special requirements derive from the answers by the C3Grid users and data providers given in the questionnaire of Appendix??. and from the discussion among the C3Grid development groups (see Appendix C) (R8) The system must be usable with databases (like CERA or Pangaea) as well as at le storages, like storage area networks (SAN) and hierarchical storage management systems (HSM). (R9) The system must support data providers of a total amount of several hundred terabytes, data sets of several hundreds of gigabytes and les of several gigabytes. (R10) For staging data from databases, the system must pass pre-processing specications from the workow specication to the data provider. The system may not be aware of the structure of these specications. 1 (R11) The system must support metadata as input and output les of sets at least as les Available Tools and Solutions Zu untersuchen hinsichtlich 1. Transfer 2. Manamgement und Replikate 3. Zugriff tertiare Speicher 4. globales Verzeichnis existierender Speicher 1 See also the quotes from the C3Grid wiki in Section C.2.5 and the use case discussed in the of Section C.1. C3Grid-T Draft 9 / 48

10 Tools: dcache (3) SRB (1,2,3) gridftp, GSI openssh (1) Chimera (2) EGEE? dcache DCache [Fuh] is a data management system which aims to provide the storage and retrieval of huge amounts of data. This data is distributed over a big number of heterogeneous data nodes. Transparent access is provides via a logical le system. The exchange of data between the backend servers is automated. the system provides features like disk space management, automatic data replication, hot-spot detection and error recovery. External hierarchical storage systems (HMS) are integrated transparently and automatically. An NFS interface allows name space operations (no reading and writing of les). DCache is a development of the CERN in the context of a high energy physics experiment which will produce a continuous stream of 400 MB/sec in The experiment consists of three tiers. A single tier 0 site (CERN) will be the main data source. Few tier 1 sites save this data in a distributed scheme providing persistence through tape backups. Many tier 2 sites provide additional CPU and storage resources. The architecture of dcache consists of a central Workload manager and resource broker accepting jobs and passing them to local sites. A site consist of a compute element (CE) accessing a storage element (SE). Each SE is controlled by a Storage Resource Manager (SRM) and is connected to remote SEs to exchange data via GsiFTP or GridFTP. Applicability for the Grid Data Management System DCache meets very few of the requirements of the Grid Data Management System. As its name already suggests, it is mainly a distributed cache integrating dierent sources of a single virtual organization. This architecture does not apply to the independent administrative domains of the C3Grid data providers. The transparent replication mechanism of dcache does not meet the need for the close collaboration between the Grid Data Management System and the Workow Scheduler. This collaboration involves detailed agreements about transfer and staging times. Furthermore, with respect to the requirements of AP2 [LS06a], dcache does not provide the managment of additional metadata Storage Resource Broker (SRB) The Storage Resource Broker (SRB) is a data management system which has been developed by the San Diego Supercomputer Center (SDSC). In the beginning, this development was motivated by the need for a uniform access to the data of the SDSC. Today it is deployed in many dierent projects worldwide. The SDSC managed projects alone use SRB for a total of 626 TB, 99 million les and 5000 users. SRB provides uniform access to heterogeneous data resources, like le systems and relational databases. Uniformity is reached by a logical namespace and a common metadata and user management. SRB also provides the creation and management of le replicas. 10 / 48 C3Grid-T Draft

11 Table 4.1.: Repository types supported by SRB. y Abstraction Database Storage Repository Systems IBM DB2, Oracle, Sybase, PostgreSQL, Informix Archives { Tape, SAM-QFS, HPSS, ADSM, UniTree, ADS ORB File systems { UNIX, NT, Mac OSX Databases { DB2, Oracle, PostgreSQL, mysql, Informix Components of the SRB y An SRB deployment is structured in three layers. The physical layer consists of the dierent types of data repositories SRB provides access to. There are two abstractions providing either a database or storage repository style access. Table 4.1 gives an overview of the repositories supported so far. The central layer of an SRB deployment consists of the SRB middleware, which provides the main functionalities { logical namespace, latency management, data transport and metadata transport. The functionalities are used by a common consistency and metadata management. The middleware also provides authorization, authentication and audit. The application layer consists of a collection of tools and interfaces providing high-level access. The following list gives an uncompleted overview. APIs for C, Java, Python, Perl,... Unix shell commands. Graphical user clients (NT Browser, mysrb...). Access via HTTP, DSpace, OpenDAP, GridFTP Webservice-based access via OAI, WSDL, WSRF. Federated Zones y The typical distributed architecture of the SRB middleware is distributed in administrative units, called SRB zones. A zone consists and one or many SRB servers. One server also keeps the metadata catalog (MCAT) of the SRB zone. The MCAT contains information like user information, access rights and resource information. It is deployed in a relational DBMS. Each SRB servers can provide access to several resources. For the connection of dierent zones, SRB provides the federated zones mechanism. It allows mutual access of resources, data and metadata of several zones. Though, federation user accounts of each local zone are made available in all remote zones. Still, these zones remain independent administrative domains. Synchronization of the MCATs of federated zones is reached by the periodical execution of a provided script. Metadata y Metadata stored in the MCAT can be classied by their purpose. administrative and system metadata allow the mapping from logical to physical le names and to authenticate users. C3Grid-T Draft 11 / 48

12 user metadata provide a means of describing data objects by attribute values. extensible schema metadata use the facility SRB provides to integrate external metadata schemas into the MCAT. SRB provides a detailed search on its MCAT metadata, e.g., for le creation times or user-given attributes. Replica Management y Replication is used by client or server side strategies which aim to enhance the access time by providing a le closer to the location it is needed. Replicas can be created in dierent ways: by copying manually, implicitly by logical resources or by user registration. Replicas are synchronized by a call of a special command. Rights Management y Users of an SRB Zone must be registered in the respective MCAT. For Authentication several techniques are provided (password, GSI, etc.), though, they are not supported by all clients. The denition of user groups is possible, as well. The SRB server provides ne-grain authorized access on its object by access control lists (ACL). An additional concept of authorization is via tickets. Users which access the SRB with tickets need not be registered in the MCAT. Scalability by Master and Slave MCATs y In order to enhance the response time of MCAT operations, the MCAT can be replicated to several SRB servers in a single zone. These MCAT replicas are able to share the load of requests. Reading requests are done on one of the many slave MCAT. Modifying access is handled by a dedicated master MCAT. To be able to replicate MCATs the underlying DBMS must support database replication. Logical Resources y Another feature of the SRB server is to provide a single logical view for a number of physical resources. A special application for this is the automatic replication of objects to several servers. For example, two servers may each have a local physical resource and share both as single logical resource. By setting the number of replicas to 2, a replication on both physical resources is guaranteed. Other Features y The SRB comprises the following additional features. Extraction of data can be combined with preprocessing, e.g. to generate previews of images. Many small les can be combined in containers to prevent their fragmentation over several tape archives. Support of parallel I/O operations. Bulk operations, which enhance the transport of many small les. 12 / 48 C3Grid-T Draft

13 SDSC SRB vs. Nirvana SRB y Currently, there are two development branches of the Storage Resource Broker. SDSC maintains the open source branch of the SRB development, whereas Nirvana oers a commercial version of SRB. Though Nirvana SRB originates from the SDSC branch, both versions are mutually incompatible. The following lists gives a brief overview of the dierences between both versions. Nirvana SRB does not support federated zones Nirvana SRB uses system daemons to guarantee the synchronicity of replicas and the global namespace. SDSC SRB uses a rsync-like mechanism, which has to be called externally. Nirvana SRB supports drivers for a wider range of repository types The licence of SDSC is restricted to non-prot use. The short-term use of the C3Grid would meet this criteria. Nirvana provides commercial support. SDSC provides free community support. Applicability for the Grid Data Management System SRB provides many features which are required in the Grid Data Management System which makes it a candidate to be assessed for two types of deployments: as main component of the Grid Data Management System and as multiple deployment providing coherent access on data at local sites individually. In the rst case, SRB's federated zones are a must, because C3Grid sites remain independent administrative domains. This requirement is met by the SDSC SRB and rules out the application of Nirvana SRB. The replica management of SRB provides basic functionality. Nevertheless, the close coordination with the Workow Scheduler to provide ecient data scheduling and process scheduling implies agreements about the availability of replica at dened periods of time (also planned periods in the future). Therefore, a lot of extra coding is required to create, transfer and manage replicas according to the agreed lifetimes. Also extra coding is required to make the Grid Data Management System aware of metadata les. The registration of a le as metadata can be done by dening a special attribute in the MCAT, the integration of the ISO in the DIS needs to be solved (see also [LS06a]) SRB is especially suited to access grid processing workspaces or le storage systems. Database access would require a deep intrusion in existing DBMS, by-passing existing abstractions and interfaces which already provide features required in the C3-Grid (spatial, temporal cuts, aggregations, etc.). The implementation of le transfers in SRB, using techniques like parallel I/O, suites well the need of optimized data exchange between C3Grid sites. Also an important requirement of the C3Grid, the user management and authorization scheme of SRB could be integrated in the Grid Data Management System. The second type of deployment, i.e., multiple SRB installations at the sites, may be considered by local data providers and is outside the scope of the Grid Data Management System. The data management could use the SRB interface as common interface for accessing all data providers. Nevertheless, this setting does not seem reasonable because it requires all data providers to either use SRB or implement also the parts of the interface which are not used gridftp GridFTP is a protocal for secure, robust, fast and ecient transfer of data. The Globus Toolkit 4 provides an implementation of GridFTP, which is the most commonly used tool for this protocol. It C3Grid-T Draft 13 / 48

14 conists of a GridFTP server, which has access to data through an appropriate Data Storage Interface. This typically means a standard POSIX le system, but can also be a storage system like the SRB. To access remote GridFTP servers, GT 4 also provides a respective client. It is capable of accessing data via a range of protocols (http, https, ftp, gsiftp, and le). Because the client is a command line tool it is especially suited for scripting solutions. For special demands, GT4 provides a set of development libraries for custom client operations. Applicability for the Grid Data Management System GridFTP is a suitable tool for transfering les between C3Grid sites. implementation of a Grid Data Management System quite easily. Still to be written. It can be integrated in any Chimera Still to be written Open Challenges Grid State Prediction A key unique feature of the C3Grid middleware is the ability to use knowledge about access and transfer times to optimize le staging and job execution. Because the DMS directly communicates with the data providers and executes le transfers within the grid, it also needs to request and manage temporal information. Based on this information, the Grid Data Management System provides the Workow Scheduler with the following information. The estimation of staging and transfer times The estimation of le availability at a given location in the future The estimation of future disk space use Furthermore, the DMS reaches agreements with the Workow Scheduler basing on this information. So the actual execution plan of workows will be based not only on knowledge at present but also on predictions about the planned grid state in the future ISO19115 as Scheme for Climate Metadata Still to be written. Timeconstraints, AP2, 5, 6-coupling, ISO / 48 C3Grid-T Draft

15 5. Architecture of the Grid Data Management System portal GUI/API distributed grid Infrastructure AP2 A global DIS B global DMS AP5 AP6 workflow scheduler catalog local DIS local DMS process execution local resources and interfaces at the institutes C primary metadata D primary data preproc. metadata data workspace DIS: Data Information Service DMS: Data Management Service internal data flow X direction of request external interface Figure 5.1.: Overview of the external interfaces of the data information service (DIS) and the data management service (DMS). y 5.1. Design The main purpose of the Grid Data Management System (DMS) is to access data stores and to manage les used as input or output of computations at grid compute resources. It must co-operate with the following components of the grid middleware. the Workow Scheduler, issuing staging requests the local data providers, oering primary base data and respective metadata the data information system, keeping track of which meta-information corresponds to which data. Virtualization of Files The DMS aims to oer a high-level view on data by hiding the actual location of les. The users should be able to specify workows by giving the required input data and compute capabilities without the need to know where these resources are actually located. Consequently, the DMS manages le transfers at a lower level. It also keeps track of distributed copies of a le to be able to refer to its metadata. To avoid confusion, we use the term logical le, replicated le or simply le, if we refer to the high-level view of the location-independent data unit. The term replica is used for a single remote C3Grid-T Draft 15 / 48

16 copy of a le. This term also implies that the DMS knows about the respective representation as logical le. Local Workspaces Each C3Grid site oers a le system share which we call workspace. A workspace is used as target location for les which were requested from data providers or produced by grid jobs. The grid workspace at each site is identied by a root path pointing to a location in the le system. All sub-directories of this root path are dedicated to be controlled by the the DMS. Especially, no other user of the local system should have read or write access to this le share. Primary Data Providers Primary data providers are the main source of data for the processing tasks of the grid. The way the DMS can access them mainly depends on whether they are storage systems providing at les or databases providing a more exible access. If the DMS gets a staging request, in the rst case, it simply keeps a reference to the location of the requested le. In the case of databases, the DMS must stage data to at les in order to make it accessible for the processing. Because databases already implement pre-processing capabilities (e.g., by selective queries of temporal or spatial cuts), the DMS provides extra operations for this type of data provider. Output of such a pre-processing is a new data object which is stored only in the grid workspaces and has no counterpart in the database it originates. Note that we consider pre-processing a processing capabilitiy which is provided as integral feature of a data repository. Independent tools may also provide on-site data reduction (e.g., CDOs) but are better specied inside a job to be passed to the Workow Scheduler. Only this way the distributed grid environment can be used. Internal Structure The Grid Data Management System consists of two conceptually independent components: a gridwide global DMS and many local DMS. The global DMS is closely coupled with the Data Information Service (DIS) [LS06a] of AP2, with which it shares a common data model, which is depicted as entity-relationship diagram 1 in Figure 5.2. This data model contains not only DMS-related information, but also DIS-related information and relations between both. The most important aspects of this model are as follows. A single set of discovery metadata describes one or many data les. For each set of discovery metadata describing les in the grid there is an extra le containing this metadata. 2 The respective discovery metadata of a le can be identied by referring to the metadata object identier OID. A le has a name and a workspace path. 1 An entity-relationship diagram uses two kinds of concepts: entities (boxes) and relations between entities (diamonds connecting boxes). Both kinds of concepts can have attributes (ovals). Key attributes (underlined attribute name) uniquely identify instances of the respective entity. Relations are annotated with (min; max) information. A pair (n; m) between an entity E and a relation R indicates that an instance of E participates in R at least n and at most m times. 2 see also Section C / 48 C3Grid-T Draft

17 Data Information System Data Management System ISO OID (0,1) keeps meta (0,1) host model disc. meta (1,*) (1,1) describes (1,*) (1,1) file is located replica project search attributes (1,1) refers to (1,1) URI logical path primary path min life max life pin OID format Cera use meta Pangaea use meta... use meta Grid Workspace use meta stored/generated at local providers Figure 5.2.: Entity-relationship diagram of the common data model of the Data Information Service and the Grid Data Management System. y A replica has a host where it is saved, a lifetime range, and a pinning ag to save it from automatic deletion. A replica may have a primary path if it is not located below the workspace root path. (In fact this 'replica' is a primary at le the DMS have direct reading access to.) A local DMS is running at each C3Grid site which oers storage and le access capabilities. Its task is to communicate with primary data providers, and to manage local le stores. Replica Management In order to facilitate the staging of processing input, the DMS creates le replicas at the site where the data are needed. In order to facilitate the management of replicas, all replications of a single le are stored at the same location relative to the workspace root path. In the management of replicas, the DMS takes special care for a at le oered by a primary data providers. In this case, the DMS creates replicas only in workspaces remote to the primary le. At the location of the primary le no extra replica is generated in the workspace because the le is accessible directly via its path. Note that this scheme of omitting unnecessary replication conforms to requirement (R2). In case a primary provider is a transparent hierarchical storage system (HSM), the DMS must assure that the le it provides to the Workow Scheduler remains staged to the disk (e.g., by requesting a pin from the HSM). If the HSM is not able to keep the primary le staged, the DMS must copy it to its local workspace. The path of a primary le points outside the sub-directory structure of the local workspace root directory. Therefore, the uniform naming scheme for replications in remote workspaces can not be C3Grid-T Draft 17 / 48

18 maintained. For this purpose the DMS keeps two paths, a primary path pointing to the primary le in its local le system and a logical path within the workspace namespace used for all remote replicas In case of a disc space shortage, the RIS may decide to remove individual replicas. By setting a special ag (pinning) a replica will be saved from deletion. The Data Management System as Data Provider From a higher-level perspective, not only a primary data source but also the Data Information Service acts as data provider. Especially for the workow scheduler, the discrimination of staging from a primary source or creation of remote replications is irrelevant. For the purpose of le requests it uses the same operations of the DMS interface. The scheme for providing replicas at an agreed host and time is presented below in the section about use cases. 18 / 48 C3Grid-T Draft

19 6. AP5 Use Cases As described in the denition of the Data Information Service [LS06a], the portal can request detailed information about how to access data. The following subset of this information is need to specify the input of a workow. An object identier Optional aggregation or pre-processing commands for databases. A list of base data le names A metadata le name With this information the Workow Scheduler starts to interact with the DMS to prepare an optimized execution of the workows Querying the Current State Workflow Scheduler DMS Primary Provider get current state lookup in catalog or workspace time get state prediction get time estimate for staging Figure 6.1.: Sequence diagram: Two types of state queries { current state and predicted state The following queries are sent by the Workow Scheduler in order to get information about the current state of data. (See also Figure 6.1) What is the provider of a data object? Input: object identier Output: host name, host type (database, at le or workspace only) What are the les of a data object? Input: object identier Output: list of logical le paths Note that in the case of a database a replica of a le may not exist. Though, there is always a logical le. C3Grid-T Draft 19 / 48

20 What is the name of the metadata le? Input: object identier Output: logical path of metadata le Note that in the case of a database a replica of a metadata le may not exist. Though, there is always a logical metadata le. What are the replicas of a le? Input: le path Output: list of replica URIs In the case of a database no replica may exist for the given le. What is the respective object identier of a le? Input: le path Output: object identier 6.2. Querying State Predictions The following queries by the Workow Scheduler require the prediction of the future state of data in the grid. This prediction is based on knowledge about respective staging and transfer times. Because the DMS knows only about transfer times within the grid, it has to ask data providers for en estimation of the time they need for staging. When can a certain le be available at a specic host? Input: le path, host Output: time stamp What is the earliest time, a specic le can be available? Input: le path, time window of interest, period of lifetime Output: time stamp Exception: availability cannot be guaranteed for the given constraints. What is the earliest time, a specic number of bytes can be available? Input: le path, time window of interest, period of lifetime Output: time stamp Exception: availability cannot be guaranteed for the given constraints. What is the transfer time for a specic le to a specic host? Input: le path, target host, start time. Output: period of time 6.3. Agreements The Workow Scheduler and the DMS meet agreements about the predictions made from the operations described in the use case of Section 6.2. Operations oering agreements are called in two steps. 1. A preliminary request 2. A nal commitment 20 / 48 C3Grid-T Draft