GT4: An introductory overview

Transkript

1 4: An introductory overview Institut für Wissenschaftliches Rechnen Olaf Schneider using material of the lobus Alliance and the National e-science Centre, Edinburgh ridka-school 2006, September

2 Outline introduction and basic concepts What is 4? History SOA, Web s and WSRF 4 components and architecture overview main functions of a grid system WS components versus Pre-WS components some details on selected components a typical use case interoperation architecture Veranstaltung Datum 2

3 What is 4? 4 is lobus oolkit Version 4 "he lobus oolkit is an open source software toolkit used for building grids." [ 4 is a WSRF compliant application development framework. 4 is collection of ready-to-use of high-level rid services. Veranstaltung Datum 3

4 History first ideas in the ealier 90s (metacomputing, I-WAY project) 1997 "lobus: a Metacomputing Infrastructure oolkit" (Forster, Kesselman) (used in glite) (OSI) (WSRF) (current stable release) Veranstaltung Datum 4

5 What 4 provides? collection of solutions, which frequently comes up when trying to build collaborative distributed applications impementation of standards: use of existing standards (IEF, W3C, OASIS, F) provide reference implementations of new/proposed standards a set of services focused on infrastructure management common interfaces to heterogeneous resources standard-based security infrastructure client APIs and command line programs for accessing services a toolkit for building new Web s service-oriented architectures (SOA) Veranstaltung Datum 5

6 Oriented Architecture (SOA) " Oriented Architecture is a paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired effects consistent with measurable preconditions and expectations." (OASIS) SOAs are systems and applications structured as communicating services with uniform ways for service describtion invocation secure access Veranstaltung Datum 6

7 What are Web s? A Web is a software system (programm, agent): interoperable over a network (WAN), typical via HP interface discribed in a machineprocessable format (WSDL) interaction with the WS using SOAP messages (request/response scheme) characteristics of the WS architectur: family of protocols (which heavily use XML) machine-to-machine interaction mechanisms for service registry and discovery (UDDI) classical WS are stateless, provide only functionality A Web is not: a Web site or portal Veranstaltung Datum 7

8 Web Architecture examples of existing implementations: WS Container Jakarta omcat, 4 container SOAP engine Apache AXIS Web server Apache HP Server Veranstaltung Datum 8

9 rid s as Web s stateless services, but stateful resources factory and instance service persistent state life cycle (create, destroy) properties (accessed via a stateless WS) notification (on state changes) standardised in the Web Resource Framework WS-Resource WS-ResourceProperties WS-Resourceroup WS-ResourceLifetime WS-BaseFaults convergence into common WS specifications Veranstaltung Datum 9

10 Modeling State: Factory and Instance Resource Allocation Discovery Register Stateful Entity Registry, e.g. JNDI WS Container Create Stateful Entity End Point Reference Factory requestor (user application, WS client) State inspection, Invocation Lifetime mgmt, Notifications Instance Stateful Entities Veranstaltung Datum 10

11 Outline introduction and basic concepts What is 4? History SOA, Web s and WSRF 4 components and architecture overview main functions of a grid system WS components versus Pre-WS components some details on selected components a typical use case interoperation architecture Veranstaltung Datum 11

12 Basic Functions of a rid System Security Authentication and Certificate Data access and move data replica management storage brokering Execution submitting and monitoring jobs manage execution environments (sandboxes) job brokering Information s resource and service discovery monitoring the grid Veranstaltung Datum 12

13 Security Delegation Community (CAS) WS Authentication & Pre-WS Authentication & Credential SI-OpenSSH Data Data Replication (DRS) [ech Preview] OSA-DAI Reliable File ransfer (RF) ridfp Replica Location (RLS) 4 components Execution CP [ech Preview] WMS Community Scheduler Framework rid Resource Allocation Mgmt (WS RAM) Pre-WS RAM Information s WebMDS rigger Index Monitoring & Discovery System (MDS2) Common Runtime Python WS Core C WS Core Java WS Core C Common Libraries extensable IO (XIO) Veranstaltung Datum 13 Web s Components Non-WS Components

14 Common Runtime XIO: extensible input/output library, provides a single API for multiple protocols (CP, UDP, HP, SI_FP, SSAPI_FP, ELNE,...) WS core libraries support both lobus services (RAM, RF, Delegation, etc.) & user-developed services Leverages existing WS standards WS-I Basic Profile: WSDL, SOAP, etc. WS-Security, WS-Addressing Adds support for emerging WS standards WS-Resource Framework, WS-Notification Java, Python, & C hosting environments Veranstaltung Datum 14

15 4 Web Core User Applications 4 Container Custom Web s Custom WSRF Web s 4 WSRF Web s WS-Addressing, WSRF, WS-Notification WSDL, SOAP, WS-Security Registry Administration Veranstaltung Datum 15

16 4 Web Container to start all Web s (standard and user deployed), just start the container the WS container can run in two security modes non-secured (http), standard port 8080 SSL secured (https), standard port 8443 substitution by omcat possible (but has some drawbacks) Veranstaltung Datum 16

17 Security Delegation Community (CAS) WS Authentication & Pre-WS Authentication & Credential SI-OpenSSH Data Data Replication (DRS) [ech Preview] OSA-DAI Reliable File ransfer (RF) ridfp Replica Location (RLS) 4 security Execution CP [ech Preview] WMS Community Scheduler Framework rid Resource Allocation Mgmt (WS RAM) Pre-WS RAM Information s WebMDS rigger Index Monitoring & Discovery System (MDS2) Common Runtime Python WS Core C WS Core Java WS Core C Common Libraries extensable IO (XIO) Veranstaltung Datum 17 Web s Components Non-WS Components

18 4 security Secured WS container delegation service needed for service orchastration classical command line tools grid-cert-request, grid-proxy-init, uses rid-mapfile statical mapping of cert. subjects onto UNIX accounts VOMS integration only as ech. preview Credential management services MyProxy (Web portal integration!) SimpleCA (poor man's credential generation) Community SI-OpenSSH interactive shell access via rid credentials Veranstaltung Datum 18

19 Security Delegation Community (CAS) WS Authentication & Pre-WS Authentication & Credential SI-OpenSSH Information s Data Data Replication (DRS) [ech Preview] OSA-DAI Reliable File ransfer (RF) ridfp Replica Location (RLS) Execution CP [ech Preview] WMS Community Scheduler Framework rid Resource Allocation Mgmt (WS RAM) Pre-WS RAM Information s WebMDS rigger Index Monitoring & Discovery System (MDS2) Common Runtime Python WS Core C WS Core Java WS Core C Common Libraries extensable IO (XIO) Veranstaltung Datum 19 Web s Components Non-WS Components

20 Information s 2 Monitoring and Discovery is depricated. new WS-based implementation: Index aggregator service collects information from registered sources to build an hierarchie of indexes (upstream or downstream) automated registration of all services in the 4 container custom Ressource Property provider for non-ws resorces like batch systems (PBS, LSF, SE) or monitoring tools (anglia) access via Xpath queries (XML mapped LUE schema) rigger perform action on condition (event driven) WebMDS browser-based client for the index adjustable via XSL Veranstaltung Datum 20

21 Security Delegation Community (CAS) WS Authentication & Pre-WS Authentication & Credential SI-OpenSSH Data Data Data Replication (DRS) [ech Preview] OSA-DAI Reliable File ransfer (RF) ridfp Replica Location (RLS) Execution CP [ech Preview] WMS Community Scheduler Framework rid Resource Allocation Mgmt (WS RAM) Pre-WS RAM Information s WebMDS rigger Index Monitoring & Discovery System (MDS2) Common Runtime Python WS Core C WS Core Java WS Core C Common Libraries extensable IO (XIO) Veranstaltung Datum 21 Web s Components Non-WS Components

22 Data 4 ridfp is a reimplementation of the older versions 100% backward compatible no licensing issues access to data sources via XIO Reliable File ransfer (RF) WS interface for ridfp only ridfp, no data transfer via SOAP messages this requires an running ridfp server with open ports on every host in the grid recovery after broken transfer OSA-DAI (data access interface) provide service-based access to structured data resources Relational: MySQL, Oracle, DB2, SQL Server, Postgres XML: Xindice, exist Veranstaltung Datum 22

23 Security Delegation Community (CAS) WS Authentication & Pre-WS Authentication & Credential SI-OpenSSH Execution Data Data Replication (DRS) [ech Preview] OSA-DAI Reliable File ransfer (RF) ridfp Replica Location (RLS) Execution CP [ech Preview] WMS Community Scheduler Framework rid Resource Allocation Mgmt (WS RAM) Pre-WS RAM Information s WebMDS rigger Index Monitoring & Discovery System (MDS2) Common Runtime Python WS Core C WS Core Java WS Core C Common Libraries extensable IO (XIO) Veranstaltung Datum 23 Web s Components Non-WS Components

24 WS-RAM details on next slide Execution Job Broker is not part of core 4 several projects in the community with potential to fill this gap the Community Scheduler Framework is one of them, but heavily biased to LSF other ech. previews rid elecontrol Protocol (CP) service for manageing instruments like microscopes Workspace (WMS) dynamic allocation of UNIX accounts (sandbox) vitual machine support is prototyped Veranstaltung Datum 24

25 WS-RAM independend of older 2 RAM provides a common WS interface to schedulers UNIX-Fork, PBS, Condor, SE, LSF, LoadLeveler the scheduler actually wanted is transmitted to the WS-RAM Factory as a parameter called Factoryype Job describtion in XML Ressource Specification Language (RSL) uses Delegation for credential management uses RF (and ridfp) for data staging and streaming output command line client available but no UI this lack will probably filled by Web Portal Fremworks like ridshere Veranstaltung Datum 25

26 WS-RAM functions ClientJob Delegate 4 Java Container RAM services Delegation RF File ransfer ransfer request Job events Local job control Delegate FP control sudo SE RAM adapter ridfp ridfp FP data Compute element Local scheduler User job Remote storage element(s) Veranstaltung Datum 26

27 A typical usecase for 4 situation: aim you are running a compute cluster with several nodes jobs are managed by a LRMS like PBS, Condor, SE, shared file system over all nodes for user home directories cluster management and user admin is done on a head node integrate your cluster as computing resource in a rid steps install and configure 4 on cluster head node configure a WS-RAM Factoryype for your batch system start 4 container and ridfp-server on the head node get the Cert Subjects of all rid users and add them to the rid-mapfile optionally allow access via SI-OpenSSH to the head node register your MDS-Index- into the global Index of your rid Veranstaltung Datum 27

28 4 Components Architecture CLIEN Your Your Java Java Client Client Your Your C Client Client Your Your Python Client Client Your Your Java Java Client Client Your Your C Client Client Your Your Python Client Client Interoperable WS-I-compliant SOAP messaging X.509 credentials = common authentication Your Your Java Java RAM RF Delegation Index rigger WMS CAS OSA-DAI CP Your Python pylobus WS Core Your C C WS Core ridfp SimpleCA MyProxy RLS Pre-WS RAM Pre-WS MDS SERVER Java s in Apache Axis Plus Libraries and Handlers Python hosting, Libraries C s using Libraries and Handlers [I. Forster: A lobus Primer] Veranstaltung Datum 28

29 Further Readings see also I. Forster A lobus Primer lobus oolkit 4: Software for -Oriented Systems NPC2005, LNCS 3779, pp. 2-13, B. Sotomayor he lobus oolkit 4 Programmer's utorial Veranstaltung Datum 29