TSM and dcache in grid projects at Forschungszentrum

Transkript

1 TSM and dcache in grid projects at Forschungszentrum Karlsruhe, Germany Institute for Scientific Computing () Autor: Dorin Daniel Lobontu Jos van Wezel Doris Ressmann 1

2 Table of Contents Overview: Research Centre Karlsruhe (FZK) and Institute for Scientific Computing () Grid Data Management: FTS, SRM and GridFTP Gridka Storage Management: dcache GridKa Online-Storage Concepts Online Storage Connection to TSM 2

3 Forschungszentrum Karlsruhe One of the biggest science and engineering research hinstitution in Europe with 3800 employees 5 research areas: Structure of matter Earth and environment Health Energy Key technologies - Nanotechnology - Grid Computing 3

4 Institute for Scientific Computing Forschungszentrum s Computer Centre German Grid Computing Centre Karlsruhe (GridKa) Research projects on grid computing area: EGEE/GGUS Grid infrastructure for e-science with 24 hours-a-day availability Global Grid-User-Support CampusGrid builds a heterogeneous grid platform D-Grid builds a German grid infrastructure Knowledge Management D-Grid e-learning e-science Initiative 4

5 RZ Uni KA Steinbuch Computing Centre (SCC) / FZK 5

6 The LHC multi-tier Computing Model The main goal of GridKa cluster at the moment is to undertake, process and store the data from LHC (Large Hadron Collider) detectors Tier 4 (Desktop) Tier 3 (Institute computer) γ Lab y Tier 2 (Uni-CCs, Lab-CCs) β Uni e ATLAS g virtual organizations Lab x Lab z CERN Tier 1 UK (RAL) USA (Fermi, BNL) France (IN2P3) Tier 1 Italy (CNAF) Uni d CE RN. Tier 0 LHCb. Germany (FZK) Uni c Uni a Lab i working groups CMS Uni b Tier 0 Centre at CERN 6

7 LHC-Data Challenge 2007 Amount of data per year: 100 MB per Sec. x ~200 days of experiment per year = 8PBd data per year + 4 PB simulated data per year 12 PB final data 1 PB = GB Idea: World Wide Grid (WWG) Comparison: When 1 Byte =. is is 1 PB. 1 Second 32 million years 1 Sheet of paper a 100 millions km high pile of paper 7

8 Grid-Computing at a glance Notebooks Desktops V I R T U A L R E S O U R C E S Tape-Storage Supercomputer Disk-Storage Users Middleware Resources Software 8

9 Table of Contents Overview: Research Centre Karlsruhe (FZK) and Institute for Scientific Computing () Grid Data Management: FTS, SRM and GridFTP Gridka Storage Management: dcache GridKa Online-Storage Concepts Online Storage Connection to TSM 9

10 Transfer Hierarchy GridFTP Transfer protocol low level protocol moves data between machines has short timeouts SRM waits to start transfer (e.g. to get file from tape) - starts connection between physical nodes FTS management of transfer channels constant bandwidth for each channel error handling (new transfer if previous failed) 10

11 File Transfer Service FTS Provides point to point movement of physical files: Provides reliable data transfer between sites (very robust, allowing breaks in connectivity and restart of the service) Offers the sites the ability to manage storage data flow (IN and OUT) Provides the state of transfer Does not deal with GUID (Global Unique ID), LFN (Logical File Name) Components: An Oracle database to hold the state A transfer server to run the transfer agents (assigning jobs to channels, responsible for running the transfers) An application server (submission and monitoring portal) 11

12 Storage Resource Manager (SRM) Manages access to shared storage resources on the Grid Uniform access to heterogeneous storage File transfer protocol negotiation Dynamic transfer URL allocation Advanced space and file reservation Access to permanent and temporary types of storage File types: Volatile - temporary & sharable copy of a MSS resident file if not pinned it can be removed by the garbage collector Durable - file cannot be removed automatically. If space is needed file may be copied to MSS Permanent - system cannot remove file 12

13 Table of Contents Overview: Reasearch Centre Karlsruhe (FZK) and Institute for Scientific Computing () Grid Data Management: FTS, SRM and GridFTP Gridka Storage Management: dcache GridKa Online-Storage concepts Online Storage Connection to TSM 13

14 What is dcache? Disk pool manager (with/without t backend Tape-Storage) Full Storage Resource Manager (SRM) support Variety of data access protocols (dcap, xroot, gridftp) Data may be distributed among a huge amount of disk servers Automatic load balancing by cost metric and inter disk pool transfers Data removed only if space is needed Allows more copies of a file for distributed data access Fine grained configuration of pool attraction scheme 14

15 dcache Environment compute nodes / login server gridftp / mountpoint file transfer head node srm pools tsm server file transfer TSM with tape s 15

16 dcache Tape Access External shell script or executable - has to be defined d for every individual pool Syntax : <binary_name> put get <pnfsid> <localfilename> \ -si=<storageinfo> s o [more options] o Multiple and different tape systems are supported Transparent access Separation between administration domains (dcache and TSM) 16

17 Table of contents Overview: Reasearch Centre Karlsruhe (FZK) and Institute for Scientific Computing () Grid Data Management: FTS, SRM and GridFTP Gridka Storage Management: dcache GridKa Online-Storage concepts Online Storage Connection to TSM 17

18 Previous Online Storage Architecture (~240TB) Worker nodes GPFS / NFS servers / dcache SAN Switch (director) Disk Collection 18

19 Storage Area Network Advantages Easier administration: i ti all storages can be connected to all file servers Expansion or exchange during production: reduces the down time Load balancing / redundancy: several paths to same storage Very fast: 4Gb/s (400MB/s) Scalable: add switches and controllers Small overhead and CPU load: HBA handles protocol Disadvantages Expensive: 1000$ per HBA or Port price will drop with iscsi a (S)ATA + IP solution would need more controllers (host) and a separate LAN No standardized (fabric) management 19

20 Online-Storage at GridKa > 4000TB Storage Units(SU) -no SAN fabric, still flexible -lower cost -easier maintenance -2 storage units per rack -storage units of 20TB -10 disk drawers per storage unit -2 servers connected to a storage unit -dcache pool node on GPFS file system - can function as GPFS IO node 20

21 Connection to tape dcache TSS TSM storage agent GPFS 21

22 Table of contents Overview: Reasearch Centre Karlsruhe (FZK) and Institute for Scientific Computing () GridKa Online-Storage concepts Grid Storage Integration: FTS, SRM and GridFTP Gridka Storage Management: dcache Online Storage Connection to TSM 22

23 dcache layout at GridKa GridKa Network workernodes Ethernet FC CE ERN und Tier1/T Tier2 Ce entres Read Pools Input buffer Write pools StorageAgents Read pool StorageAgent File hopping Read/Write pools StorageAgents Tapes Pools TSM Server Disk only CE RN und Tier1/T Tier2 Ce entres Constant t stream 23

24 dcache -TSM connection possible scenarios I dcachepool dcachepool dcachepool LAN TSMServer FC Tape Libraries dcachepool dcachepool dcache pool nodes call an external program to connect to TSMServer TSMServer writes/reads files to/from tape Data flow over LAN (can be slowly) TSM Server needs high I/O rate (can be overloaded bottleneck) NOT SCALABLE 24

25 dcache -TSM connection possible scenarios II dcachepool StorageAgent LAN TSMServer Tape Libraries dcachepool StorageAgent dcachepool FC StorageAgent t dcachepool StorageAgent dcache pool nodes need only a metadata connection to TSM Server via LAN Data flow over fibre channel via Storage Agents TSM Server keeps the database and manages the Tape Libraries This solution scales well with dcache pool nodes and tape drives but 25

26 Problem: dcache -TSM connection possible scenarios II The storage agents share the tapes and the TSM server is not able to keep a tape mounted until it is full => unnecessarily mount/unmount actions Adding more storage agents increases the number of unnecessarily mount/unmount actions => the throughput is affected Solution: A new dcache-tsm interface TSM Session Server developed d by GridKa in collaboration with the dcache team Further information in the talk TSS, an interface to TSM for storage-management middleware by Jos van Wezel 26

27 Data Flow 27

28 TSM Considerations dcache has a load balancing mechanism => files may be stored from one node and retrieved from another one =>all nodes must be able to access all files =>the files must be stored under a TSM-Proxy node - MAXNUMMP must be set appropriately SAN device mapping is very useful uses SNIA HBA API to perform SAN discovery supports pages for TSM and vendor must be checked persistent binding is no longer required since June 2007 x86_64 is supported 28

29 TSM Considerations continued TSM DEFine DRIVE/PATH with autodetect=yes is mandatory DEFine LIBRary LIB_NAME SERial=AUTODetect DEFine PATH AUTODetect=yes DEVIce=xxx DEFine DRive LIB_NAME DR_NAME SERial=AUTODetect ELEMent=AUTOdetect DEFine PATH AUTODetect=yes DEVice=xxx Performance issues to consider determine number of tape drives per storage agent (per HBA) determine number of concurrent tape transfer at full speed fibre channel speed versus tape drive speed keep drive streaming tape as optimally as possible (TSS issue) consider when is TSM database expected to be a bottleneck? 29

30 Tape-Libraries used as tape backend 2xIBM 3584 Library (TS3500): 5 Logical Libraries Drives: 8LTO1, 16LTO2, 8LTO3 5TS1120 Current data volume: ~1,7PB Grau ITL-XL - up to 52drives Issues: - up to 5376 slots serial drive and element address autodetect didn t work the problem was solved in cooperation with the IBM and Grau developer s teams 30

31 Planning Numbers Tape (TB) Disk (TB) CPU (# cores)

32 Conclusions The performance of a system does not only depend on the hardware, but also on the architecture t of the application The throughput of a TSM server is limited by the I/O rate of the machine Using storage agents improves the throughput data flow passes through fibre channel the system is more scalable, but still not enough adding more storage agents increases mount/unmount actions unnecessarily mount/unmount actions downgrade the performance of the system A new DCache-TSM interface had to be developed TSS (TSM Session Server) reduces the mount/unmount actions TSS keeps the drives streaming the system scales very well 32

33 33