Wissenschaftliches Rechnen an der Christian- Albrechts-Universität zu Kiel

Transkript

1 Wissenschaftliches Rechnen an der Christian- Albrechts-Universität zu Kiel ZKI-Tagung, AK Supercomputing, Holger Marten Landeshauptstadt Kiel - Bodo Quante

2 1. Etwas Historie Dr. Holger Marten 2

3 Gründung des s, Nov Dr. Holger Marten 3

4 Gründung des s, Nov Dr. Holger Marten 4

5 Z22 im Original Z22 im Computermuseum der FH Kiel Dr. Holger Marten 5

6 RZ-Neubau 1963/65 ein MeilenGrundstein Dr. Holger Marten 6

7 Der nächste MeilenGrundstein Dr. Holger Marten 7

8 Neue Maschinenräume Dr. Holger Marten 8

9 Neue Maschinenräume Dr. Holger Marten 9

10 Neue Maschinenräume DECsystem-10 im neuen Maschinenraum Dr. Holger Marten 10

11 Dr. Holger Marten 11

12 Die Cray-Ära der 1980ger/1990ger Jahre Dr. Holger Marten 12

13 2. Das heute Dr. Holger Marten 13

14 Luftbild LMS Dr. Holger Marten 14

15 Organisation des s GF Informationssysteme, Services und Basisdienste Geschäftsführung S. Lorenz, H. Marten GF Hochleistungsrechnen und Digitale Forschungsinfrastruktur PC-Systeme WWW, Informationsdienste, Mail, Anwendersyst. High Performance Computing Server- Systeme und Netzwerk Technik & Operating 44 Personen auf 38,25 Stellen ToDo: Integration der Verwaltungs-IT Campus Management, Campus Card, Lehr-Lern-Systeme, 19 Personen auf 17,25 Stellen Dr. Holger Marten 15

16 IT Governance -Struktur der CAU Senats-Ausschuss für Informationsverarbeitung Vom Senat bestellte Mitglieder Gäste : u.a. Sprecher Beiräte, GF RZ, berät Senat Präsidium gibt Empfehlungen Fach- und Dienstaufsicht Beirat Lehre, Services, Basisdienste (Ltg.: Sprecher/in) berät Geschäftsführung berät Beirat Wissenschaftliches Rechnen, digitale Forschungsinfrastruktur (Ltg.: Sprecher/in) Dr. Holger Marten 16

17 3. Hochleistungsrechnen heute Dr. Holger Marten 17

18 HPC at CAU Traditionally Collaboration between CAU and the former Leibnitz Institute for Marine Science today named GEOMAR Helmholtz Centre for Ocean Research Kiel Vector and parallel computing architectures Codes that profit from vector architectures, memory band width and low latency Today Collaboration with NEC in three components SX-5 SX-9, SX-ACE Scalar, parallel x86 systems File systems gfs,, ScaTeFS Financed 60/40 by GEOMAR/CAU Traditionally also Supporting next higher level (Tier-2) HPC demand on HLRN Covering local entry level demand for scientific computing (rzcluster) Dr. Holger Marten 18

19 Die NEC Vektor-Ära seit 2002 Vektorrechner NEC SX-9 ( ) Dr. Holger Marten 19

20 NEC vector component SX-ACE 256 nodes, each with 1 CPU = 4 vector cores = 256 Gflops peak 64 GB RAM, 256 GB/s memory bandwidth 8 GB/s (x2) IXS interconnect 8 GB/s (x2) file i/o bandwidth Inauguration 4-dec-2014 In sum 256 nodes, 1024 vector cores, 65,5 Tflops, IXS crossbar 16,4 TB mem; 1 GB/s mem bandwidth per Gflops OS: Super-UX Dr. Holger Marten 20

21 NEC scalar (parallel) x86 component 116 nodes x86, each with 2x Intel GHz = 16 cores = 332,8 Gflops peak 128 GB RAM (4 nodes wt 256) InfiniBand FDR In sum 116 nodes, 1856 cores, 38,6 Tflops, InfiniBand 15,1 TB mem OS: Red Hat Dr. Holger Marten 21

22 NEC ScaTeFS In sum TB istorage M300 4 IO server, 8 M300 disk arrays, 6 disk enclosures per disk array IO server = 10 GbE to CPU; FC to controller Scalable Technology Filesystem (ScaTeFS) Dr. Holger Marten 22

23 NEC scalar (parallel) x86 component II 18 nodes x86, each with 2x Intel GHz = 24 cores = 480 Gflops peak 128 GB RAM (DDR MHz) InfiniBand FDR In sum 18 nodes, 432 cores, 8,6 Tflops, InfiniBand 2,3 TB mem OS: Red Hat 6.4 Under construction feb Dr. Holger Marten 23

24 The NEC hybrid HPC environment NQS Master SX-ACE vector job submit x86- Cluster parallel x86- Cluster parallel ScaTeFS CC backup front-end front-end front-end front-end Dr. Holger Marten 24

25 Usage of systems before upgrade NEC SX-9: Utilization per User Group jan-aug Dr. Holger Marten 25

26 Usage of systems before upgrade NEC Cluster: Utilization per User Group jan-aug Dr. Holger Marten 26

27 4. Das HLRN Tier Dr. Holger Marten 27

28 Current, general idea of HPC resources for users in Germany capability Tier-0 European Tier-1 national Tier-2 regional Tier-3 local capacity Dr. Holger Marten 28

29 North-German Supercomputing Alliance Established in North-German states 50% of investment funded by German federal government Dr. Holger Marten 29

30 Technical Commission Representatives of 7 leading computing centers or professorships technical planning & coord. Management Board Representatives of the 7 member states financing & supervision ministry involved 2 Operating Centers LUIS (Hannover) ZIB (Berlin) Scientific Committee 12 leading scientists from scientific domains approval of project proposals involved involved Local operations staff LUIS, ZIB, other centers Domain-specific consultants Distributed; several universities and relevant scientific disciplines involved Support users with proposal application, system usage, application optimization, Dr. Holger Marten 30

31 Hannover MPP system Cray XC30, XC nodes dual Intel Haswell, Ivy Bridge cores 1,2 Pflops 4,2 PB disk 4 PB Quantum i6000 archive SMP system 64 nodes Intel Sandy Bridge + Nvidia K cores 256 / 512 GB RAM per node Berlin MPP system Cray XC30, XC nodes dual Intel Haswell, Ivy Bridge cores 1,4 Pflops 4,2 PB disk 3 SL8500, slots Test and development system Cray XC40 Many-Core 16 nodes Xeon Phi & Xeon KNL Cray XC40 DataWarp I/O Accelerator, 26 TB SSD Dr. Holger Marten 31

32 5. Die rzcluster-umgebung Dr. Holger Marten 32

33 rzcluster at CAU Individual parts financed by several CAU institutes Grown historically 213 nodes, cores 8 different CPU-generations Intel and AMD 3 different networks; DDR, QDR, (FDR) and Gbit Ether Several islands with different RAM-sizes from 8 to 256 GB 534 TB file systems Several hardware generations Several individual file systems Common front-ends, $HOME, OS (but exceptions), Batch (but individual queues) Dr. Holger Marten 33

34 Usage analysis: who? Many more user groups than financiers:: cluster fraction paid by computing center Dr. Holger Marten 34

35 rzcluster issues and new ideas Infrastructure Floor space, electricity and climatization cost need more attention Started energy optimization with water-cooled racks Heterogeneity leads to Inefficient cluster management (OS, application software, ) Respective difficulties for the support team Inefficient usage pattern Started to design a new operations model Huge demand for massive data storage and analysis by bio and medical informatics Design for higher data throughput; prevent data replication!? Further requests for resources in teaching scientific computing, scientific computing under Windows, Big Data, accelerators like Xeon Phi, Nvidia Keppler, Extent of demand? Restricted resources! keep in mind Dr. Holger Marten 35

36 BioFX-Infrastruktur des IKMB G. Hemmrich-Stanisak, Hardware im RZ Dr. Holger Marten 36

37 BioFX-Infrastruktur des IKMB Grobanalyse Hardware im RZ Data Source Experiment Charakteristik: :: [Bandbreite * Messzeit] :: Experiment reproduzierbar / nicht reprod.bar == Preis / Wert Storage Raw Data hohe Bandbreite groß genug (GB/s * Zeit) [ggf. pre-processing zur Datenreduktion] Wert der Daten entscheidet: Backup oder nicht Data Analysis (facility) Storage Compute Part Part Tape Cache Archive Storage Ergebnisse Datenanalysen GGf. inkl. Rohdaten Dr. Holger Marten 37

38 A potential setup for teaching, post processing, data analysis and publication? HPC simulations SX-ACE vector Work (1st level) x86 parallel Experiments 1st level Big Data (2nd level) Indexing, publication, long term, newcluster farm 3rd level Dr. Holger Marten 38

39 Thank you Dr. Holger Marten 39