SuperMUC: Energieeffizientes Supercomputing

Transkript

1 Torsten Bloth, HPC Systems Architect - IBM Deep Computing SuperMUC: Energieeffizientes Supercomputing ZKI AK Supercomputing, May 2011 DESY, Zeuthen

2 SuperMUC in Germany and Europe The Leibniz Computing Center of the Bawarian Academy of Science is a member of the Gauss Centre for Supercomputing. GCS is the German Federal Computing center with the three sites at LRZ in Garching FZJ in Jülich HLRS in Stuttgart GCS is a member of PRACE (Partnership for Advanced Computing in Europe) With 21 member states in Europe And four Hosting-Partners: D, F, I, SP commiting to fund 100 Mio EUR for 5 years Access to SuperMUC will be regulated by the PRACE Scientific Steering Committee (with peer review process) GCS Steering Comitee

3 LRZ Power Consumption over the Years Power Consumption in MWh Source: LRZ

4 LRZ Major Goals Build and operate a Tier-0 multipetaflops supercomputer for european researchers Serve computational power to German universities and research institutions Accommodate a wide range of applications w/o porting code to a properitary architecture a highly scalable system software stack and programming environment Deliver an higly energy efficient system, reduce operational costs following this, the primary goal for the procurement of the petascale computer is: Source: LRZ

5 LRZ Objective for SuperMUC To establish an integrated, highly energy efficient system and a programming environment which enable the solution of the most challenging scientific problems from widely varying scientific areas. * * Ref [1] RfP, Description of Goods and Services for the European High Performance Computer SuperMUC at LRZ, 23. September 2010, Page 1 6

6 LRZ Procurement Source: LRZ

7 LRZ - Extension to the Facilities 8 Source: LRZ

8 SuperMUC Executive Summary Next PetaFLOP Computer in Germany in Gauss-Center 9324 Nodes with 2 Intel SandyBridge-EP CPUs 207 Nodes with 4 Intel Westmere-EX CPUs 9531 Nodes with total CPUs / Cores 3 PetaFLOP/s Peak Performance 324 TB Memory InfiniBand FDR10 Interconnect ~ (optical) IB cables Large File Space for multiple purpose 10 PetaByte File Space based on IBM GPFS with 200GB/s aggregated I/O Bandwidth 2 PetaByte NAS Storage with 10GB/s aggregated I/O Bandwidth No GPGPUs or other Accelerator Technology Innovative Technology for Energy Effective Computing, established as HW and SW Warm Water Cooling Energy Aware Scheduling with xcat and LL Highly Energy Efficient High End HPC System PUE 1.1 Linux powered

9 SuperMUC Proposed Architecture Dense HPC Cluster with QDR/FDR10 Infiniband Interconnect Island configuration of minimal 8K cores Scalable for up to 36 islands IB blocking ratio within islands of 1:1 IB blocking ratio between islands of 1:4 in phase 1 18 thin compute node islands 1 island as migration island and later used as fat compute island 1 island with GPFS I/O and services and login nodes Fully warm water cooled system for thin compute nodes to support free cooling Energy aware software stack, with xcat and LoadLeveler

10 SuperMUC Nodes thin node design Based on IBM idataplex 2 x Sandy Bridge-EP 8C 2GB per core, 1600MHz InfiniBand FDR, PCIgen3 diskless fat node design native 4-socket WM-EX IBM HX5 Blade 4 x WM-EX 10Core, 2.40GHz 4GB/core InfiniBand QDR diskless

11 SuperMUC FatTree Interconnect I/O Island Fat Node Island to LRZ backbone GPFS log I/O 72 I/O 1 m f 02 f 01 Infiniband Infiniband x 126 x 126 InfiniBand FDR10 ( port spine switches) x 126 Infiniband Infiniband Infiniband node 1 node 2... node 518 node 1 node 2... node node 1 node 2... node 518 thin island #1 thin island #2 thin island #18

12 Goal of Energy Aware Scheduling with LoadLeveler Provide an energy report for all jobs Power(i), Time(i), Energy(i) {, GFlops(i)} Stored in xcat/loadleveler DB When the same job is submitted a second time Change the frequency of the job to match a given energy policy defined by the Sys Admin

13 Energy Aware LoadLeveler Features 1. Goals Identify idle nodes in the cluster and put them in the lowest power mode Provide to system admins query capability on historical usage of power and energy by workload, user, etc. Reduction of energy consumption on workloads with minimal impact to performance 2. Choices for system admin: Decide to use Energy Optimize policy or not Decide the max power/performance/energy degradation one application will be impacted by, if the Energy policy is applied 3. If Energy Policy is on, policy is applied only to jobs that match the performance degradation criteria 4. System admin can query LoadLeveler DB to evaluate the impact of the potential policy on performance degradation and energy saving

14 IBM xcat and LoadLeveler Power and Energy Aware Goals xcat Cluster Systems Management Suite Manage power consumption on an ad hoc (time) basis For example, while cluster is being installed, or when there is high power consumption in other parts of the lab for a period of time Query: p-state, frequency, power consumed info, CPU usage, fan speed, environment temperature Set: p-state, idle mode LoadLeveler Report power and energy consumption per job Energy report Optimize power and energy consumption: When nodes are idle Energy Report Set nodes to the least power consumption mode : C6 state or Shutdown When nodes are running jobs xcat -LL DB

15 Aquasar A Prototype of Hot Water cooling Design and build a hot-water cooling for the - IBM BladeCenter H - QS22 blade (IBM Cell/B.E.) - HS22 blade (Intel, x86_64) The Design Goal and Challenge - Build water cooling in BC-H Blades, which had been designed for air cooling only. - Design a hot-water cooling for world record low emission in data centers (MFlops/gCO 2 ). - Achieve a high energy efficiency of Aquasar (MFlops/W) The project was funded by the ETH Zurich and by IBM Research.

16 Aquasar Vision: The Zero-Emission-Datacenter o C o C >95% recovered heat

17 CAD Model of the Aquasar Water Cooled Blade HS22

18 Hot-Water Cooled Blade HS22

19 LRZ Water Cooled Node HOT AND COLD MANIFOLDS ATTACHED TO RACK QUICK DISCONNECT COUPLERS RACK NODE IN RACK

20