KNIME HPC INTEGRATION VIA UNICORE

Transkript

1 KNIME HPC INTEGRATION VIA UNICORE *, Florian Jug*, Bernd Schuller, René Jäkel, Gene Myers, Wolfgang E. Nagel

2 OVERVIEW Introduction KNIME UNICORE Data Oriented Processing KNIME HPC Integration via UNICORE Conclusion and Outlook 2

3 OVERVIEW Introduction KNIME UNICORE Data Oriented Processing KNIME HPC Integration via UNICORE Conclusion and Outlook 3

4 INTRODUCTION Varying user data sets and workflows in KNIME Challenge of parallelization and HPC integration KNIME for workflow creation and execution But lacks generic HPC integration UNICORE as middleware for generic HPC access HPC Integration of KNIME via UNICORE 4

5 KNIME HPC INTEGRATION VIA UNICORE Evaluation within image analysis use case In cooperation with MPI-CBG (Jug, Myers) JSC Jülich (Schuller) 5

6 OVERVIEW Introduction KNIME UNICORE Data Oriented Processing KNIME HPC Integration via UNICORE Conclusion and Outlook 6

7 KNIME KNIME The Open Analytics Platform Flexibly analyze data with workflows Re-use of sub-workflows Graphical ease-of-use Widely used in academia and industry 7

8 KNIME 8 V. Na me: Tite l des Vor trag s

9 KNIME 9 V. Na me: Tite l des Vor trag s

10 KNIME 10 V. Na me: Tite l des Vor trag s

11 KNIME 11 V. Na me: Tite l des Vor trag s

12 KNIME 12 V. Na me: Tite l des Vor trag s

13 KNIME 13 V. Na me: Tite l des Vor trag s

14 KNIME 14 V. Na me: Tite l des Vor trag s

15 KNIME 15 V. Na me: Tite l des Vor trag s

16 KNIME 16 V. Na me: Tite l des Vor trag s

17 OVERVIEW Introduction KNIME UNICORE Data Oriented Processing KNIME HPC Integration via UNICORE Conclusion and Outlook 17

18 UNICORE DATA ORIENTED PROCESSING UNICORE monitors directory Pre-defined rule trigger actions for new files Incoming data directly triggers analysis tasks E.g. for preprocessing, metadata extraction, compression, 18

19 UNICORE DATA ORIENTED PROCESSING 19

20 OVERVIEW Introduction KNIME UNICORE Data Oriented Processing KNIME HPC Integration via UNICORE Conclusion and Outlook 20

21 KNIME HPC INTEGRATION VIA UNICORE Configuration of UNICORE DOP feature for remote directory User graphically exports workflow in KNIME to directory Automatic execution of workflow on HPC system Existing data sets on cluster extended with results User result access via directory Easy to use 21

22 KNIME HPC INTEGRATION VIA UNICORE 22

23 KNIME HPC INTEGRATION VIA UNICORE Mean processing time and speedup with varying # of threads Balance at about 8 threads 23

24 KNIME HPC INTEGRATION VIA UNICORE UNICORE induced overhead per workflow execution Average of 27s out of 10 repetitions 24

25 KNIME HPC INTEGRATION VIA UNICORE Measurements with up to 1,76 TB in 7,488 M files Overall runtime previously 17d - now 2h 400x faster 25

26 OVERVIEW Introduction KNIME UNICORE Data Oriented Processing KNIME HPC Integration via UNICORE Conclusion and Outlook 26

27 CONCLUSION Challenge of parallelization and HPC integration KNIME as widely used workflow system UNICORE for HPC and Big Data capabilities Integration of both KNIME now HPC-enabled in generic and easy way 27

28 THANK YOU FOR YOUR ATTENTION! CONTACT Dr. Technische Universität Dresden Zentrum für Informationsdienste und Hochleistungsrechnen Dresden Telefon:

29 CONTACT SCIENTIFIC COORDINATOR Prof. Dr. Wolfgang E. Nagel Technische Universität Dresden Zentrum für Informationsdienste und Hochleistungsrechnen Dresden Telefon: DEPUTY SCIENTIFIC COORDINATOR Prof. Dr. Erhard Rahm Universität Leipzig Fakultät für Mathematik und Informatik Augustusplatz Leipzig Telefon: MANAGEMENT DIRECTOR Dr. René Jäkel Technische Universität Dresden Zentrum für Informationsdienste und Hochleistungsrechnen Dresden Telefon: