Stratosphere. Next-Generation Big Data Analytics Made in Germany

Transkript

1 Stratosphere Next-Generation Big Data Analytics Made in Germany Robert Metzger Stratosphere Core Developer Technische Universität Berlin Ronald Fromm Head of Big Data Science Telekom Innovation Laboratories BITKOM Big Data Summit, 26. März 2014, Hanau

2 Was ist Big Data? Quelle: Management von Big-Data-Projekten, Leitfaden, BITKOM 2013, Seite 10

3 Neue Systeme ermöglichen mehr Anwendungsfälle Analysen gehen über einfache SQL Abfragen hinaus. Ergebnisse werden in Minuten erwartet Twitter Community Analyse: Von 10 Stunden auf 10 Minuten. Besonders bei Advanced Analytics 200 sind Verbesserungen von Tagen zu Minuten möglich! Hadoop Stratosphere

4 Was kommt nach MapReduce? What the Hadoop ecosystem needs is a successor system that is more powerful, more flexible and more realtime than MapReduce. new applications could use such a generalpurpose engine to be faster and to do more than is possible with MapReduce. Mike Olson, CSO Cloudera

5 Einführung Stratosphere Stratosphere wurde von Grund auf als General Purpose Data Analytics Platform neu entwickelt. RDBMS Hadoop MapReduce Stratosphere Deklarativität (SQL) Optimierer Pipelining Effiziente Runtime Iterationen Komplexe Datenflüsse Code Analyse Deklarativität (Scala) Skalierbarkeit Benutzerdefinierte Funktionen (UDFs) Komplexe Datentypen HDFS Stratosphere bringt Datenbank-Technologie in die Welt von Big Data.

6 Stratosphere Big Picture Data Mining Netzwerkanalysen Relationale Abfragen ETL / Data Integration Sensordaten Logdaten Mobile Sales Data Market Data RFID GPS Archive Open Data... Stratosphere Hadoop Storage Layer Cluster of Commodity Hardware/ Aus dem Regal Bestehende Datenbank

7 Stratosphere System Stack Operatoren Konnektoren Stratosphere integriert sich in bestehende Big Data Infrastrukturen

8 Kooperationen und Partnerschaften Stratosphere ist eine preisgekrönte Open-Source (Apache Lizenz) Plattform. 150k Lines of code, 5m Forschungsetat Stratosphere ist das einzige Hadoop-kompatible Big Data Analytics System, das in Deutschland und Europa entwickelt wird und fertig zum Download bereit steht!

9 Telekom Innovation Laboratories High Quality and High Impact Innovation 9

10 Telekom Innovation Laboratories Project Big Data Analytics Workbench Task: Evaluate common Big Data frameworks Implement real scenarios Offer prototypes for urgent problems Infrastructure: Two nodes with 16 CPUs 2,2 GHz 128 GB RAM 12 x 3 TB HDD Benchmarks and Blueprints Telekom Innovation Laboratories Strategy Enhancements with OpenStack Support Challenge Disrupt Impact business Hands on productization Look beyond the horizon Drive markets & technologies Find the hidden Jokers Shape industries 10

11 Telekom Innovation Laboratories Risk Management - Real World Use Case The task of DTAG Risk Management Center (RMC) is to mitigate financial risks from the company. Amongst others, RiBS@RMC (risk scoring systems) is one of the components, that calculates a set of indicators on selected customer data on a daily basis. Definition of 20 key score values to be implemented with Big Data technology A partial copy of all required data, completely anonymized (and confirmed by Internal Security Authority - GIS) 11

12 Backup 12

13 Telekom Innovation Laboratories Big Data Analytics Workbench - Testing Environment Transform all input data to a compressed, serialized format Implement key scores for several frameworks, using the same AVRO schemes from HDFS Every use case and framework creates a separate file, containing all customers with their scores Output is transfered to a Hbase data store for comparative purposes and statistical evaluations 13

14 Telekom Innovation Laboratories Other Big Data Activities ( ) Big Data Analytics Plasticine enables workbench users to provision and ma-nage big data architectures like Hadoop, Stratosphere or HBase via click. It utilizes our OpenStack cloud infrastruc-ture to dynamically provision clusters of any size and number with a rich set of selectable framework flavours. It will be used to determine the scalability of a developed prototypes as well as for finding out the right number and size of a target architecture with tight time limits. 14

15 Starten Sie heute in die Zukunft! Probieren Sie Stratosphere aus: Kooperieren Sie mit uns für Proof-of-Concepts und Use Cases Hands-on Experience in der Deutschen Telekom AG Stratosphere Core Developer Robert Metzger TU Berlin, Database Research Group Phone: