Infrastructures for BIG DATA. Ulf Leser

Transkript

1 Infrastructures for BIG DATA Ulf Leser

2 Pennisi, E. (2011). Science Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/2015 2

3 Scary Trends Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/2015 3

4 Large Data Producer 12+ TBs of tweet data every day 30 billion RFID tags today (1.3B in 2005) 4.6 billion camera phones world wide? TBs of data every day 25+ TBs of log data every day 76 million smart meters in M by s of million s of GPS enable d devices sold annually 2+ billion people on the Web by end 2011 K. Kannan, IBM Research Labs, 2013 Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/2015 4

5 Data and What to do with it Web Logs, click stream analysis ecommerce transactions Genome sequencing Smart meter, smart home Traffic surveillance, car-2-car High energy physics, astronomy Retail data warehouses Advertisement, personalized search results recommendation engines Personalized medicine Reduced energy consumption, improved resource planning Less congestion, improved maintenance Basic research Better supply chain management, targeted advertisements Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/2015 5

6 Fact or Fiction Web Logs, click stream analysis ecommerce transactions Genome sequencing Smart meter, smart home Traffic surveillance, car-2-car High energy physics, astronomy Retail data warehouses Advertisement, personalized search results recommendation engines Personalized medicine Reduced energy consumption, improved resource planning Less congestion, improved maintenance Basic research Better supply chain management, targeted advertisements Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/2015 6

7 Biggest Issue Data Privacy, legislation Completely ignored in most of the seminar Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/2015 7

8 What is Big Data? Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/2015 8

9 What is the Data? Log files (IT systems, web) Sensor readouts, measurements (smart X, science) Transactions (ecommerce, retail) Images & video (satellite, street-view, smart phones) Text (Web, publications, twitter) Graphs (Social web, web, networks) Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/2015 9

10 Example: SNV Detection Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

11 Details 100 million reads Quality estimation Quality filtering Quality metrics Functional assessment, GWAS, Read mapper 1 Union Local realign Read mapper 2 Unmapped reads SNV filtering Union Quality adaptation Pileup SNV assessment Cross-species search DB 1 DB 2 Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

12 GeneView Thomas et al. (2013). Experiences from Developing the Domain-Specific Entity Search Engine GeneView. BTW. Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

13 Need for Speed UDF-heavy: Very few selection, projection, Processing step Time [Min] HTML conversion 528 Sentence detection 280 Parsing 100,437 Total 165,372 min ~ 114 days 21M citations, 12M abstracts, 400K full texts Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

14 Who is involved? Distributed Systems: We need to build IT systems on large clusters of commodity hardware Scheduling, distributed file systems, fault tolerance, cloud computing, parallelization Databases: We need to build systems that efficiently can execute complex queries on very large datasets Parallel databases, query optimization, shared-nothing architectures, user defined functions, NoSQL, Machine Learning: We need to build algorithms that can learn from very large and heterogeneous data Classification, pattern mining, forecasting, recommendation engines, unsupervised learning, Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

15 Data Science Source: Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

16 Infrastructures for BIG DATA Conventional approaches reach their limits Scalability and through-put Languages and methods to specify complex analytics workflows Algorithms for analyzing large and heterogeneous data sets Infrastructures for Big Data Scale gracefully with rapidly increasing data sets Declarative languages, rich and extensible operator set Support for workflow specification, execution, monitoring, Management of distributed, elastic, virtualized resources Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

17 Who should be here Master Informatik, [Diplominformatik] Ability to read English papers Good knowledge in databases Cost models, optimization, index structures, Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

18 How it will work Today: Introduction and choice of topics Meet advisor by to discuss topic and papers Present topic in 5min flash-presentation before Christmas Meet your advisor by to discuss slides Present your topic (30-40min) at the Blockseminar Write seminar thesis (~15 pages) by Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

19 ToC Introduction Topics Assignment Hints on presenting your topic and writing your thesis Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

20 General Literature To be read by everyone! Dean, Jeffrey, and Sanjay Ghemawat. "MapReduce: simplified data processing on large clusters." Communications of the ACM 51.1 (2008): Russom, P. (2011). Big data analytics. TDWI Best Practices Report, Fourth Quarter. Sakr, Sherif, et al. "A survey of large scale data management approaches in cloud environments." Communications Surveys & Tutorials, IEEE 13.3 (2011): Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

21 Topic Assigned to Supervision Hadoop Bux Hive Leser Pig Latin Brandt Asterix Leser Meteor: Information Extraction as QL Leser Pregel, Giraph Bux Spark Brandt Mahout Leser Scuba Leser Naiad Brandt Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

22 Google s MapReduce and Apache Hadoop Hadoop v1: open source implementation of Google s MapReduce programming model and DFS Hadoop v2: yet another resource negotiator (YARN) Release in Oct 2013 Separated job scheduling from resource management MapReduce is one of many possible ApplicationMasters Topic Outline: Overview of M/R programming model, Hadoop architecture and Tez ApplicationMaster Suitable for the thorough practitioner Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

23 Hive First DWH system build on top of Hadoop (Facebook) Many successors: Cloudera, HortonWorks, Pentaho, Quite comprehensive (read-only) SQL support Focus: Query optimization over batch-oriented singlesource Map/Reduce Joins on Map/Reduce Index support Now apache project Practical, for the data warehouse fan Also look at commercial competitors Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

24 Pig Latin A not-so-foreign language Combines advantages of SQL Declarative style queries and Hadoop MapReduce Scalability (distributed computation, data locality) HDFS (replication, failure recovery) Allows data other than tables Execution plans are translated to MapReduce programs Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

25 AsterixDB Oger or Parfait? Relational approach to BIG DATA processing of semistructured data Focus on optimization and expressiveness (loops) Rich literature, appropriate for database freaks Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

26 Meteor: IE as Dataflow Not all big data is structured (logs, sensors) Text is ubiquitous (Web, news, books, social media, ) Idea: Express information extraction as dataflow And compile to parallel execution plan / cluster POS Gene NER Drug NER RE POS Gene NER Filter Drug NER Filter RE POS Drug NER Filter Gene NER Filter RE Focus: Optimization of UDF-heavy dataflows Good knowledge in optimization, affinity to text Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

27 Google s Pregel and Apache Giraph Pregel: compute model for processing large-scale graphs Example: Find distance from a person to any person in a social network Vertices, edges have mutable values Vertices send messages to each other Iterative graph transformation: vertices adjust values in each iteration Giraph: open source implementation of Pregel on Hadoop Topic Outline: Description of Pregel s computational model and realizations of well-studied graph algorithms Suitable for the graph-loving theorist (difficulty: moderate) Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

28 Spark Distributed large-scale data analysis platform Extends Scala Complete OO language with Functional elements Built around Resilient Distributed Datasets (RDDs) Data is held in memory and spilled to disk only when necessary Knows how it was computed; can be recomputed if data gets corrupted Algorithmic skeletons processed in parallel: reduce collect foreach Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

29 Mahout Apache platform for large-scale machine learning Used to compile to Hadoop, recently moved to Spark Focus on recommendations, clustering, classification Technically: Large, parallel matrix manipulations Example: user vectors of movie ratings Collaborative filtering: Find similar users Matrix: Transform user/rating matric into user/user similarity matrix Outlook on matrix DSL would be cool Demos welcome, prior knowledge of ML and LA helpful Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

30 Real time analytics:.. Scuba is a fast, scalable, distributed, in-memory database [AAB+14] MySQL cluster, Hive, Hadoop too slow, long latency 70TB main mem for petabytes of raw data sampling Distributed processing for systems guys Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

31 Naiad Timely dataflow system Dataflow is modelled as a Directed Graph Nodes: processors which can send and receive messages Edges: Connections over which messages can be passed Feedback loops allowed Use case: Real-time data analysis Iterative algorithms Input constantly updated Messages have consistent timestamps Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

32 Potential Further Topics More streaming: Storm, Stream-X, More DWH: Cloudera, More main memory / column stores: SAP HANA More declarative queries: Stratosphere/Flink, Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

33 A Unifying View Source: V. Markl / Stratosphere Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

34 Topic Assigned to Supervision Hadoop Bux Hive Itzerott Leser Pig Latin Brandt Asterix Stiehler Leser Meteor: Information Extraction as QL Leser Pregel, Giraph Völker Bux Spark Borchert Brandt Mahout Shi Leser Scuba Manthey Leser Naiad Brandt Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

35 Data Scientist Source Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

36 ToC Introduction Topics Assignment Hints on presenting your topic and writing your thesis Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

37 Allgemeine Hinweise Dozenten sind ansprechbar! Vorbesprechung des Themas Folien durchgehen Abgrenzung der Ausarbeitung Diskussion erwünscht Keine Angst vor Fragen: Fragen sind keine Kritik Eine Frage nicht beantworten können ist in Ordnung Tiefe, nicht Breite Lieber das Thema einengen und dafür Details erklären Bezug nehmen Vergleich zu anderen Arbeiten (im Seminar) Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

38 Allgemeine Hinweise Werten und bewerten Keine Angst vor nicht ganz zutreffenden Aussagen solange gute Gründe vorhanden sind Begründen und argumentieren Kritikloses Abschreiben ist fehl am Platz Literaturrecherche ist notwendig Die ausgegebenen Arbeiten sind Anker Weiterführende Arbeiten müssen herangezogen werden Auch Grundlagen nachlesen Wir schicken eine Liste zum Abhaken rum Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

39 Wie halte ich einen Seminarvortrag 1. Wenn man nun so einen Seminarvortrag halten muss, dann empfiehlt es sich, möglichst lange Sätze auf die Folien zu schreiben, damit die Zuhörer nach dem Vortrag aus den Folienkopien noch wissen, was man eigentlich gesagt hat. 2. Während so einem Vortrag schaut sowieso jeder zum Projektor, also kann man das selbst ruhig auch tun - damit kontrolliert man gleichzeitig auch, ob der Beamer wirklich alles projeziert, was auf dem Laptop zu sehen ist. Ausserdem kann man so den Strom für das Laptop-Display sparen. 3. Übersichtsfolien am Anfang sind langweilig, enthalten keinen Inhalt und nehmen den Zuhörern die ganze Spannung. Schliesslich gibt s im Kino am Anfang auch keine Inhaltsangabe. 4. Powerpoint kann viele lustige Effekte, hat tolle Designs und Animationen. Die sollte man zur Auflockerung des Vortrags unbedingt alle benutzen, um zu zeigen, wie gut man das Tool im Griff hat. 5. Nicht zu wenig auf die Folien schreiben. Man weiß ja nie, ob man sie nicht doch ausdrucken muss, und man kann so wertvolle Zeit sparen, wenn man nicht weiterschalten muss. 6. Man sollte versuchen, möglichst lange zu reden. Die Zeitvorgaben sind nur für die Leute, die nicht genug wissen - eigentlich will der Prüfer sehen, dass man sich auch darüber hinaus mit dem Thema beschäftigt hat. Bloß keine Hervorhebungen im Text sonst müssen die Zuhörer ja gar nicht mehr aufpassen! Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

40 Hinweise zum Vortrag Minuten plus Diskussion Klare Gliederung Ab und an Hinweise geben, wo man sich befindet Themenauswahl: Lieber verständlich als komplett Bilder und Grafiken; Beispiele Font: mind. 16pt Eher Stichwörter als lange Sätze Vorträge können auch unterhaltend sein Gimmicks, Rhythmuswechsel, Einbeziehen der Zuhörer, etc. Adressat sind alle Teilnehmer, nicht nur die Betreuer Technik: Laptop? Powerpoint? Apple? Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

41 Hinweise zur Ausarbeitung Eine gedruckte Version abgeben Selbstständigkeitserklärung unterschreiben Eine elektronische Version schicken Referenzen: Alle verwendeten und nur die Im Text referenzieren, Liste am Schluss Korrekt zitieren Vorsicht vor Übernahme von kompletten Textpassagen; wenn, dann deutlich kennzeichnen Aussagen mit Evidenz oder Verweis auf Literatur versehen Verwendung von gefundenen Arbeiten im Web Möglich, aber VORSICHT Eventuell Themenschwerpunkt verschieben Betreuer fragen Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

42 Hinweise zur Ausarbeitung 2- Gezielt und sachlich schreiben Füllwörter vermeiden (dabei, hierbei, dann, ) Knappe Darlegung, präzise Sprache Eine gute Gliederung ist die halbe Miete Kommen Sie zu Aussagen Vorteile, Nachteile, verwandte Arbeiten, mögliche Erweiterungen, Anwendbarkeit, eigene Erfahrungen, Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/

43 Format Benutzung unserer Latex-Vorlage Nur eine Schriftart, wenig und konsistente Wechsel in Schriftgröße und stärke Inhaltsverzeichnis Bilder: Nummerieren und darauf verweisen Referenzen: [1] Yan, X., Yu, P. S. and Han, J. (2004). "Graph Indexing: A Frequent Structure-Based Approach". SIGMOD, Paris, France. [YYH04] Yan, X., Yu, P. S. and Han, J. (2004). "Graph Indexing: A Frequent Structure-Based Approach". SIGMOD, Paris, France. Darf man Wikipedia zitieren? Ja, aber nicht dauernd Ulf Leser: Infrastructures for BIG DATA, Winter Semester 2014/