MATERNA GmbH

Transkript

1 MATERNA GmbH

2 Agenda Herausforderungen BigData Größeres Pferd oder Pferdegespann? Apache Hadoop Geschichte, Versionen, Ökosystem Produkte HDFS Daten speichern und verteilen Map/Reduce parallelisieren der Verarbeitung Pig,Hive Verarbeitung von Daten HBase Verteilte und skalierbare Speicherung von Daten Benchmark: Wie schnell 1TB sortieren? BigData-UseCases mit ORACLE Fazit, Ausblick Materna GmbH

3 Eigenschaften von Big Data Steigende Nutzerzahlen Datenvolumen Geschwindigkeit der Datenproduktion Datenquellen Materna GmbH

4 Kennzeichen von Big Data: die vier Vs Value Volumen Velocity Variety Materna GmbH

5 Größeres Pferd oder Pferdegespann? vertikal vs horizontal Materna GmbH

7 Der Elefant hat den Laden verlassen Materna GmbH

8 Hadoop Entstehung, Hintergründe: Da hinter steckt ein Kopf Doug Cutting Lucene-Suche 1997 SourceForge, 2001 ASF Nutch-Webcrawler Hadoop 2003, 2005 ASF Google Labs paper: The Google File System, October, 2003 MapReduce algorithm, December 2004 Bigtable: A Distributed Storage System for Structured Data, November 2006 H-store: a high-performance, distributed main memory transaction processing system, August 2008 Dremel: Interactive Analysis of WebScale Datasets, September 2010 Board of directors of the Apache Software Foundation July 2009 Doug Cutting Cloudera August 2009 ASF chairman September 2010 Materna GmbH

9 Hadoop Releases Feature 0.20/ /2.4 Secure authentication X X Old configuration names X Deprecated Deprecated New configuration names X X Old MapReduce API X X X New MapReduce API (X) X X MapReduce 1 runtime (Classic) X X MapReduce 2 runtime (YARN) X HDFS federation X HDFS high-availability X Materna GmbH

10 Materna GmbH

11 Das Hadoop-Ökosystem (De-Facto-Standard) Zookeeper Coordination HBase Columnar NoSQL Store Pig Data Flow MapReduce Hive SQL Distributed Programming Framework HCatalog Table & Schema Management HDFS Hadoop Distributed File System Materna GmbH

12 Evolution von Hadoop 2006 HDFS MapReduce 2008 HBase Zookeeper Pig Hive Flume Avro Whirr Sqoop Mahoot Oozie HCatalog Bigtop Ambari YARN Materna GmbH

13 Wer entwickelt Hadoop? (Quelle: Hadoop in Practice) Materna GmbH

14 Einsatzgebiet: Datenbereinigung Applications Business Analytics Custom Applications Enterprise Applications Collect Data and apply a known algorithm to it in trusted operational process Data Systems RDB MS EDW Traditional Repos MPP 3 Hadoop Platform Capture Capture all data Process Parse, cleans, apply structure in all form Data Sources Traditional Sources RDBMS, OLTP, OLAP New Sources Web logs, , sensors, social media 3 Exchange Push to existing data warehouse for use with existing analytic tools Apache Hadoop Patterns of Use: Hortonworks 2013 Materna GmbH

15 Einsatzgebiet: Datenauswertung Applications Custom Applications Enterprise Applications Collect Data, analyze and present salient results for online apps 3 Data Systems RDB MS EDW Traditional Repos MPP NOSQL Hadoop Platform Capture Capture all data Process Parse, cleans, apply structure & transform Data Sources Traditional Sources RDBMS, OLTP, OLAP New Sources Web logs, , sensors, social media 3 Exchange Incorporate data directly into applications Apache Hadoop Patterns of Use: Hortonworks 2013 Materna GmbH

16 HDFS, MapReduce, NameNode, DataNode MapReduce JobTracker TaskTracker 1 TaskTracker N YARN ResourceManager (RM) NodeManager 1 NodeManager N ApplicationMaster (AM) AM 1 AM N HDFS NameNode DataNode 1 DataNode N 64 MB 64 MB 64 MB 18 MB Worker 1 Worker N Materna GmbH

17 File 1 Daten in HDFS schreiben: Rackawareness Replication B1 B2 B3 NameNode n1 B1 n1 B2 n1 B1 n2 B1 n2 B2 n2 B3 n3 B2 n3 B3 n3 B3 n4 n4 n4 Rack 1 Rack 2 Rack 3 Materna GmbH

18 MapReduce-Verfahren 2 parallele Phasen, Pipes&Filter (UNIX), funktionale Programmierung, Fehlertoleranz Eingabe Ausgabe Mary had a little lamb, It's fleece was white as snow, Every where that Mary Went, The little lamb was sure to go. (key, value) list(out_key, intermediate_value) Map Map Map Reduce list(out_value) Reduce berechnete Werte Materna GmbH

19 Wie werden MapReduce Jobs ausgeführt? MapReduce program 1. run job client JVM client node 3. copy job resources JobClient 2. get new job ID 4. submit job 6. retrieve input splits JobTracker jobtracker node 5. initialize job 7. heartbeat (returns task) Distributed file system (e.g. HDFS) 8. retrieve job resources TaskTracker 9. launch child JVM Child 10. run MapTask or ReduceTask tasktracker node Materna GmbH

20 MapReduce bei Hadoop Client Job Hadoop MapReduce Job parts Job parts Map Input data Map Map Reduce Reduce Output data (Quelle: Hadoop in Practice) Materna GmbH

21 Wie hängen Teile zusammen? locate ZooKeeper Client HBase Master read/write HBase region HBase region HBase region HDFS HDFS HDFS Partitionierung Replikation (Quelle: Hadoop in Practice) Materna GmbH

22 Wie hängen Teile zusammen? Client Client PigLatin Pig submit job Hadoop MapReduce submit job HiveQL Hive read/write read/write Hadoop HDFS (Quelle: Hadoop in Practice) Materna GmbH

23 Pig und Hive im Vergleich Tez Characteristic Pig Hive Developed by Yahoo! Facebook Language Pig Latin HiveQL Type of language Data flow Declarative (SQL dialect) Data structures Complex Better suited for structured data Schema Optional Not optional Materna GmbH

24 Pig Komponenten Pig Pig Latin Compiler An operation as a statement A command as a statement LOAD input.txt ; ls *.txt Logical Plan Execution Environment Local Distributed DUMP Compile Three steps: LOAD Load data from HDFS TRANSFORM Translated to a set of map and reduce tasks DUMP or STORE Display or store result Physical Plan Execute Materna GmbH

25 Hive Architektur DDL JDBC/ODBC Queries Metastore (Relational database for metadata) CLI Web Interface Parser, Planner Optimizer Hadoop Materna GmbH

27 Hadoop Cluster Hardwareanforderungen: eine Menge Blech Intel Cloud Builders Guide: Apache Hadoop Materna GmbH

28 Hardware components of Intel Hadoop cluster Master Job Tracking and HDFS*/ Storage Metadata Slave 1 Slave N Data Storage and Processing Rackawareness Replication Map Reduce Task Tracker Task Tracker Task Tracker Job Tracker Zookeeper* Hive* Pig* HDFS Name Node Data Node Data Node Data Node Oozie* Avro* HDFS Client* R720/ C2 100 R720XD/C2100/C6100/C6105 R720/ C2 100 Materna GmbH

29 Optimierungsbereiche Hadoop-Installationen Encryption Security & API s Benchmark tuning HI-Tune HI-Bench Network Storage Compute Fast fabric 10 GbE SSDs Non-volatible memory Disk write/ memory Materna GmbH

30 Unterschiedliche Kompressionsverfahren Size (Mbytes) Compression Speed (sec) Compression memory used (Mbytes) Decompression speed Decompression memory used (Mbytes) Un 96 Y compressed Gzip N Bzip Y lzo (Y) Split table (HADOOP-1824) want InputFormat for zip files Client hadoop-site.xml <property><name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.defaultcodec,org.apache.hadoop.io.compress.gzipcodec,org.apache.hadoop.io.compress.lzopcodec</value> </property> Materna GmbH

31 Wichtig: Teste deine Infrastruktur bin/hadoop jar hadoop-*test*.jar, Benchmarks time hadoop jar../hadoop-examples jar wordcount 2 4 hadoop jar hadoop*examples*.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output time hadoop jar../hadoop-test jar Kontrolle über: web UI of the NameNode daemon web UI of the JobTracker daemon web UI of the TaskTracker daemon Materna GmbH

32 HBase Master user interface Materna GmbH

33 ZooKeeper debugging HBase Materna GmbH

34 WordCount-Algorithmus mit MapReduce Mary had a little lamp Its fleece was white as snow And everywhere that Mary went The lamb was sure to go map map map map Map.class Mary1 had1 a1 little1 lamp1 Its1 fleece1 was1 white1 as1 snow1 And1 everywhere1 that1 Mary1 went1 The1 lamb1 was1 sure1 to1 go1 reduce reduce Reduce.class had1 a1 little1 Lamp2 Mary2 was2 white1 snow1 Materna GmbH

35 Example: WordCount Hadoop Tutorial public class WordCount { public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> { } public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> { } public static void main(string[] args) throws Exception { JobConf conf = new JobConf(WordCount.class); conf.setjobname("wordcount"); conf.setoutputkeyclass(text.class); conf.setoutputvalueclass(intwritable.class); conf.setmapperclass(map.class); conf.setcombinerclass(reduce.class); conf.setreducerclass(reduce.class); conf.setinputformat(textinputformat.class); conf.setoutputformat(textoutputformat.class); FileInputFormat.setInputPaths(conf, new Path(args[0])); FileOutputFormat.setOutputPath(conf, new Path(args[1])); JobClient.runJob(conf); } Materna GmbH

36 Example: WordCount Hadoop Tutorial public class WordCount { public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> { public void map(longwritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { String line = value.tostring(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasmoretokens()) { word.set(tokenizer.nexttoken()); output.collect(word, one); } } } public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { int sum = 0; while (values.hasnext()) { sum += values.next().get(); } output.collect(key, new IntWritable(sum)); } } Materna GmbH

37 Example: Hive WordCount HQL CREATE TABLE docs (line STRING); LOAD DATA INPATH /user/cloudera/wordcount/input/file' OVERWRITE INTO TABLE docs; CREATE TABLE word_counts AS SELECT word, count(1) AS count FROM (SELECT explode(split(line, '\s')) AS word FROM docs) w GROUP BY word ORDER BY word; Materna GmbH

38 Example: Pig WordCount Script input_lines = LOAD /user/cloudera/wordcount/input/file' AS (line:chararray); words = FOREACH input_lines GENERATE FLATTEN(TOKENIZE(line)) AS word; filtered_words = FILTER words BY word MATCHES '\\w+'; word_groups = GROUP filtered_words BY word; word_count = FOREACH word_groups GENERATE COUNT(filtered_words) AS count, group AS word; ordered_word_count = ORDER word_count BY count DESC; STORE ordered_word_count INTO /user/cloudera/wordcount/output/part '; Materna GmbH

39 Example: WordCount Hadoop Tutorial $ echo "Hello World Bye World" > file0 $ echo "Hello Hadoop Goodbye Hadoop" > file1 $ hadoop fs -mkdir /user/cloudera /user/cloudera/wordcount /user/cloudera/wordcount/input $ hadoop fs -put file* /user/cloudera/wordcount/input $ hadoop fs -cat /user/cloudera/wordcount/output/part Bye 1 Goodbye 1 Hadoop 2 Hello 2 World 2 Materna GmbH

40 Example: WordCount Hadoop Tutorial first input map : < Hello, 1> < World, 1> < Bye, 1> < World, 1> second input map : < Hello, 1> < Hadoop, 1> < Goodbye, 1> < Hadoop, 1> first output map: < Bye, 1> < Hello, 1> < World, 2> second output map: < Goodbye, 1> < Hadoop, 2> < Hello, 1> The Reducer sums up the values: < Bye, 1> < Goodbye, 1> < Hadoop, 2> < Hello, 2> < World, 2> Materna GmbH

41 Terasort benchmark Hadoop: Wie lange dauert es 1TB zu sortieren? asort/package-summary.html hadoop jar hadoop-*examples*.jar terasort <input dir> <output dir> 2008, 1TB 3,48 minutes 910 nodes x (4 dual-core processors, 4 disks, 8 GB memory) 2009, 100 TB in 173 minutes 3452 nodes x (2 Quadcore Xeons, 8 GB memory, 4 SATA) TB sort in 10,369 seconds IBM InfoSphere BigInsights 100 TB (1.000 virtual machines, 200 nodes, Cores) Materna GmbH

43 The Forrester Wave: Big Data Hadoop Solutions, Q Materna GmbH

44 Das Hadoop Ecosystem Partnerbeziehungen Materna GmbH

45 Hortonworks Data Platform 2.1 Materna GmbH

46 Historie Hortonworks Data Platform mit Komponentenversionen Materna GmbH

47 Hadoop als unternehmensweite Plattform BITKOM-Leitfaden: Big-Data-Technologien - Wissen für Entscheider Materna GmbH

48 Anwendung der Hortonworks Data Platform für die Analyse von Twitter-Daten BITKOM-Leitfaden: Big-Data-Technologien - Wissen für Entscheider Materna GmbH

49 Big-Data-Architektur bei Ebay, Stand 2011 BITKOM-Leitfaden: Big-Data-Technologien - Wissen für Entscheider Materna GmbH

50 Traditional vs Big Data Information Architecture Capabilities Oracle: Big Data for the Enterprise, Whitepaper, 2012 Materna GmbH

51 Oracle Integrated Information Architecture Capabilities Oracle: Big Data for the Enterprise, Whitepaper, 2012 Materna GmbH

52 Use Case #1: Initial Data Exploration Oracle Information Architecture: An Architect s Guide to Big Data, Whitepaper 2012 Materna GmbH

53 Use Case #2: Big Data for Complex Event Processing Oracle Information Architecture: An Architect s Guide to Big Data, Whitepaper 2012 Materna GmbH

54 Use Case #3: Big Data for Combined Analytics Oracle Information Architecture: An Architect s Guide to Big Data, Whitepaper 2012 Materna GmbH

56 Use Case #5: Einsatzmöglichkeit Oracle Big Data Appliance Oracle Information Architecture: An Architect s Guide to Big Data, Whitepaper 2012 Materna GmbH

58 Oracle integrated Big Data Solution Oracle Information Architecture: An Architect s Guide to Big Data, Whitepaper 2012 Materna GmbH

59 Oracle Big Data Appliance Oracle Information Architecture: An Architect s Guide to Big Data, Whitepaper 2012 Materna GmbH

60 Oracle NoSQL Database integrates into the data management Oracle Information Architecture: An Architect s Guide to Big Data, Whitepaper 2012 Materna GmbH

61 Wo stehen wir: Der Sprung über den großen Graben Crossing the Chasm: Geoffrey A. Moore Materna GmbH

62 Crossing the Chasm : Koexistenz & Kooperation RDBMS ORACLE Hadoop Wenn ich die Menschen gefragt hätte, was sie wollen, hätten sie gesagt schnellere Pferde Henry Ford Materna GmbH

63 Fazit billige Standard-Hardware, Umgang mit Ausfällen billiger Hauptspeicher günstiger als großes Cluster Daten-Lokalitäts-Prinzip Verteiltes parallelisiertes Dateisystem mit Replikation Spezialisierte Datenspeicher (Spalten, Key/Value) Divide-et-Impera, parallelisierter MapReduce-Algorithmus Interaktive SQL-Abfrageengine für HDFS/HBase (Impala) Mehr Realtime-Verarbeitung, weniger Batch Betriebsthemen wichtiger: Update, Monitoring, Sicherheit Materna GmbH

64 Ausblick Hadoop ist DeFacto-Standard für BigData-Processing LINUX bleibt bevorzugte Hadoop-Plattform Nur wenige Hadoop Distributionen werden überleben Das Hadoop Ökosystem wird wachsen Der Hadoop-Dienstleistungsmarkt wird wachsen Hadoop Appliance reduzieren Kosten, Komplexität Hybride RDBMS werden Lücke schließen Benchmarks wichtig für Sizing, Tuning, Systemauswahl Keep your ecosystem simple! Materna GmbH

65 Literatur Materna GmbH

66 Vielen Dank für Ihre/Eure Aufmerksamkeit! MATERNA GmbH Dipl. Inform. Frank Pientka Senior Software Architect Business Division Applications Telefon: Telefax: Materna GmbH