Evaluation von Hadoop mit der Talend Big Data Sandbox. Michael Pretsch Pre-Sales Talend Germany GmbH

Transkript

1 Evaluation von Hadoop mit der Talend Big Data Sandbox Michael Pretsch Pre-Sales Talend Germany GmbH 2015 Talend Inc. 1

2 Talend Überblick Die wichtigsten Fakten Gegründet im Jahr Mitarbeiter in 7 Ländern Zwei Zentralen in Los Altos (Kalifornien) und Paris Open-Core-Geschäftsmodell Subskriptionslizenz Services und Schulungen

3 Talend Platform 3

4 Die klassische DWH Infrastruktur steht unter Druck 4

5 Hadoop: parallel vs. monolytisch Monolithisch Parallel teuer hoch spezialisierte HW single point of failure Commodity hardware fehlertolerant 5

6 Ansatzpunkte für Hadoop und NoSQL 6

7 Hemmschwelle Technologiepark Komplexität - ~ 25 Einzelprojekte - Administration großer Clusterumgebungen - Abstimmung der Releasestände - Entwicklerqualifikation 7

8 Hemmschwelle Programmierparadigma 8

9 Von diesem Ansatz 1. package org.myorg; 2. import java.io.ioexception; 3. import java.util.*; 4. import org.apache.hadoop.fs.path; 5. import org.apache.hadoop.conf.*; 6. import org.apache.hadoop.io.*; 7. import org.apache.hadoop.mapred.*; 8. import org.apache.hadoop.util.*; 9. public class WordCount { 10. public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> { 11. private final static IntWritable one = new IntWritable(1); 12. private Text word = new Text(); public void map(longwritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { 15. String line = value.tostring(); 16. StringTokenizer tokenizer = new StringTokenizer(line); 17. while (tokenizer.hasmoretokens()) { 18. word.set(tokenizer.nexttoken()); 19. output.collect(word, one); } } } 20. public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> { 21. public void reduce(text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException { 22. int sum = 0; 23. while (values.hasnext()) { 24. sum += values.next().get();} 25. output.collect(key, new IntWritable(sum));}} public static void main(string[] args) throws Exception { 28. JobConf conf = new JobConf(WordCount.class); 29. conf.setjobname("wordcount"); 30. conf.setoutputkeyclass(text.class); 31. conf.setoutputvalueclass(intwritable.class); 32. conf.setmapperclass(map.class); 33. conf.setcombinerclass(reduce.class); 34. conf.setreducerclass(reduce.class); 35. conf.setinputformat(textinputformat.class); 36. conf.setoutputformat(textoutputformat.class); 37. FileInputFormat.setInputPaths(conf, new Path(args[0])); 38. FileOutputFormat.setOutputPath(conf, new Path(args[1])); 39. JobClient.runJob(conf); }} 9

10 zu diesem Ansatz 10

11 Talend Big Data Einfach bedienbare, leistungsstarke Integrationslösung für Big Data Grafisches, Drag and Drop Design 800+ Konnektoren/Komponenten Code-Generator: MapReduce, Java, SQL Skaliert mit dem Cluster Load balancing & failover Optimierter MapReduce Code Native Einbindung in Hadoop Zentrales Job-Management Scale Design Manage Deploy Collaborate Gemeinsames Repository Automatische Dokumentation Keine Installation in Hadoop erforderlich Bereinigen und Anreichern Kerberos Unterstützung 11

12 Talend: Code-Generator Keine black-box engine Ermöglicht leichtgewichtige, verteilte, anpassbare und parallelisierbare Ausführungsumgebung - Spark - Storm 12

13 Talend: graphische IDE Talend Studio - Job Designer (ETL, ELT, Hadoop ) - Wizards - optimierter Map Reduce Code - Umfangreiche Schnittstellen inkl. NoSQL - Entwicklung und Test direkt im Studio Vorteile - deutlicher Produktionsgewinn im Vergeich zur manuellen Programmierung - schnelles Design und Test von ETL logic - einheitliches Vorgehensmodell für unterschiedliche Integrationsaufgaben (ETL, ELT, ESB, DQ, profiling, MapReduce, Pig, Hive, NoSQL ) 13

14 Talend Big Data 14

15 Talend Big Data Sandbox Virtuelle Maschine Hadoop ausprobieren ohne Aufwand Prototypen for FREE Verfügbar für drei Hadoop Distributionen 15

16 Talend Big Data Sandbox Demo Szenarien Clickstream-Analyse Twitter Sentiment-Analyse Apache Weblog-Analyse ETL offloading mit Hadoop 16

17 Schritt für Schritt Anleitung Vollständig eingerichtete Demo VM Sandbox Beispielszenarien wie Twitter and Clickstream Analysis 17

18 Wie funktioniert es? kostenlose Registrierung & Download VM importieren und starten 30 Tage Testlizenz einspielen und los geht s 18

19 Talend Big Data Sandbox einrichten VM Software (VMware, Virtual Box) Linux VM Talend Lizenz einspielen 19

20 Talend Big Data Sandbox live 2015 Talend Inc. 20

21 What Is Talend s Solution? 2015 Talend Inc. 21

22 Simplify Big Data Integration Make the job of working with big data technologies easy to drive and improve business performance Dramatically Faster and More Productive Development - Rich, graphical tools - Use code generation vs hand-coding - Easier to test, debug and maintain Lowest Total Cost of Ownership - Zero footprint runtime with native big data integration - Leverage existing ETL /Java developers - Linear, predictable pricing $ Future Proof Architecture - Robust, secure and scalable architecture - Easily connect to Big Data distributions including Amazon EMR, Cloudera, Hortonworks, MapR, Pivotal HD and SAP Hana - Continuously leverage the latest Hadoop capabilities 22

23 Key Features and Benefits 2015 Talend Inc. 23

24 Productive, Graphical IDE Design Talend Studio - Job Designer for Hadoop - Graphical wizards - Visual optimization - Broad NoSQL support - Run live in Studio Key Benefits - 30X productivity gain over hand-coding - Quickly build and test big data ETL logic - Reuse skills across project vs having specialist coders (MapReduce, Pig, Hive, NoSQL) 24

25 High Performing, Optimized and Native Code Generation Design Talend Studio Code Generator - Creates native Hadoop, Pig, Java, or SQL Optimization algorithms created by Talend experts - No runtime engines No black-box, proprietary software You can see the code! - Build once, run anywhere In the cloud or on-premises Key Benefits - Save money by using existing infrastructure - Do not need to learn MapReduce or get expensive developers - Higher quality code (since generated) - Easy to setup and maintain (no black-box to manage) - No code performance tuning required - Lowest cost of ownership 25

26 The Performance Benefits of Native Scale 100X 10X Only Talend runs native on Hadoop 1 st on MapReduce 1 st on YARN 1 st on Spark (preview) 1 st on Storm (preview) 2X Stinger Tez Spark.. 26

27 Pre-built Components Speed Project Delivery Design 800+ Drag-n-Drop Visual Components - Single Eclipse tool to transform any data - Open and extensible SDK - 60% from Talend Community - Included in per-developer pricing, so predictable, not CPU-based (like SAP) Key Benefits - Less coding using pre-built connectors and transforms - Reuse skill sets by learning Talend tool once and applying to other components (vs learning each components) - Higher quality work, since using pre-built components 27

28 Team Collaboration Powers Faster Development Collaborate Versioning Data Lineage Shared Repository / Metadata Manager - Shared by all stakeholders - All items versioned: Business Models, Jobs, Routines, Metadata, Documentation - Get back to any version easily Key Benefits - Increased reusability - Clearer communications across development team - Less duplicate work through better visibility of projects and jobs 28

29 Create High Quality, Trusted Information Deploy Big Data Quality - Explore, profile and monitor big data Hive profiling - Parse, cleanse, standardize and reconcile big data - Match, enrich and certify big data, then and share it widely and securely - Map any data source to your business context (customers products, organizations locations ) Key Benefits - Runs native on Hadoop no need to extract and reload! - More accurate information 29

30 Simplified Big Data Management and Security Manage Easily integrates with Big Data management consoles - Cloudera Manager - Hortonworks - MapR Hadoop security with Kerberos Key Benefits - Easy to manage and secure - Fast to troubleshoot Hortonworks Console with Talend 30

31 Talend Unified Platform Scale 1 Comprehensive Eclipse-based user interface 2 3 Consolidated metadata & project information Web-based deployment & scheduling Benefit: Faster to integrate and maintain using familiar tools Single web-based 4 monitoring console Same container for batch processing, message routing & services 5 31

32 Talend Differentiators 2015 Talend Inc. 32

33 Why Talend? Native Generates native Hadoop code, requiring zero footprint on your Hadoop cluster. The only data quality solution that runs natively in Hadoop no need to move data to cleanse it Open Built on open source and open standards Easy to learn, no vendor lock-in, more agile development. Vast Talend Forge community (80,000 members) Unified Resources are more productive using same tools for big data integration and data quality Predictable Subscription pricing with lower upfront investment Per developer vs per CPU pricing 800+ connectors included 33