90 Prozent der heute weltweit vorhandenen Daten wurden dabei erst in den letzten zwei Jahren generiert.

Transkript

1 QualysoftGruppe

2

3 Jeden Tag werden 2,5 Trillionen Byte an Daten erstellt. 90 Prozent der heute weltweit vorhandenen Daten wurden dabei erst in den letzten zwei Jahren generiert. Diese Daten stammen aus den unterschiedlichsten Quellen: Sensoren, die Klimadaten sammeln, Beiträgen auf Social-Media-Sites, digitalen Bildern und Videos, Datensätzen zu Kauftransaktionen und GPS-Signalen von Mobiltelefonen, um nur einige zu nennen. Diese großen Datenmengen sind Big Data.

4

5 5 Quelle: Spiegel.de

6 IBM Watson 6

7

8 BigData Warehouse - DWH und BigData im Einklang. Dominik Jaeth Country Manager Germany

9 BigData? Die Vier V Menge Schnelle (Bewegung) Vielfalt Qualität

10 Was ist -BigData McKinsey Big Data refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze. Gartner Big Data are high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable enhanced decision making, insight discovery, and process optimization. BARC Big Data designates methods and technologies for the highly scalable acquisition, storage, and analysis of polystructured data

11 Kimball zu BigData Building a Hadoop Data Warehouse It s a rennaisancethat is happening here A Data Warehouse needs to encompass Big Data and I hope that all folks working with those (Big Data) topics realize that they are part of the larger Data Warehouse team We want to use SQL and SQL like languages but we don t want the RDBMS storage constraints.the disruptive solution: Hadoop

12 RDBMS vs. Hadoop& Co. Schema Datentransofmation vor Verwendung ETL. ca. 70%der gesamten DWH Kosten RDBMS Data Tables System Tables RAW oder Strukturiert Storage Meta Data Hadoop& Co HDFS Filesystem HCatalog Scheme less Transformation/Inter pretation bei Verwendung Nur optionale SchemaViews (HCatalog) SQL Query Engine Alle drei Layer sind voneinander abhängig Proprietäre Systeme Query Multiple Schemata SQL und nosql Multiple Engines Alle drei Layer sind voneinander unabhängig multiaccess

13 Warum ETL? S implicity ETL transformiert die Daten in gut lesbare Strukturen H istorization ETL ist Revisionssicher I ntegration ETL integriert und bereinigt Daten aus verschiedenen Quellen P erformance ETL bereitet Daten zum effizienten lesen vor (pre-processes) How to SHIP data for analytics without ETL?

14 SHIPon the fly -Schema-less Clean, transform, integrate and aggregate on read time Your business case drives the query and defines the requirements History is maintained nevertheless (you never delete) Performance by using the suitable tool the data to analyze and simple scale-out

15 Schema vs. Schema less It s trivial to decide if You know your data and your requirements You know how often your data is used Just calculate What does it cost (effort, execution time) to transform the data? How often do you need to read it? What is the expected response time? For common BI requirements, ETL is the most efficient and effective way regardless if you use a DWH or a Big Data platform

16 BigData Technologies anddwh aremadetoworktogether

17 Explorative Analysis with Big Data Technologies Use- and Business Case Prototype on Big Data platform ( Self Service ) Access any data source Virtually supports any adhoc business case based on Big Data Example: Product Trend Analysis requires product data and more Analyze own data, social media, blogs, tracking etc.. Reduces risks, improves revenue, delivers marketing suggestions and feedback for R&D etc. Data Scientist supports Invents and creates Use result Analytical Prototypes & One-time Solutions Plus evaluation of - New data, methods & algorithms - Data quality - Tools Capture & adopts for common use Management BI Specialist Admin / Expert

18 DWH Enhancements Scenario 1 Architecture BI Tools Advanced Analytical Tools Developer s Environments Tools Connectors Loaders Processing Metadata Programming Languages NoSQL DBs Other FS Scripting Languages Analytical Tools Map Reduce (S)QL Engines HDFS (incl. various file formats) Direct HDFS access Big Data Platform Images Video Web, Feeds OLTP DWH Sensordata Logs Audio Messages Services Data Sources

19 DWH Enhancements Scenario 1 Technology Oracle BI, Microsoft BI, SAP BO, IBM Cognos R, SAS, SPSS, RapidMiner, Knime, Weka, Eclipse, MS Studio, JDeveloper, Tableau, QlikView, SpotFire, Tools JDBC, ODBC, SQL*Net, Oozie, Flume, HCatalog, Java, Python, C++, Cassandra, HBase, CFS (?), Pig, Mahout, R, Map Reduce Hive, Impala, HDFS (incl. various file formats) Direct HDFS access Big Data Cluster. Hadoop& Co. Images Video Web, Feeds OLTP DWH Sensordata Logs Audio Messages Services Data Sources

20 DWH Enhancements Scenario 1 Technology Example Product Trend Analysis Process runs on Infrastructure BI tool connects to DWH, Hive & Impala via ODBC and SQL*Net CSV structures defined with HCatalogand used by HIVE Impala table definitions bound to Hive CSV structures Oracle BI, Microsoft BI, SAP BO, IBM Cognos R, SAS, SPSS, RapidMiner Knime, Weka, Hive query runs M/R against CSV Logs loaded via scoop as csv files on a daily base Impala s InMemorySQL Engine caches CSV file content Talendjob reads SM data from stream and loads into HDFS DWH data accessed directly via SQL*Net Java M/R job aggregates and matches SM data & region/classificat ion BI tool joins hadoopbased rating with DWH based classification and revenue data to identify and compare external and internal trends JDBC, ODBC, SQL*Net, Oozie, Flume, OLTP DWH HCatalog, Java, Python, C++, Cassandra, HBase CFS (?), Sensordata Logs

21 Big Data Archive for Data Warehouses Using Hadoop & Co as a DWH archive Cost efficient storage / archiving / unloading of (old) mass data Example: Account transaction inquiry for commercial banks allow millions of customers to check their transactions of the last 20 years including correspondence, documents etc. as well as internal analysis of historical, finest grained data DWH SQL Access Analytics Data Warehouse Old data unloading SQL Data link Web App Big Data System Archive Data & Documents OLTP Documents

22 Big Data Archive for Data Warehouses Process runs on Infrastructure Most current OLTP Txdata loaded to DWH via ETL tool Related documents transferedto HDFS for web access Old TxData exported to HBaseand/or HDFS files Oracle BI, Microsoft BI, SAP BO, IBM Cognos R, SAS, SPSS, RapidMiner Knime, Weka, Old Txdata tied to DWH via Hive/Impala via ODBC based DB Link Additionally added Big Data JDBC connect to BI Tool and Web Application HDFS API Access to documents for Web Applicatio JDBC, ODBC, SQL*Net, Oozie, Flume, HCatalog, Java, Python, C++, Cassandra, HBase CFS (?), JDBC access for BI Tool to Big Data system and DWH HDFS & DWH Access for Web Application OLTP DWH Sensordata Logs

23 Next Steps Controller fit für BigData machen Mit Big Data wird das Controlling vor neue Herausforderungen gestellt. Hierfür ist Weiterbildung, in zum Teil ungewohnten Bereichen, nötig: 1. Datenmanagement 2. Technologie Neue Aufgabenbereiche qualifiziert besetzen Mit Big Data entwickeln sich zwei neue Berufsfelder die dem Controller sehr ähnlich sind. 1. Data Scientist 2. Business Analyst Auf pilothafte Use Cases setzen Start auf Basis eines konkreten UseCase Big-Data-Innovationsprozess aufbauen Ideenwettbewerb, UseCases Kosten-Nutzen Analyse

24 Time for

25 ThankYou Dominik Jaeth Country Manager Qualysoft Germany