Data Virtualization als Motor für Big Data

Transkript

1 Data Virtualization als Motor für Big Data Dr. Christian Kurze Principal Sales Engineer DACH Denodo Technologies

2 Denodo im Überblick Plattform Standorte Spezialist Kunden Denodo integriert Daten unabhängig von ihrer Quelle und Technologie (DB & DWH bis Hadoop, intern bis Web & Cloud, strukturiert bis unstrukturiert) in einer einheitlichen virtuellen Schicht. Applikationen nutzen die standardisierten Data Services in Real-Time (Right-Time). Headquarter: Palo Alto, CA Leader in Data Virtualization. Cool Vender in Data Integration. Weltweit: USA, United Kingdom, Spanien, Deutschland 15 Jahre Fokus auf Datenvirtualisierung & Data Services. Führendes Produkt. Lösungserfahrung. >250 Kunden aus allen Branchen, davon viele F500 und G2000 Unternehmen: Healthcare & Life Sciences, Technology, Media, Telecommunications, Insurance, Financial Services, Consumer/Retail, Public Sector. Top Product Innovation Top 10 IT Companies to Watch. Globale Präsenz in Nordamerika, EMEA, APAC, Lateinamerica. Analysten Konstantes Ranking als Leader in Datenvirtualisierung: Innovation, Flexibilität, Ease-of-Use und ROI. Best-of-Breed DV.

3 ETL Real World of Data

4 Datenvirtualisierung schließt die Lücke Query, Reference, Browse, Events/Alerts, Search

5 mit einer Datenmanagement-Strategie JDBC/ODBC Security, WS Security, HTTPS, Encryption SSL, Pass-Through Anonymous Web Browsing Encryption,

6 und einer ausgereiften Plattform Multiple Protocols, Formats Design Tools Optimizer Cache Scheduler Linked Data Services Query, Search, Browse Request / Reply, Event-Driven Secure Delivery Publish Real-time (Right-time) Data Services Combine Transform, Improve Quality, Integrate Connect Normalized Views of Disparate Sources Library of Wrappers Any Data or Content, Web Automation Read and Write Monitoring Governance Metadata Security

7 Wie funktioniert Datenvirtualisierung? PUBLISH Real-time (Right-time) Data Services PUBLISH Expose entities as data services Data services support multiple standard protocols including JDBC, ODBC, REST and SOAP/XML WS, etc. COMBINE COMBINE Transform, Improve Quality, Integrate CONNECT Normalized Views of Disparate Data Library of functions for transformation, normalization and data cleansing Graphical drag&drop tool for combining views using relational algebra Extended relational model supports hierarchical data sources CONNECT Normalized view of sources Pre-packaged connectors

8 Connect

9 Connect

10 Connect

11 Connect

12 Combine

13 Publish

14 Publish SELECT * FROM retailer; SELECT * FROM retailer WHERE retailer_code = 1101; Message Queue: <userdata> <retailer_code>1101</retailer_code> </userdata>

15 Wie funktioniert Datenvirtualisierung? SELECT * FROM customer; Kerngedanke: Keine Datenreplikation, Abfrage der Quellen in Echtzeit Anfragen werden an die virtuelle Schicht gesendet, analysiert und automatisch PUBLISH optimiert (z.b. Kostenbasierte Real-time (Right-time) Data Services Abfrageoptimierung, Caching, ). PUBLISH Expose entities as data services Data services support multiple standard protocols including JDBC, ODBC, REST and SOAP/XML WS, etc. Übersetzung der Anfrage in die Sprache COMBINE der Quelle (SQL, API, WebService-Calls, ); Transform, Improve Ausführung innerhalb der Quelle. Quality, Integrate COMBINE Nutzung der gesamten Funktionalität der CONNECT Datenquelle, z.b. Aggregationen innerhalb Normalized Views of einer DWH-Appliance bei analytischen Disparate Data Abfragen. Die Quellen sind und bleiben oftmals die optimale Art, Daten zu halten. Extended relational model supports hierarchical data sources Rückgabe des Ergebnisses an die virtuelle Schicht, ggf. Weiterverarbeitung und Rückgabe an den Konsumenten. Library of functions for transformation, normalization and data cleansing Graphical drag&drop tool for combining views using relational algebra CONNECT Normalized view of sources Pre-packaged connectors

16 Grafische Analyse der Abfragen

17 Data Virtualization Einsatzfelder Agile Business Intelligence Real-time Reporting Unified Dashboards Agile Single View Applications Customer Single View - Call Center, Portals Unified Desktops - Underwriting, Loans, Virtual Logical DW / Marts Operational Decision Data Big Data / Predictive Analytics DS for Mobile Apps Self-Service & Mobile BI Big Data for Enterprise Cloud / SaaS Integration Web Data Aggregation Social Media, Feedback Multi-structured Big Data, Cloud Integration Data Services for BPM Data Virtualization Linked Data Services (IaaS) Unified Data Layer Unified Source of Reference Virtual MDM Logical Data Abstraction Migration / Modernization Linked Data Services

18 Data Virtualization Kunden Agile Business Intelligence Agile Single View Applications Data Virtualization Big Data, Cloud Integration Linked Data Services

19 Data Virtualization Kundenerfolge Agile Business Intelligence Agile Single View Applications Access new data sources 60% faster with change requests met in just a few days with IT using 40% less analyst time to support. 50% improvement in productivity with 75% reduction in-store waiting times and 87% reduction in errors due to manual data input. Big Data, Cloud Integration Cost savings of $600,000 and 15% ROI in 3 months to delivers secure and timely manner information. Data Virtualization New suite of agile reports to sales & management in 3x the speed with 1/3rd the team size. Linked Data Services

20 Data Virtualization: Türöffner für Big Data (= ALLE Daten) Data Lakes und Enterprise Data Hubs Alle Quellen sind nutzbar: Hadoop (Cloudera, Hortonworks, Amazon, etc.) NoSQL (MongoDB, Cassandra, Aerospike, etc.) SQLifizierung und Standardisierung des Zugriffs auf Big Data und NoSQL Abstraktion der technischen Komplexität zur einfachen Integration von Big Data mit Unternehmensdaten Analytical Data Integration Sandboxing & Prototyping

21 Die Challenge für BI

22 Data Virtualization als Lösung Benefits of Virtual Data Services 1. Unique Business Model by seamless integration of analytical results with enterprise data (30 Trillion of simulation values; 34 TB; unique statistical models) 2. Data Services are directly used by customers and internal departments to create better and unique offers to customers (Sales, Marketing, Risk, Finance, etc.) 3. Centralized Security by offering a companywide virtualization layer.

23 Telematics & Predictive Maintenance Tableau: Dealer / Customer Dashboard Benefits of Virtual Data Services and Analytics Dealer 1. Improved Service and less Downtime through realtime analytics of sensor data and proactive service to customers Maintenance 2. Predictive Analytics drive Resource Optimization in maintenance, distribution, service by integrating detailed sensor data to enterprise data 3. New Business Model to offer based on the real-time analysis of detailed sensor data Parts Inventory OSI PI Hadoop Cluster

24 Global Bank: NoSQL for Cold Data Storage Benefits of Virtual Data Services 1. Massive Reduction of Storage space and Cost since the warehouse only contains the most current entries (e.g. last year), historical data is stored in Hadoop. Also works for vertical partitioning, i.e. the most actively used attributes are stored in the DWH, lessoften used attributes are stored in Hadoop Dimension 1 (product) Fact table (sales) Dimension 2 (country) Data Warehouse Data Warehouse Without Data Virtualization 2. Performance Increase by only querying the necessary data set recent vs. historical data Fact table (sales) Fact table (sales > 2 years old) Data Warehouse Hadoop With Data Virtualization 3. Query Simplifaction by transparent union of recent and historical data within the virtual layer, not within the application which requests data

25 Zusammenfassung Data Virtualization Middleware für virtuelle Datenintegration Standardisiertes logisches Datenmodell für alle Applikationen und Nutzer Universeller Datenzugriff: intern/extern, Schätze aus allen Daten heben Minimierung der Replikation Wiederverwendbare Datenservices Zentrale Security und Governance Web/Cloud/Big Data, strukturiert/unstrukturiert Flexible Integrationsoptionen real-time, cached, scheduled batch Enterprise Class Powerful and Agile Abstrahierte und vereinheitlichte Datenservices, Zugriffskontrolle, SLAs Data Governance, Data Lineage, Data Management, Einfache Integration in Infrastruktur Performance und Skalierbarkeit

26 Vorteile aus Data Virtualization Hochqualitative Informationen Integration getrennter Datensilos Integration von Web / Cloud, Big Data, unstrukturiert Real-time (Right-time) Data Services Einhaltung von SLAs: IT <-> Fachabteilung <-> Kunde Agilität für die Fachabteilungen Flexibilität für die IT Sicherung von Wettbewerbsvorteilen Geringe Kosten & hohe Agilität Integrationskosten um 80% gesenkt Flexibilität bei Changes Schnelle Realisierung von Lösungen Projekte in 4-6 Wochen, ROI in <6 Monaten Neue Möglichkeiten für IT und Business

27 The Fastest Way to Data Virtualization. Download Denodo Express now from community.denodo.com

28 The Fastest Way to Data Virtualization. Your Way to Become a Data Ninja. Download Denodo Express now from community.denodo.com

29 The Fastest Way to Data Virtualization. Your Way to Become a Data Ninja. Download Denodo Express now from community.denodo.com

30

31 Copyright Denodo Technologies All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.