Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal 2

Transkript

1

2 Agenda Big Data und relationale DB Big Data Technik Hadoop FS, Hadoop MapReduce, NoSQL Big Data Ökosystem Cloudera und Big Data Appliance DB 12c Json DB12c und Big Data SQL Fragen und Antworten Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal 2

4 Business Mehrwerte über Technologie Innovation Die passende Technologie für entsprechende Aufgaben UND Technologien im Verbund nutzen Hadoop NoSQL Relational Neue Geschäftsmodelle und Optionen Wettbewerbern zuvor kommen Vollständige Informationsketten Neu Paradigmen nutzen Neue Analysearten Scale the Business Schneller Daten bereitstellen Herausforderung Mobility Ökononisch sinnvolle Skalierung Run the Business Integration von Systemen Kritische Geschäftsprozesse Investitionsschutz Eingespielte Verfahren und Know how Copyright 2014 Oracle and/or its affiliates. All rights reserved. 4

5 Die Schwierigkeiten bei der Adaption neuer Technologien INTEGRATION SKILLS SECURITY Big Data in bestehende Architekturen einbinden und die Systeme beherrschen Noch fehlen sinnvolle Tools und das Know how um Big Data Daten direkt auszuwerten Keine klare Sicherheitsstrategie erkennbar Copyright 2014 Oracle and/or its affiliates. All rights reserved. Confidential 5

6 Unternehmen adaptieren Technologie in Phasen Geschäftlicher Nutzen Hadoop-Systeme in separaten Projekten parallel neben RDBMS Erfahrungen sammeln Zugriffsbarriere mit SQL Überwinden -> Beschäftigung mit Hive / HiveQL -> Performarmance- Fragen 1 Parallel Deployment 1) Zentrales DWH und klassisches ETL Hadoop wird als Vorsystem begriffen 2) BI Tool zieht Daten aus beiden Welten: Konsolidierte Sicht 3) Führendes Hadoop-System zieht Daten verschiedenen Quellen ab 2 Master Access Ein zusammenhängendes System ohne die jeweiligen Nachteile 3 Single Logical System System Reife Copyright 2014 Oracle and/or its affiliates. All rights reserved. 6

7 Ziel-Szenario / Technische Komponenten HADOOP und RDBMS im technischen Verbund (auch bezogen auf Hardware) SQL als durchgängige Access-Sprache Durchgängige Verwaltungsschicht -> Wo liegen welche Daten -> Metadaten-Information Copyright 2014 Oracle and/or its affiliates. All rights reserved. 7

8 Die Schwierigkeiten bei der Adaption neuer Technologien Oracle s - Weg INTEGRATION SKILLS SECURITY SQL Engineered Systems SQL auf allen Daten Database Security auf allen Daten Copyright 2014 Oracle and/or its affiliates. All rights reserved. Confidential 8

10 Cloudera Hadoop Ökosystem Hadoop (FS + MR), Flume (Process), HBase (Datenbank), HCatalog (Storage), Hive (SQL), Hue, Impala (SQL), Mahout (DataMining), Oozie, Pig, Cloudera Search, Sentry (Sicherheit), Spark (Analyse), Sqoop (), Whirr (Cloud), ZooKeeper Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal 10

11 Was wäre wenn das alles könnte Alle Datenarten für alle Datenbank-Anwendungen verfügbar machen Die volle Bandbreite der Oracle SQL Query Language nutzen Mit allen Security-Features von Oracle 12c Ohne Daten zwischen dem Hadoop-Cluster und der Datenbank hin und her zu kopieren Eine sehr hohe Query Performance erreichen Das bestehende Wissen (SQL-Skills) nutzen Und dennoch immer die aktuellsten Hadoop-Neuerungen nutzen Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 11

12 Oracle Big Data SQL Eine neue Architektur Mächtiges und hoch-performantes SQL auf Hadoop Alle Oracle SQL Möglichkeiten auf Hadoop SQL query Processing auf den lokalen Hadoop Knoten Einfache Daten-Integration von Hadoop und Oracle Database Single SQL Zugriffspunkt auf alle Daten Skalierbare Joins zwischen Hadoop und RDBMS-Data Optimierte Hardware High-speed Infiniband Netzwerk zwischen Hadoop und Exadata Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 12

13 Standardisierte Metadaten: Übernahme Hive Metadaten in den Oracle Datenbank-Katalog Hive Metastore CREATE TABLE movieapp_log_json (click VARCHAR2(4000)) ORGANIZATION EXTERNAL (TYPE ORACLE_HIVE DEFAULT DIRECTORY DEFAULT_DIR ) REJECT LIMIT UNLIMITED; Oracle Catalog Hive metadata External Table External Table Big Data Appliance + Hadoop/NoSQL Exadata + Oracle Database Copyright 2014 Oracle and/or its affiliates. All rights reserved. 13

14 Abfragen auf Hadoop ausführen Hive metadata HDFS Name Node HDFS Data Node HDFS Data Node HDFS Data Node HDFS Data Node I/O und Smart Scan ausführen Auswahl der Zeilen und Spalten Nur relevante Daten bewegen relevante Zeilen relevante Spalten Select c_customer_id, c_customer_last_name, ca_county From customers, customer_address where c_customer_id = ca_customer_id and ca_state = CA Oracle Catalog Hive metadata External Table External Table Join mit Daten aus der relationalen Datenbank Tables Copyright 2014 Oracle and/or its affiliates. All rights reserved. 14

15 Scans auf Hadoop optimieren Hive metadata HDFS Name Node HDFS Data Node HDFS Data Node HDFS Data Node HDFS Data Node Storage Indexes Min Max Min Max Min Max Min. und Max. Werte der Daten im Block festhalten Vor jedem Scan überprüfen, ob gesuchte Werte im Min. / Max. Bereich des Blockes liegen Wenn nicht, Blockscan überspringen um Zeit zu gewinnen Blocks Copyright 2014 Oracle and/or its affiliates. All rights reserved. 15

16 Automatisierung: Oracle Data Modeler Übernahme der Hive Definitionen in das Modell Generieren Oracle DDL für die importierten Tabellen Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 16

17 Neue Daten-Quellen für Oracle External Tables CREATE TABLE movielog (click VARCHAR2(4000)) ORGANIZATION EXTERNAL ( TYPE ORACLE_HIVE DEFAULT DIRECTORY Dir1 ACCESS PARAMETERS ( com.oracle.bigdata.tablename logs com.oracle.bigdata.cluster mycluster) ) REJECT LIMIT UNLIMITED Neue Eigenschaften ORACLE_HIVE und ORACLE_HDFS Access Driver Identifizuierung Hadoop Cluster, Data Source Objects, Column Mapping, Error Handling, Overflow Handling, Logging Neue Tabellen Metadaten werden von Oracle DDL zu den Hadoop Lese-Routinen zur Query-Zeit geschickt Auf Erweiterung hin entwickelt StorageHandler wird künftig auch weitere Quellen-Arten lesen können Z. B.: MongoDB, HBase, Oracle NoSQL DB Copyright 2014 Oracle and/or its affiliates. All rights reserved. 17

18 Intelligente Query Optimierung Eine Abfrage kann sich über die Oracle Databank, Hadoop & NoSQL erstrecken Query Data in RDBMS, Hadoop & NoSQL Oracle SQL Fast Massive Parallelisierung Lokale Filterung Minimierte Daten-Bewegung Oracle NoSQL DB BDS Server Oracle NoSQL DB BDS Server HDFS Data Node BDS Server HDFS Data Node BDS Server Oracle Database Storage Server Oracle Database Storage Server Copyright 2014 Oracle and/or its affiliates. All rights reserved.

19 Daten in Hadoop Example: 10GB File Hadoop/NoSQL Ecosystem {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:07","recommended":null,"activity":8} {"custid": ,"movieid":1948,"genreid":9,"time":" :00:00:22","recommended":"n","activity":7} {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:26","recommended":null,"activity":9} Block B1 {"custid": ,"movieid":11547,"genreid":44,"time":" :00:00:32","recommended":"y","activity":7} {"custid": ,"movieid":11547,"genreid":44,"time":" :00:00:42","recommended":"y","activity":6} {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:43","recommended":null,"activity":8} {"custid": ,"movieid":null,"genreid":null,"time":" :00:00:50","recommended":null,"activity":9} {"custid": ,"movieid":608,"genreid":6,"time":" :00:01:03","recommended":"n","activity":7} Block B2 {"custid": ,"movieid":null,"genreid":null,"time":" :00:01:07","recommended":null,"activity":9} {"custid": ,"movieid":27205,"genreid":9,"time":" :00:01:18","recommended":"y","activity":7} {"custid": ,"movieid":1124,"genreid":9,"time":" :00:01:26","recommended":"y","activity":7} {"custid": ,"movieid":16309,"genreid":9,"time":" :00:01:35","recommended":"n","activity":7} {"custid": ,"movieid":11547,"genreid":44,"time":" :00:01:39","recommended":"y","activity":7}} Block B3 {"custid": ,"movieid":424,"genreid":1,"time":" :00:05:02","recommended":"y","activity":4} 1 block = 256 MB Example File = 40 blocks InputSplits = 40 Potential parallelism on the BDA Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 19

20 Verbesserte Oracle External Table Performance CREATE TABLE movieapp_log_json (click VARCHAR2(4000)) ORGANIZATION EXTERNAL (TYPE ORACLE_HIVE DEFAULT DIRECTORY DEFAULT_DIR ) PARALLEL 20 REJECT LIMIT UNLIMITED; Früher waren External Tables file-centric 1 file == 1 unit of parallelism Jetzt verstehen External Tables Parallelisierung Automatisches Mapping External Units Of Parallelism zu Oracle Granules 1 Input Split == 1 Oracle Granule Copyright 2014 Oracle and/or its affiliates. All rights reserved. 20

21 Query Abarbeitung auf Hadoop select last_name, state, movie, genre from movielog m, customer c where genre= comedy and c.custid = m.custid 1 Query compilation determines: Data locations Data structure Parallelism 3 2 Parallel reads using Big Data SQL Server: Parallel unit: PQ Slaves & InputSplits Filter rows and project columns HDFS NameNode 1 Hive Metastore HDFS Data Node BDS Server 2 HDFS Data Node BDS Server 3 Process filtered result Move relevant data to database Join with database tables Apply database security policies Copyright 2014 Oracle and/or its affiliates. All rights reserved.

22 Big Data SQL Server Minimizes Data Movement Big Data SQL Server Hadoop Smart Scan 1. Apply filter predicates 2. Apply column projections 3. Apply row-level functions JSON Parsing External Table Services 1. Read using Hadoop Classes 2. Convert to Oracle Data Stream Data Node Direktes Arbeiten auf den Daten Scans and serializations from Hadoop classes Transformation into Oracle data stream Smart Scan: Nur relevante Daten werden werden weitergeleitet Apply filter predicates Include complex predicates, e.g. JSON_EXISTS Bloom filters for faster joins Score Data Mining models Project columns Include projections from nested structures Disk Copyright 2014 Oracle and/or its affiliates. All rights reserved. Oracle Confidential Internal/Restricted/Highly Restricted 22

23 Alle Daten ohne Konvertierung abfragen Select customers_document.address.state, revenue from customers, sales where customers_document.id=sales.custid group by customers_document.address.state; JSON Verarbeitung auf Hadoop JSON auflösen Spalten auswählen Bloom Filter für schnellere Joins JSON JSON Daten in nicht konvertierter Form in Hadoop speichern Geschäftskritsiche Daten in Oracle speichern (JSON oder relational) Copyright 2014 Oracle and/or its affiliates. All rights reserved. 23

25 Komplettes Set des Oracle SQL über alle Daten hinweg nutzen Ausschnitt Oracle SQL Analytische Funktionen Ranking functions rank, dense_rank, cume_dist, percent_rank, ntile Window Aggregate functions (moving and cumulative) Avg, sum, min, max, count, variance, stddev, first_value, last_value LAG/LEAD functions Direct inter-row reference using offsets Reporting Aggregate functions Sum, avg, min, max, variance, stddev, count, ratio_to_report Statistical Aggregates Correlation, linear regression family, covariance Linear regression Fitting of an ordinary-least-squares regression line to a set of number pairs. Frequently combined with the COVAR_POP, COVAR_SAMP, and CORR functions Descriptive Statistics DBMS_STAT_FUNCS: summarizes numerical columns of a table and returns count, min, max, range, mean, stats_mode, variance, standard deviation, median, quantile values, +/- n sigma values, top/bottom 5 values Correlations Pearson s correlation coefficients, Spearman's and Kendall's (both nonparametric). Cross Tabs Enhanced with % statistics: chi squared, phi coefficient, Cramer's V, contingency coefficient, Cohen's kappa Hypothesis Testing Student t-test, F-test, Binomial test, Wilcoxon Signed Ranks test, Chi-square, Mann Whitney test, Kolmogorov-Smirnov test, One-way ANOVA Distribution Fitting Kolmogorov-Smirnov Test, Anderson-Darling Test, Chi-Squared Test, Normal, Uniform, Weibull, Exponential Copyright 2014 Oracle and/or its affiliates. All rights reserved.

26 } next = linenext.getquantity(); if (!q.isempty() && (prev.isempty() (eq(q, prev) && gt(q, next)))) { state = "S"; return state; } if (gt(q, prev) && gt(q, next)) { state = "T"; return state; Pattern } Matching mit Oracle SQL Ausschnitt Oracle SQL Analytische Funktionen if (lt(q, prev) && lt(q, next)) { state = "B"; return state; } if (!q.isempty() && (next.isempty() (gt(q, prev) && eq(q, next)))) { state = "E"; return state; Simplified, } sophisticated, standards based syntax if (q.isempty() eq(q, prev)) { state = "F"; return state; } Beispiel finden von Mustern in Aktien-Markt-Bewegung (W-Form) Ticker 10:00 10:05 10:10 10:15 10:20 10:25 } return state; private boolean eq(string a, String b) { if (a.isempty() b.isempty()) { return false; } return a.equals(b); } private boolean gt(string a, String b) { if (a.isempty() b.isempty()) { return false; } return Double.parseDouble(a) > Double.parseDouble(b); } private boolean lt(string a, String b) { if (a.isempty() b.isempty()) { return false; } return Double.parseDouble(a) < Double.parseDouble(b); } public String getstate() { return this.state; } } BagFactory bagfactory = public Tuple exec(tuple input) throws IOException { SELECT first_x, last_z FROM ticker MATCH_RECOGNIZE ( PARTITION BY name ORDER BY time MEASURES FIRST(x.time) AS first_x, LAST(z.time) AS last_z ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.time - FIRST(x.time) <= 7 )) long c = 0; String line = ""; String pbkey = ""; V0Line nextline; V0Line thisline; V0Line processline; V0Line evalline = null; V0Line prevline; boolean nomorevalues = false; String matchlist = ""; ArrayList<V0Line> linefifo = new ArrayList<V0Line>(); boolean finished = false; 250+ Lines of Java UDF 12 Lines of SQL 26 DataBag output = bagfactory.newdefaultbag(); if (input == null) { return null; } if (input.size() == 0) { return null; } Object o = input.get(0); if (o == null) { return null; } Copyright 2014, Oracle and/or its affiliates. All rights reserved. //Object o = input.get(0); if (!(o instanceof DataBag)) { int errcode = 2114; 20x mal weniger Code Copyright 2014 Oracle and/or its affiliates. All rights reserved.

27 Security Schutz auf alle Daten Oracle Big Data Appliance Oracle Database 12c SQL Store JSON data unconverted in Hadoop Store business-critical data in Oracle Data analyzed via SQL Apply advanced security on Hadoop Masking/Redaction Virtual Private Database Fine-grained Access Control DBMS_REDACT.ADD_POLICY( object_schema => 'hr', object_name => 'employee', column_name => 'social_sec_num', policy_name => 'redact_ssn', function_type => DBMS_REDACT.FULL, expression => '1=1' ); Copyright 2014 Oracle and/or its affiliates. All rights reserved. 27

28 Zwei Level des Zugriffs-Monitoring 1) Standard Auditing 2) Fine Grained Auditing By session By access Statement level. all statements in a sessions any objects X X Granting Priviliges activities X X Object level (update, insert delete, select) X X Monitoring of all activities by User by session, by object. - Action based auditing (e. g. analysing the predicate example: value > ) - all detailed commands - Timestemps -trials and successes - Additional triggers - notifications and other actions Audit select on F_UMSATZ; EXECUTE DBMS_FGA.ADD_POLICY( - object_schema => 'DWH, - object_name => 'F_UMSATZ, - policy_name => 'Chk_F_Umsatz', - statement_types => 'SELECT, - enable => TRUE); Copyright 2014 Oracle and/or its affiliates. All rights reserved. 28

29 Fine Grained Security on Row level User: DataMart1 User_Table User 2 User 3 Build Predicate Procedure C o n t e x t Set User Context select sum(umsatz) from umsatz; Set User Context select sum(umsatz) from umsatz; Partner Catalog Internet Direct Sales Add / DROP Policy Channels Umsatz Kunde P r e d i c a t e Sieht nur Partner-Umsatz Sieht nur Katalogwaren-Umsatz Zeit Produkte Copyright 2014 Oracle and/or its affiliates. All rights reserved. DATA WAREHOUSE 29

30 Advanced Query & Analysis Full Power of SQL and Advanced Analytics Leverages All Your Data Relational, Hadoop and NoSQL Secure Unified Governance on All Data Fastest Performance Utilize SQL Processing Across the Platform Transparent to Applications No Changes to Application Code Big Data Management System Copyright 2014 Oracle and/or its affiliates. All rights reserved.

33 Oracle R Enterprise User R Engine Database Server Maschine R Engine(s) managed by Oracle DB R Engine Other R packages Oracle R Enterprise packages SQL Results Oracle Database R-Skripte Lineare Modelle Clusterung Segmentierung Neuronale Netze R Results R Engine Other R packages Oracle R Enterprise packages Hadoop Cluster (BDA) Copyright 2014 Oracle and/or its affiliates. All rights reserved.

34 HCache Hadoop Abstraction Layer Oracle R Advanced Analytics for Hadoop R script Hadoop Cluster (BDA) ORD R Client {CRAN packages} Hadoop Job Mapper Reducer R HDFS R MapReduce R Hive R sqoop/olh MapReduce Nodes {CRAN packages} ORD HDFS Nodes Oracle Database Copyright 2014 Oracle and/or its affiliates. All rights reserved.

35 corpus <- scan("corpus.dat", what=" ",quiet= TRUE, sep="\n") corpus <- " ", corpus) input <- hdfs.put(corpus) res <- hadoop.exec(dfs.id = input, mapper = function(k,v) { x <- strsplit(as.character(v[[1]]), " ") x <- unlist(x) x <- x[x!=''] orch.keyvals(x,rep(1,length(x))) }, reducer = function(k,vv) { orch.keyval(k, sum(vv$val)) }, config = new("mapred.config", job.name = "wordcount", map.output = data.frame(key='a', val=0), reduce.output = data.frame(key='a', val=0), reduce.tasks = 30) ) res hdfs.get(res) Copyright 2014 Oracle and/or its affiliates. All rights reserved.

36 Function ORAAH Analytics Description Functions orch.cor Correlation matrix computation orch.cov orch.kmeans orch.lm orch.lmf orch.neural orch.nmf orch.princomp orch.sample Covariance matrix computation Perform k-means clustering on a data matrix stored as an HDFS file. Score data using orch.predict. Fits a linear model using tall-and-skinny QR (TSQR) factorization and parallel distribution. The function computes the same statistical parameters as the Oracle R Enterprise ore.lm function. Score data using orch.predict. Fits a low rank matrix factorization model using either the jellyfish algorithm or the Mahout alternating least squares with weighted regularization (ALS-WR) algorithm. Provides a neural network to model complex, nonlinear relationships between inputs and outputs, or to find patterns in the data. Score data using orch.predict. Provides the main entry point to create a nonnegative matrix factorization model using the jellyfish algorithm. This function can work on much larger data sets than the R NMF package, because the input does not need to fit into memory. Principal components analysis of HDFS data. Score data using orch.predict. Sample HDFS data by percentage or explicit number of rows specification Copyright 2014 Oracle and/or its affiliates. All rights reserved.

38 Nicht nur die Quellen ändern sich Service Servicekunde Logistik Logistikaufwand Controlling Profitabilität Einkauf Produkte&Trends Vertrieb Kundenhistorie Marketing Marketingsicht Log Files Web-Clicks Mails Call-Center Verträge Berichte Integration Layer HDFS / NoSQL Enterprise Layer Core - DWH / Info Pool Referenzdaten (extern) (10%) Stammdaten (10%) Bewegungsdaten / Transaktionen (80%) Marketingsicht Sozialsituation Kundenhistorie Profitabilität Kunde Logistikaufwand für einen Kunden Produkte&Trends Servicekunde Unstructured Data User View Layer Geschäfts- Objekte Kennzahlen Multidimensionale Sichten Themenbezogene Sichten Top 100 unternehmensweite Kennzahlen Es geht um Gesamtsichten Top 5 Geschäftsfelder Profitabilität eines Kunden Profitabilität eines Produktes über den gesamten Lebenszyklus Bedarfsprognose von sozialen Gruppen Aktualität und Entwicklung des eigenen Portfolios Eigene Marktstärke im Vergleich zu den Wettbewerbern Copyright 2014 Oracle and/or its affiliates. All rights reserved.

39 Szenario DWH/ In-Memory / BigData OLTP Systeme Disk-Daten In Memory Integration Layer Temporäre Daten T T Referenzdaten Stammdaten Enterprise Layer Core - DWH / Info Pool R Partitionierte Transaktionsdaten nur wenn sie abgefragt werden R R S S S User View Layer Vorberechnete Kennzahlen Alle Dimensionen Bei sehr großen Dimensionen nur die am häufigsten genutzten Spalten Die aktuellen Partitionen In-Memory Operational Data C A MJ Q L Data Mining Statistikdaten A HDFS / NoSQL Unstructured Data Oracle R Copyright 2014 Oracle and/or its affiliates. All rights reserved.

40 Szenario DWH/ In-Memory / BigData OLTP Systeme Disk-Daten In Memory Integration Layer Temporäre Daten T T Referenzdaten Stammdaten Enterprise Layer Core - DWH / Info Pool R Partitionierte Transaktionsdaten nur wenn sie abgefragt werden R R S S S User View Layer Vorberechnete Kennzahlen Alle Dimensionen Bei sehr großen Dimensionen nur die am häufigsten genutzten Spalten Die aktuellen Partitionen In-Memory Operational Data C A MJ Q L Data Mining Statistikdaten A HDFS / NoSQL Unstructured Data Oracle R Copyright 2014 Oracle and/or its affiliates. All rights reserved.

41