BARC T1: Auswirkungen der Digitalisierung auf die Analytische Infrastruktur München, 23.06.2015 Otto Görlich, Senior Analyst Timm Grosser, Senior Analyst
Business Application Research Center (BARC) B Europas führendes IT-Analysten- und -Beratungshaus für Business Software und IT Services (Analystengruppe CXP / PAC / BARC) A R 140 Mitarbeiter, davon 80 Analysten an 17 Standorten in acht Ländern Portfolio aus Research, Beratung und Weiterbildung C Themen: Business Intelligence, Big Data, Datenmanagement, Customer Relationship Management, Enterprise Content Management, IT-Management, HR, Finance, ERP, IT Sourcing und IT Services 26.06.2015 BARC 2015 2
BARC: Expertise für datengetriebene Unternehmen Beratung Strategie & Organisation Prozesse & IT- Architektur Softwareauswahl Data Science Weiterbildung Konferenzen Seminare Kamingespräche Expertenworkshops Datengetriebene Unternehmen Research Produktvergleiche Marktforschung BI Manager 26.06.2015 BARC 2015 3
Was uns unterscheidet Im Vergleich zu anderen Analysten-Häusern Tiefgreifendes Technologie-Verständnis: Eigenes Test-Labor, Begleitung von Proof of Concepts, QS im Rahmen der Software-Einführung oder als Review Große Endkunden-Surveys, z. B. The BI Survey mit >3000 Teilnehmern p.a. Echte Projekt-Erfahrung: Unsere Analysten sind auch als Consultants in Strategie-, Architektur- und Softwareauswahl-Projekten aktiv Fokus auf Anwender und Anwendung von Business Applications U.a. detailliertes SAP-Know-how Im Vergleich zu System-Integratoren Unabhängigkeit und Neutralität Fokus auf Strategie und Architektur Kein Interesse an Implementierungsprojekten, kein Software-Verkauf Breites Markt-Wissen: Alleine im BI & DM Markt kennen wir 250 Anbieter mit 600 Werkzeugen u. aggregieren Erfahrungen von mehreren tausend Anwendern 26.06.2015 BARC 2015 4
Agenda Impact of digitalization on the analytical infrastructure Trends and developments for BI and Big Data Modern architecture concepts and technologies 26.06.2015 BARC 2015 5
Ready for the future? 26.06.2015 BARC 2015 6
Zwei wesentliche Trends in IT und BI sind die Treiber zur Weiterentwicklung der Analytischen Infrastruktur Consumerization Analytische Infrastruktur Digitalisierung 26.06.2015 BARC 2015 7
Consumerization - Die Erwartungshaltung und das Verhalten der Anwender ändert sich Performance, Skalierbarkeit und Anwenderoberfläche Ständige Informationsverfügbarkeit und Informationsnutzung, Enge Zusammenarbeit/Austausch Nutzerinteraktion, Informationsvisualisierung, Operationale Integration, Self- Service, 26.06.2015 BARC 2015 8
Zwei wesentliche Trends in IT und BI sind die Treiber zur Weiterentwicklung der Analytischen Infrastruktur Consumerization Analytische Infrastruktur Digitalisierung 26.06.2015 BARC 2015 9
Datenkategorien und ihre Eigenschaften Source data category Transactional business data Machinegenerated Humangenerated Quality Complexity Interpretability Noise Data warehouse process-affinity Big data process-affinity 26.06.2015 BARC 2015 10
Vermehrte Nutzung von nicht nur strukturierten Daten zur Steuerung des Unternehmens Daten aus Transaktionssystemen 35% 48% Maschinen-Daten von IT- Systemen (Log-Daten) Maschinen-Daten, z.b. aus der Produktion (BDE) 15% 20% 32% 45% Web-Log/Web-Analysedaten 14% 44% +30% Dokumente/Texte 14% 50% +36% Social-Media-Daten 9% 47% +38% Event-Streams 7% 28% Sensor-Daten (z.b. RFID) 6% 27% Video-/Bild-Daten 2% 23% Im Einsatz Geplant Q: BARC Big Data Analytics Survey 2014, n = 212 26.06.2015 BARC 2015 11
Die Bedeutung von Daten war nie größer als heute 54% der Befragten sehen Daten zukünftig als Vermögenswert BARC Information Culture Survey 2014, n: 337 26.06.2015 BARC 2015 12
Drei Aspekte der Nutzung von Daten im Geschäftskontext Mehr Daten und umfangreichere Verwendung (bspw. durch neue Technologien) Hohe Marktdynamik und verschärfter Wettbewerb (bspw. Produktstandardisierung) Steuerung Automatisierung Innovation 26.06.2015 BARC 2015 13
IT Metatrends Digitalisierung Consumerisation Agilität und Kosteneffizienz Sicherheit und Datenschutz Cloud Computing und Virtualisierung Geschäftsmodelle der IT-Anbieter BI and Data Management Metatrends Marktreife und Commoditisation Fachkräftemangel Prozessorientierung 26.06.2015 BARC 2015 14
BI und Data Management Trends Wachsendes Interesse Early Mover Akzeptanz Heiße Diskussion Breite Relevanz Laufende Diskussion Nicht hype, aber relevant Big Data Analytics Predictive Analytics Collaborative BI Search BI Streaming/Real Time Analytics SaaS und Cloud für BI & DM Big Data Data Management Self Service Data Management Data Storytelling Data as a product / Open Data Erweiterte Rolle des Business Analysten Self Service BI Visual Analysis & Data Discovery Advanced Planning Hadoop Technologie Analytische Datenbanken & In-Memory Computing Datennutzung Information Design Organisation Data Governance BI Organisation 2.0 Data Labs & Data Science Integrierte Plattformen für BI und Performance Management Spatial Intelligence Data Integration Stammdaten- und Datenqualitätsmanagement Mobile BI Agile BI Development 26.06.2015 BARC 2015 15
Ausblick wesentliche Entwicklungen für Big Data? Strategie, Organisation und Governance Analytics i.s.v. Anwenderwerkzeuge Datenmanagement operative Systeme / weitere Datenquellen Strategie, Organisation und Governance Verstärkt Ausbau von Data Science-Teams Vermehrt Konzeption von Datenstrategien und Governance Anpassung von Organisation (Rollen, Verantwortung), insb. auch agilen Vorgehensmodellen zur Lösungsentwicklung Betrieb: Wachsender Bezug von Analysen und Daten über Cloud- Services Analytics Höhere Benutzerfreundlichkeit und Beziehbarkeit von Anwendungen durch zunehmende Kapselung von Komplexität in Software Wachsendes Angebot von Standardanwendungslösungen, insb. für Predictive Analytics (Pricevorhersage, Kundenklassifizierung, Fraud) Weiterhin hohe und wachsende Relevanz von Self-Service Datenmanagement Anpassung analytischer Infrastrukturen unter Berücksichtigung innovativer Technologien (neue Architekturen, logischer SPoT) Vermehrt Einsatz von Big Data Technologien Wachsende Rolle der Datenintegration und verarbeitung Wachsende Relevanz von Self-Service (siehe auch Analytics) Operative Systeme Vermehrt Einbettung von Analysefunktionen in den operativen Prozessen Höhere Akzeptanz und vermehrte Nutzung weiterer Datenquellen (IoT, Open Data, Social Media, ) 26.06.2015 BARC 2015 16
Status quo 26.06.2015 BARC 2015 17
Der Nutzen einer Datenstrategie ist klar, Quelle: BARC Survey Datenmanagement im Wandel, Infografik 26.06.2015 BARC 2015 18
, doch Unternehmen handeln noch nicht entsprechend Quelle: BARC Survey Datenmanagement im Wandel, Infografik 26.06.2015 BARC 2015 19
Realität DWH-Architektur Komplexität: Wartung, Weiterentwicklung, Nutzung Fehlende Flexibilität: neue Daten, neue Analysen Unzureichende Abfragezeiten Unbefriedigende Aktualisierung: Zyklen und Zeiten Kostenintensiv Unzureichende funktionale Abdeckung 26.06.2015 BARC 2015 20
Die Bereitschaft zur Änderung ist da und das Datenmanagement im Wandel 26.06.2015 BARC 2015 21
Wie wichtig sind die folgenden Themen für Ihr Unternehmen? Datenintegration 42% 54% 4% Data Warehouse 25% 63% 11% Datenarchitektur 22% 68% 10% Big Data 6% 28% 66% Kritisch Wichtig Nicht so wichtig Quelle: BARC Survey Datenmanagement im Wandel, n=339 26.06.2015 BARC 2015 22
Insbesondere die Positionierung zu Big Data wird zunehmend wichtiger für das Datenmanagement Datenintegration 58% 42% 1% Data Warehouse 46% 50% 4% Spürbare wachsende Bedeutung Datenarchitektur 51% 48% 1% Big Data 53% 42% 6% Wird wichtiger Gleichbleibend Wird unwichtiger Quelle: BARC Survey Datenmanagement im Wandel, n=340 26.06.2015 BARC 2015 23
Was verändert sich in Ihrem Unternehmen konkret bzw. wird sich konkret verändern bei den Themen Datenintegration und Data Warehousing? Zunehmende Integration von heterogenen Datenquellen Befähigung der Fachbereich, Daten selbstständig zu integrieren/auszuwerten Nutzung neuer Technologien, um die Komplexität der Datenarchitektur zu reduzieren Versuch der Reduzierung des ressourcenbindenden Aufwands Nutzung einer Datenstrategie/Governance Einsatz neuer Technologien/Methoden zur Abbildung eines Unternehmensgedächtnis Zunehmender Einsatz von Hilfsmittel zur einfachen Integration neuer Datenquellen Zunehmender Einsatz von Big-Data-Technologien 22% 27% 31% 43% 49% 53% 53% 69% Rückgriff auf neue Datenarchitekturansätze Auslagerung/paralleler Betrieb von einzelnen Funktionsbereichen des Data Warehouse Rückgriff auf ein cloud-basiertes Date Warehouse Sonstige 6% 5% 14% 12% Quelle: BARC Survey Datenmanagement im Wandel, n=322 26.06.2015 BARC 2015 24
Neue Architekturen sind erforderlich 26.06.2015 BARC 2015 25
Agenda Impact of digitalization on the analytical infrastructure Trends and developments for BI and Big Data Modern architecture concepts and technologies 26.06.2015 BARC 2015 26
Mind shift through Big Data How Big Data approaches change the behavior to analyze information and to transform insights into action? 26.06.2015 BARC 2015 27
Mind Shift Big Data: Analysis Process, Data Persistency and Data Volumne regulated analysis Data Warehousing process explorative analysis Big Data process Visualization Known questions, protected results Visualization Unknown question, unprotected results ( fast fail ) Analytics Analytics Data preparation Data interpretation Data storage Verify first, then collect Data preparation Data interpretation Data storage Collect first, then verify Data provisioning Data provisioning 26.06.2015 28
Mind shift by Big Data requires an expansion of the existing infrastructure/architecture to meet the requirements to satisfy 26.06.2015 BARC 2015 29
IQ Services DP Services Classic Data Warehouse Concept of the 1990s The typical environment for the management of enterprise data Core Processes: ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Tactical BI Data Mart Data Mart Data Mart Data Mart Data Distribution Conformed Facts Conformed Dimensions Data Distribution Data Warehouse Data Integration Staging Area Extraction ERP SCM CRM Business transaction data (structured) External systems 26.06.2015 BARC 2015 30
How companies transform their data and analysis environments, so that information can be provided better, faster and more cost-effective? 26.06.2015 BARC 2015 31
IQ Services DP Services New Data Sources, Technologies, and Features Core Processes: ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Explorative BI: Advanced Analytics, Extended Data Reporting & dashboards Analysis Search & discover Predictive modeling Text analysis Tactical BI Explorative BI Data Mart SSBI/DI Data Mart Data Mart Data Mart Expl. SSBI/DI Data Distribution Analytical DB Conformed Conformed Facts Dimensions Data Distribution Analytical DB Data Warehouse Data Integration Hadoop Staging Area Extraction Search Index Catalog NoSQL DB Hadoop ERP SCM CRM Business transaction data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) 26.06.2015 BARC 2015 32
IQ Services DP Services And Operational BI and Big Data Integration Core Processes: ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Explorative BI: Advanced Analytics, Extended Data Reporting & dashboards Analysis Search & discover Predictive modeling Text analysis Operational BI Tactical BI Explorative BI CEP Data Mart SSBI/DI Data Mart Data Mart Data Distribution Data Mart Expl. SSBI/DI Analytical DB Analytical DB Conformed Conformed Facts Dimensions Data Distribution Analytical DB Search Index Catalog Streaming NoSQL DB Data Warehouse Data Integration Hadoop Staging Area Extraction NoSQL DB Hadoop Analytical DB, Hadoop, Integrated Data Virtualization, logical business Data Integration model Services, Cloud ERP SCM CRM Business transaction data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) 26.06.2015 BARC 2015 33
Data Lake Concept and new terms are added Reporting & dashboards CEP Streaming Engine ERP SCM CRM Business transaction data (structured) Analysis Search & discover Predictive modeling Text analysis SSBI SSDI External systems Data Lake Analytical Database File System Sensor data Data Lake Concept Concept was introduced by Hadoop Distributors (eg Explor Pivotal (EMC), -ation Horton Works) A mainly on Hadoop based concept with the aim to collect or to store Search data of NoSQL any type in the Index, Hadoop file DB system Catalog Evaluation and transmission of the data to other infrastructure elements (repositories), if necessary Web log data Machine-generated data (struct.) File System Strict data governance necessary - otherwise the Data Lake easily will be a data swamp Social media Documents Human-generated (polystr.) 26.06.2015 BARC 2015 34
Big Data Lakes or Swamps? As we collect data Can we preserve clarity? Do we know what we are collecting? Can we find the data we need? Are we creating a data swamp? How do we build trust in big data? Do we know what data is being used for? Solution: Data Integration and Governance provides you with an Information Fabric to turn a swamp into valuable supervised Data Lake 26.06.2015 BARC 2015 35
What is a supervised Data Lake? A supervised data Lake is an information environment that provides data to an organization for a variety of analytics processing It is possible to deploy analytics into the supervised Data Lake to generate additional insight from the data loaded into the supervised Data Lake A supervised Data Lake manages shared repositories of information for analytical purposes Data values may be replicated in multiple repositories in the supervised Data Lake. However the supervised Data Lake ensures the copying and updating of this data is managed and governed using well-defined information supply chains Supervised Data Lake = Efficient Management, Governance, Protection and Access 26.06.2015 BARC 2015 36
Four main disciplines around the Data Lake 1 Act on business transactions move changed or new data to the Data Lake act on events immediately 2 Explore raw data in the Data Lake advanced analytics on top of Hadoop 3 Analyze refined data in-memory databases for fast access on subject oriented data 4 Refine and Govern the Data Lake enable flexible forms of data movement and data transformation guarantee data quality government of all processes in and around the Data Lake 26.06.2015 BARC 2015 37
The support of operational processes through real-time analytics requires analysis close to the operational systems Mapping the disciplines of the Data Lake Architecture - Acting (1) Analysis (2), Exploration (3) Refine and Govern (4) Operational BI: Core Processes ERP, CRM, SCM, OLTP Reporting & dashboards CEP Streaming Engine 1b 1a 2 RDBMS Data Virtualization, Data Integration Services, Cloud Refined Data (DWH, Data Marts, etc.) ERP SCM CRM Business transaction data (structured) Classic BI: Reporting, Dashboards, OLAP Analysis Search & discover Predictive modeling Text analysis SSBI SSDI External systems File System 2 Analytical Database Sensor data Web log data Machine-generated data (struct.) Acting on (Business) Transactions Explorative BI: Advanced Analytics, Extended Data 1a Transactional Systems Data Movement from Explor operationale -ation Systems (e.g. RDBMS) via (real-time) Data Integration 1b Streaming Analytics NoSQL Stream- DB and CEP Processing File System Social media Search Index, Catalog Analysis of refined data Documents Human-generated (polystr.) 26.06.2015 BARC 2015 38
Data Governance Complement the architecture / infrastructure through an exploratory environment and governance Mapping the disciplines of the Data Lake Architecture - Acting (1) Analysis (2), Exploration (3) Refine and Govern (4) Operational BI: Core Processes ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Explorative BI: Advanced Analytics, Extended Data Reporting & dashboards Refine and Govern the CEP Data Lake Streaming 4a Refine Engine Data Integration, MeDM 4b Govern Catalog, MeDM, Governance Processes, etc. 4a ERP SCM CRM Business transaction data (structured) Analysis Search & discover Predictive modeling Text analysis SSBI Exploration of raw data 3a Hadoop Data Storage 3b Analysis External systems File System Analytical Database Iterative / exploratory SSDI analysis of data Sensor data Explor -ation Web log data Machine-generated data (struct.) NoSQL DB Data Virtualization, Data Integration Services, Cloud 3b File System Social media 3a Search Index, Catalog Documents Human-generated (polystr.) 4b 26.06.2015 BARC 2015 39
The Data Lake is technology, vendor and deployment agnostic Typically a mix of technology and vendors, cloud and on premise Technologies It is key to use the best fitting technology for a given task Each technology has its strength in different areas, e.g. Hadoop stores any data in any format, offers flexible analytics, at low price per TB In-memory databases offer best query response times and high compression rates Vendors Glue between technologies is the use of open standards (e.g. SQL) Vendors that support open standards can be integrated easily Cloud / IaaS / PaaS / on-premise The decision for deployment of technologies is not primarily driven by technology but by ease of use, flexibility/agility, privacy concern, compliance, cost structure, 26.06.2015 BARC 2015 40
Data storage options for the analytical infrastructure 26.06.2015 BARC 2015 41
The two Worlds of Big Data Data in Motion Data at Rest Steaming/CEP Proprietary Operations SW Process Manufacturing Fraud and Risk Calculations High Frequency Trading Etc. Architecture Data Warehouses Hadoop, NoSQL Data Stores, Relational Databases OLTP Systemes CRM, ERP,... Real-time Business Intelligence 26.06.2015 BARC 2015 42
Different Data Different Problems Data at Rest Main Problem: Storage Cost Curve Secondary Problem: Ability to Act on large Data Sets Data in Motion Real-Time Data Main Problem: Distributed Data Secondary Problem: Ability to Act on large Data Sets Streaming Data Main Problem: Unconnected Data Secondary Problem: Large number of stand-alone action points 26.06.2015 BARC 2015 43
Data at Rest Storage Decision Drivers where to store which data? Metric Cost Security Performance Description Cost of Data Operations/Skills Sensitivity of Data Regulatory Requirements Raw Query Performance Concurrency 26.06.2015 BARC 2015 44
Example (I) Data: System and Web etc. Log Files Performance-Metric: Need to ingest massive quantities quickly and continuously No single record queries required, mostly aggregations Security-Metric: No PII (Personally Identifiable Information) like Information All Users Cost-Metric: Low cost required Loosing data is ok (tolerable) Hadoop 26.06.2015 BARC 2015 45
Example (II) Data: Manufacturing Sensor Data Performance-Metric: Need to ingest massive quantities quickly and continuously Need to see operational status across sensors Security-Metric: No PII like Information All Users Cost-Metric: Low cost required Support low cost long retention periods NoSQL Hadoop 26.06.2015 BARC 2015 46
Example (III) Data: Web Store Orders / Purchases Performance-Metric: Relatively low volume when compared to activity logs Need to see and act on individual transactions Security-Metric: Contains sensitive data (or links to it) Requires restricted access to some of the data Cost-Metric: Business critical Consistency is most important RDBMS/ NoSQL RDBMS 26.06.2015 BARC 2015 47
Example (IV) Data: Web Store Recommendations (publish) Performance-Metric: High volume of low latency data, but simple requests Need to see individual recommendations Security-Metric: No sensitive data Application level access (automated) Cost-Metric: Business critical low latency requirement Distributed environment NoSQL 26.06.2015 BARC 2015 48
Changes in the architecture through in-memory 26.06.2015 BARC 2015 50
Complement the architecture / infrastructure with in-memory components for operational reporting and analysis Operational BI: Core Processes ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Explorative BI: Advanced Analytics, Extended Data Reporting & dashboards Analysis Search & discover Predictive modeling Text analysis SSBI Explor -ation In-Memory Database RDBMS (operational) SSDI Analytical Database (In-Memory) File System Analytical Database NoSQL DB File System Search Index, Catalog Data Virtualization, Data Integration Services, Cloud ERP SCM CRM Business transaction data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) 26.06.2015 BARC 2015 51
In-memory databases (IMDB) - characteristics Utilization of Hardware Trend - processors are every 18-24 Months twice as fast and memory becomes cheaper in the same proportion Data are permanently in main memory Memory is the primary "persistence layer Still: Logging to disk / recovery from the hard drive Processor cache conscious algorithms / data structures Use of multi-core processors and processor features Column-based data storage with a high compression ratio (memory is the limiting factor) Memory-optimized algorithms and processing 26.06.2015 BARC 2015 52
Areas of application of In-Memory in the analytical infrastructure for Big Data Operational BI: Core Processes ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Explorative BI: Advanced Analytics, Extended Data Reporting & dashboards SSBI Core Data Warehouse Faster response times Reduced data latency No/less Performance Objects No/less aggregates/summaries In-Memory Database RDBMS (operational) Reporting directly on the operative data No ODS required Faster response times (near)real-time Low data latency Data Virtualization, Data Integration Services, Cloud ERP SCM CRM Business transaction data (structured) Analysis Search & discover Predictive modeling Text analysis SSDI File System External Faster Sensor data systems response times Web log data Machine-generated data (struct.) File System Explor -ation Data Marts Faster response times No/less Performance NoSQL Objects DB Only virtual data marts Analytical Database Analytical (In-Memory) Database Operational Data Store (Near Real Time) Reduced data latency No Performance Objects Social media Search Index, Catalog Spark - In-Memory for ML/Iiterative data exploration Faster response times Reduced data latency Documents Human-generated (polystr.) 26.06.2015 BARC 2015 53
Historically Grown DWH Architecture The layered Data Warehouse since the early 90 s Classical BI System of Record Reporting, Dashboards, Analysis Conformed dimensions, lookup tables, Hierarchies, fact tables System of record & history Business Intelligence layer Data Mart layer Data integration layer Data Warehouse layer Data integration layer Complex business Calculations (ETL) Complex business rules & dependencies (ETL) Staging area ERP SCM CRM Business transaction data (structured) External systems 26.06.2015 BARC 2015 54
Advanced DWH Architecture Self Service BI, reduced DI efforts, improved agility, better Performance Reporting, Dashboards, Analysis Business Intelligence layer Virtual Data Marts, conformed dimensions, facts Data Mart layer Views Complex business calculations System of record & history Data Warehouse layer Data integration layer Faster, less complex (batch, mini-batch, continuous ingest) Staging area ERP SCM CRM Business transaction data (structured) External systems Bring business rules closer to the business Improved IT/BICC reaction time Reduced costs and impact on the DWH Higher flexibility in provisioning data to the business 26.06.2015 BARC 2015 55
Extended analytical infrastructure/architecture for Big Data new and expanded applications are possible Customer experience Business optimization & innovation Financial performance & risk management Operations & fraud protection IT cost effectiveness Reporting & dashboards Operational BI Analysis Search & discover Predictive modeling Text analysis SSBI Tactical BI Explor -ation Explorative BI In-Memory Database RDBMS (operational) SSDI Analytical Database (In-Memory) File System Analytical Database NoSQL DB File System Search Index, Catalog Data Virtualization, Data Integration Services, Cloud ERP SCM CRM Business transaction data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) 26.06.2015 BARC 2015 56
Meta Data Data Governance The role of the SPoT in the age of Big Data "to spot or not to spot" Transformation from a single SPoT to a logical SPoT or logical Data Warehouse Customer experience Business optimization & innovation Financial performance & risk management Operations & fraud protection IT cost effectiveness Reporting & dashboards Operational BI Analysis Search & discover Predictive modeling Text analysis SSBI Tactical BI Explor -ation Explorative BI In-Memory Database RDBMS (operational) Analytical Database (In-Memory) Analytical Database Logical Single SPoT SPoT logical SSDIData Warehouse File System NoSQL DB File System Search Index, Catalog Data Integrated Virtualization, logical Data business Integration data Services, model Cloud ERP SCM CRM Business transaction data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) 26.06.2015 BARC 2015 57
Exemplary applications show the use of new capabilities in the extended data management environment 26.06.2015 BARC 2015 58
Comprehensive traffic safety and control system with improved response to accidents Task Cost-effective solution for the improvement of traffic safety and control system Accuracy in the detection of events with consideration of traffic conditions (speed limits) and prediction of arrival of buses Analysis of GPS data whose flow rate is high and their capture difficult Analysis of accident black spots Goals Daily monitoring of 600 buses on 150 routes Analysis of 50 updates per second for the Bus Locations Acquisition, processing and visualization of location data of public transport vehicles 26.06.2015 BARC 2015 59
Infrastructure / Architecture for Traffic Safety and Control System Analyzes in real-time to improve the customer experience/satisfaction Operational BI Customer experience Business optimization & innovation Financial performance & risk management Operations & fraud protection IT cost effectiveness Reporting & dashboards Analysis Search & discover Predictive modeling Text analysis Operational BI SSBI Tactical BI Explor -ation Explorative BI CEP CEP Streaming Engine SSDI Analytical Database (In-Memory) Analytical Database NoSQL DB Search Index, Catalog File System File System Data Virtualization, Data Integration Services, Cloud ERPSensor data SCMWeb log datacrm Machine-generated Business transaction data (struct.) data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) 26.06.2015 BARC 2015 60
More effective email marketing campaigns with Big Data Task Analysis of customer e-mails by service provider - customer of the service provider will receive information on best days and times for their marketing e-mails in order to maximize the return responses. Goals Significant improvement in the analysis depth and scope of the analysis Improve performance of customer e-mail campaigns (better and more focused approach, thus more successful responses) Reduction of the analysis time 26.06.2015 BARC 2015 61
Infrastructure / architecture for e-mail marketing Analysis to maximize the response rates Classic BI Explorative BI Customer experience Business optimization & innovation Financial performance & risk management Operations & fraud protection IT cost effectiveness Reporting & dashboards Analysis Search & discover Predictive modeling Text analysis Operational BI SSBI Tactical BI Explor -ation Explorative BI CEP Streaming Engine SSDI Analytical Analytical Database Database (In-Memory) (In-Memory) Analytical Database NoSQL DB Search Index, Catalog File System File System File System Data Virtualization, Data Integration Services, Cloud External ERP SCM CRM Sensor data Web log data Social media Documents systems Business transaction data (structured) Machine-generated data (struct.) Human-generated (polystr.) 26.06.2015 BARC 2015 62
Reducing energy costs and increasing reliability and performance of the power system Task Assessing the feasibility of a smart grid method with the name "Transactive Control" Goals Involvement of consumers and reactive power assets across the whole power grid for optimizing the system and integration of renewable energies Ability to analyze and extracting information from large volumes of data Higher grid efficiency and reliability by self-monitoring of the system and Feedback capabilities City can prevent potential power failure 26.06.2015 BARC 2015 63
Infrastruktur/Architektur für Smart Grid Analysis to reduce costs and optimize grid Operational BI Classic BI Customer experience Business optimization & innovation Financial performance & risk management Operations & fraud protection IT cost effectiveness Reporting & dashboards Predictive modeling Reporting & Analysis dashboards Search & discover AnalysisPredictive modeling Text analysis Operational BI SSBI Tactical BI Explor -ation Explorative BI CEP CEP Streaming Engine SSDI Analytical Analytical Database Database (In-Memory) (In-Memory) Analytical Database NoSQL DB Search Index, Catalog File System File System Data Virtualization, Data Integration Services, Cloud External External ERP Sensor data SCMWeb log data CRM ERP SCM Sensor datacrm Web log data Social media Documents systems systems Machine-generated Business transaction data (struct.) data (structured) Business transaction Machine-generated data (structured) data (struct.) Human-generated (polystr.) 26.06.2015 BARC 2015 64
The interesting dilemma the context makes the difference A man goes into a jewelers and buys an expensive watch Is it fraud in which case the bank must stop it Is it money-laundering in which case the bank must report it Does he have an expensive trophy wife in which case perhaps he would be interested in a loan? Has he just won the lottery should the bank improve the services offered? Threat Obligation Opportunity The same event is of interest by different departments. There is major overlap in the data required to answer the question. It may not be possible to determine the answer with just the information in the channel - Previous or subsequent activity is required It is all a matter of coordination and timing
Conclusion New technologies and concepts extend the data space for analysis Analysis on poly-structured data and machine-generated data is possible - also in real-time New applications enable extended or new business models Expansion of the traditional DWH (SPoT) to a logical DWH (SPoT) with specialized engines Data integration and governance platforms provide the opportunity to ensure data integrity in the logical DWH 26.06.2015 BARC 2015 66
Conclusion New and performant technologies (eg in-memory) allow more streamlined and simpler DWH architectures Easier data integration, less performance objects, less Layer Operational reporting or analysis directly on the OLTP system by high performance in-memory technologies is possible Questions towards data governance and data quality are still to be resolved 26.06.2015 BARC 2015 67
Ihr Kontakt bei BARC Otto Görlich Senior Analyst BI & Datenmanagement Tel +49 (0) 931-880651-0 ogoerlich@barc.de BARC GmbH Berliner Platz 7 97080 Würzburg www.barc.de @BARC_Research Timm Grosser Senior Analyst BI & Datenmanagement Tel +49 (0) 931-880651-0 tgrosser@barc.de BARC GmbH Berliner Platz 7 97080 Würzburg www.barc.de @BARC_Research 26.06.2015 BARC 2015 68
BARC-Tagung: Data Governance Day 2015 03.September 2015 in Baden (Schweiz) Themen: Datenstrategie und Governance Data Life Cycle Data Driven Organisation Data Integration Data Quality Master Data Management Data Life Cycle Document Management Fachvorträge der BARC-Analysten und Fachverbände Seminardokumentation mit Fachartikeln Herstellerpräsentationen 26.06.2015 BARC 2015 69
BARC-Tagung: Advanced und Predictive Analytics 29. September 2015 in Frankfurt Themenschwerpunkte: Anwendungsgebiete (z.b. Auslastungsoptimierung, Wartung, Forschung, CRM, Social Media) Praxisbeispiele und Erfolgsfaktoren Management des Analytischen Prozesses Marktübersicht Software- und Serviceanbieter Fach-, Praxis-und Produktvorträge zu den verschiedenen Anwendungsmöglichkeiten Herstellerpräsentationen 26.06.2015 BARC 2015 70
BARC Congress Business Intelligence & Datenmanagement am 10.+11. November 2015 in Würzburg Highlights & News: Anbietervorträge, Case Studies, Analystenvorträgen von BARC und PAC, Best Practice Award uvm. Neuer Track mit Fokus auf Datenmanagement und Big Data Videoaufzeichnung aller Vorträge BARC-Service zur Terminvereinbarung im Vorfeld zwischen Teilnehmern und Ausstellern Attraktive Abendveranstaltung mit Verleihung des BI Best Practice Awards Erweiterung der Zielgruppe durch den parallel stattfindenden CRM Summit 26.06.2015 BARC 2015 71