BARC T1: Auswirkungen der Digitalisierung auf die Analytische Infrastruktur

Transkript

1 BARC T1: Auswirkungen der Digitalisierung auf die Analytische Infrastruktur München, Otto Görlich, Senior Analyst Timm Grosser, Senior Analyst

2 Business Application Research Center (BARC) B Europas führendes IT-Analysten- und -Beratungshaus für Business Software und IT Services (Analystengruppe CXP / PAC / BARC) A R 140 Mitarbeiter, davon 80 Analysten an 17 Standorten in acht Ländern Portfolio aus Research, Beratung und Weiterbildung C Themen: Business Intelligence, Big Data, Datenmanagement, Customer Relationship Management, Enterprise Content Management, IT-Management, HR, Finance, ERP, IT Sourcing und IT Services BARC

3 BARC: Expertise für datengetriebene Unternehmen Beratung Strategie & Organisation Prozesse & IT- Architektur Softwareauswahl Data Science Weiterbildung Konferenzen Seminare Kamingespräche Expertenworkshops Datengetriebene Unternehmen Research Produktvergleiche Marktforschung BI Manager BARC

4 Was uns unterscheidet Im Vergleich zu anderen Analysten-Häusern Tiefgreifendes Technologie-Verständnis: Eigenes Test-Labor, Begleitung von Proof of Concepts, QS im Rahmen der Software-Einführung oder als Review Große Endkunden-Surveys, z. B. The BI Survey mit >3000 Teilnehmern p.a. Echte Projekt-Erfahrung: Unsere Analysten sind auch als Consultants in Strategie-, Architektur- und Softwareauswahl-Projekten aktiv Fokus auf Anwender und Anwendung von Business Applications U.a. detailliertes SAP-Know-how Im Vergleich zu System-Integratoren Unabhängigkeit und Neutralität Fokus auf Strategie und Architektur Kein Interesse an Implementierungsprojekten, kein Software-Verkauf Breites Markt-Wissen: Alleine im BI & DM Markt kennen wir 250 Anbieter mit 600 Werkzeugen u. aggregieren Erfahrungen von mehreren tausend Anwendern BARC

5 Agenda Impact of digitalization on the analytical infrastructure Trends and developments for BI and Big Data Modern architecture concepts and technologies BARC

6 Ready for the future? BARC

7 Zwei wesentliche Trends in IT und BI sind die Treiber zur Weiterentwicklung der Analytischen Infrastruktur Consumerization Analytische Infrastruktur Digitalisierung BARC

8 Consumerization - Die Erwartungshaltung und das Verhalten der Anwender ändert sich Performance, Skalierbarkeit und Anwenderoberfläche Ständige Informationsverfügbarkeit und Informationsnutzung, Enge Zusammenarbeit/Austausch Nutzerinteraktion, Informationsvisualisierung, Operationale Integration, Self- Service, BARC

9 Zwei wesentliche Trends in IT und BI sind die Treiber zur Weiterentwicklung der Analytischen Infrastruktur Consumerization Analytische Infrastruktur Digitalisierung BARC

10 Datenkategorien und ihre Eigenschaften Source data category Transactional business data Machinegenerated Humangenerated Quality Complexity Interpretability Noise Data warehouse process-affinity Big data process-affinity BARC

11 Vermehrte Nutzung von nicht nur strukturierten Daten zur Steuerung des Unternehmens Daten aus Transaktionssystemen 35% 48% Maschinen-Daten von IT- Systemen (Log-Daten) Maschinen-Daten, z.b. aus der Produktion (BDE) 15% 20% 32% 45% Web-Log/Web-Analysedaten 14% 44% +30% Dokumente/Texte 14% 50% +36% Social-Media-Daten 9% 47% +38% Event-Streams 7% 28% Sensor-Daten (z.b. RFID) 6% 27% Video-/Bild-Daten 2% 23% Im Einsatz Geplant Q: BARC Big Data Analytics Survey 2014, n = BARC

12 Die Bedeutung von Daten war nie größer als heute 54% der Befragten sehen Daten zukünftig als Vermögenswert BARC Information Culture Survey 2014, n: BARC

13 Drei Aspekte der Nutzung von Daten im Geschäftskontext Mehr Daten und umfangreichere Verwendung (bspw. durch neue Technologien) Hohe Marktdynamik und verschärfter Wettbewerb (bspw. Produktstandardisierung) Steuerung Automatisierung Innovation BARC

14 IT Metatrends Digitalisierung Consumerisation Agilität und Kosteneffizienz Sicherheit und Datenschutz Cloud Computing und Virtualisierung Geschäftsmodelle der IT-Anbieter BI and Data Management Metatrends Marktreife und Commoditisation Fachkräftemangel Prozessorientierung BARC

15 BI und Data Management Trends Wachsendes Interesse Early Mover Akzeptanz Heiße Diskussion Breite Relevanz Laufende Diskussion Nicht hype, aber relevant Big Data Analytics Predictive Analytics Collaborative BI Search BI Streaming/Real Time Analytics SaaS und Cloud für BI & DM Big Data Data Management Self Service Data Management Data Storytelling Data as a product / Open Data Erweiterte Rolle des Business Analysten Self Service BI Visual Analysis & Data Discovery Advanced Planning Hadoop Technologie Analytische Datenbanken & In-Memory Computing Datennutzung Information Design Organisation Data Governance BI Organisation 2.0 Data Labs & Data Science Integrierte Plattformen für BI und Performance Management Spatial Intelligence Data Integration Stammdaten- und Datenqualitätsmanagement Mobile BI Agile BI Development BARC

16 Ausblick wesentliche Entwicklungen für Big Data? Strategie, Organisation und Governance Analytics i.s.v. Anwenderwerkzeuge Datenmanagement operative Systeme / weitere Datenquellen Strategie, Organisation und Governance Verstärkt Ausbau von Data Science-Teams Vermehrt Konzeption von Datenstrategien und Governance Anpassung von Organisation (Rollen, Verantwortung), insb. auch agilen Vorgehensmodellen zur Lösungsentwicklung Betrieb: Wachsender Bezug von Analysen und Daten über Cloud- Services Analytics Höhere Benutzerfreundlichkeit und Beziehbarkeit von Anwendungen durch zunehmende Kapselung von Komplexität in Software Wachsendes Angebot von Standardanwendungslösungen, insb. für Predictive Analytics (Pricevorhersage, Kundenklassifizierung, Fraud) Weiterhin hohe und wachsende Relevanz von Self-Service Datenmanagement Anpassung analytischer Infrastrukturen unter Berücksichtigung innovativer Technologien (neue Architekturen, logischer SPoT) Vermehrt Einsatz von Big Data Technologien Wachsende Rolle der Datenintegration und verarbeitung Wachsende Relevanz von Self-Service (siehe auch Analytics) Operative Systeme Vermehrt Einbettung von Analysefunktionen in den operativen Prozessen Höhere Akzeptanz und vermehrte Nutzung weiterer Datenquellen (IoT, Open Data, Social Media, ) BARC

17 Status quo BARC

18 Der Nutzen einer Datenstrategie ist klar, Quelle: BARC Survey Datenmanagement im Wandel, Infografik BARC

19 , doch Unternehmen handeln noch nicht entsprechend Quelle: BARC Survey Datenmanagement im Wandel, Infografik BARC

20 Realität DWH-Architektur Komplexität: Wartung, Weiterentwicklung, Nutzung Fehlende Flexibilität: neue Daten, neue Analysen Unzureichende Abfragezeiten Unbefriedigende Aktualisierung: Zyklen und Zeiten Kostenintensiv Unzureichende funktionale Abdeckung BARC

21 Die Bereitschaft zur Änderung ist da und das Datenmanagement im Wandel BARC

22 Wie wichtig sind die folgenden Themen für Ihr Unternehmen? Datenintegration 42% 54% 4% Data Warehouse 25% 63% 11% Datenarchitektur 22% 68% 10% Big Data 6% 28% 66% Kritisch Wichtig Nicht so wichtig Quelle: BARC Survey Datenmanagement im Wandel, n= BARC

23 Insbesondere die Positionierung zu Big Data wird zunehmend wichtiger für das Datenmanagement Datenintegration 58% 42% 1% Data Warehouse 46% 50% 4% Spürbare wachsende Bedeutung Datenarchitektur 51% 48% 1% Big Data 53% 42% 6% Wird wichtiger Gleichbleibend Wird unwichtiger Quelle: BARC Survey Datenmanagement im Wandel, n= BARC

24 Was verändert sich in Ihrem Unternehmen konkret bzw. wird sich konkret verändern bei den Themen Datenintegration und Data Warehousing? Zunehmende Integration von heterogenen Datenquellen Befähigung der Fachbereich, Daten selbstständig zu integrieren/auszuwerten Nutzung neuer Technologien, um die Komplexität der Datenarchitektur zu reduzieren Versuch der Reduzierung des ressourcenbindenden Aufwands Nutzung einer Datenstrategie/Governance Einsatz neuer Technologien/Methoden zur Abbildung eines Unternehmensgedächtnis Zunehmender Einsatz von Hilfsmittel zur einfachen Integration neuer Datenquellen Zunehmender Einsatz von Big-Data-Technologien 22% 27% 31% 43% 49% 53% 53% 69% Rückgriff auf neue Datenarchitekturansätze Auslagerung/paralleler Betrieb von einzelnen Funktionsbereichen des Data Warehouse Rückgriff auf ein cloud-basiertes Date Warehouse Sonstige 6% 5% 14% 12% Quelle: BARC Survey Datenmanagement im Wandel, n= BARC

25 Neue Architekturen sind erforderlich BARC

26 Agenda Impact of digitalization on the analytical infrastructure Trends and developments for BI and Big Data Modern architecture concepts and technologies BARC

27 Mind shift through Big Data How Big Data approaches change the behavior to analyze information and to transform insights into action? BARC

28 Mind Shift Big Data: Analysis Process, Data Persistency and Data Volumne regulated analysis Data Warehousing process explorative analysis Big Data process Visualization Known questions, protected results Visualization Unknown question, unprotected results ( fast fail ) Analytics Analytics Data preparation Data interpretation Data storage Verify first, then collect Data preparation Data interpretation Data storage Collect first, then verify Data provisioning Data provisioning

29 Mind shift by Big Data requires an expansion of the existing infrastructure/architecture to meet the requirements to satisfy BARC

30 IQ Services DP Services Classic Data Warehouse Concept of the 1990s The typical environment for the management of enterprise data Core Processes: ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Tactical BI Data Mart Data Mart Data Mart Data Mart Data Distribution Conformed Facts Conformed Dimensions Data Distribution Data Warehouse Data Integration Staging Area Extraction ERP SCM CRM Business transaction data (structured) External systems BARC

31 How companies transform their data and analysis environments, so that information can be provided better, faster and more cost-effective? BARC

32 IQ Services DP Services New Data Sources, Technologies, and Features Core Processes: ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Explorative BI: Advanced Analytics, Extended Data Reporting & dashboards Analysis Search & discover Predictive modeling Text analysis Tactical BI Explorative BI Data Mart SSBI/DI Data Mart Data Mart Data Mart Expl. SSBI/DI Data Distribution Analytical DB Conformed Conformed Facts Dimensions Data Distribution Analytical DB Data Warehouse Data Integration Hadoop Staging Area Extraction Search Index Catalog NoSQL DB Hadoop ERP SCM CRM Business transaction data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) BARC

33 IQ Services DP Services And Operational BI and Big Data Integration Core Processes: ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Explorative BI: Advanced Analytics, Extended Data Reporting & dashboards Analysis Search & discover Predictive modeling Text analysis Operational BI Tactical BI Explorative BI CEP Data Mart SSBI/DI Data Mart Data Mart Data Distribution Data Mart Expl. SSBI/DI Analytical DB Analytical DB Conformed Conformed Facts Dimensions Data Distribution Analytical DB Search Index Catalog Streaming NoSQL DB Data Warehouse Data Integration Hadoop Staging Area Extraction NoSQL DB Hadoop Analytical DB, Hadoop, Integrated Data Virtualization, logical business Data Integration model Services, Cloud ERP SCM CRM Business transaction data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) BARC

34 Data Lake Concept and new terms are added Reporting & dashboards CEP Streaming Engine ERP SCM CRM Business transaction data (structured) Analysis Search & discover Predictive modeling Text analysis SSBI SSDI External systems Data Lake Analytical Database File System Sensor data Data Lake Concept Concept was introduced by Hadoop Distributors (eg Explor Pivotal (EMC), -ation Horton Works) A mainly on Hadoop based concept with the aim to collect or to store Search data of NoSQL any type in the Index, Hadoop file DB system Catalog Evaluation and transmission of the data to other infrastructure elements (repositories), if necessary Web log data Machine-generated data (struct.) File System Strict data governance necessary - otherwise the Data Lake easily will be a data swamp Social media Documents Human-generated (polystr.) BARC

35 Big Data Lakes or Swamps? As we collect data Can we preserve clarity? Do we know what we are collecting? Can we find the data we need? Are we creating a data swamp? How do we build trust in big data? Do we know what data is being used for? Solution: Data Integration and Governance provides you with an Information Fabric to turn a swamp into valuable supervised Data Lake BARC

36 What is a supervised Data Lake? A supervised data Lake is an information environment that provides data to an organization for a variety of analytics processing It is possible to deploy analytics into the supervised Data Lake to generate additional insight from the data loaded into the supervised Data Lake A supervised Data Lake manages shared repositories of information for analytical purposes Data values may be replicated in multiple repositories in the supervised Data Lake. However the supervised Data Lake ensures the copying and updating of this data is managed and governed using well-defined information supply chains Supervised Data Lake = Efficient Management, Governance, Protection and Access BARC

37 Four main disciplines around the Data Lake 1 Act on business transactions move changed or new data to the Data Lake act on events immediately 2 Explore raw data in the Data Lake advanced analytics on top of Hadoop 3 Analyze refined data in-memory databases for fast access on subject oriented data 4 Refine and Govern the Data Lake enable flexible forms of data movement and data transformation guarantee data quality government of all processes in and around the Data Lake BARC

38 The support of operational processes through real-time analytics requires analysis close to the operational systems Mapping the disciplines of the Data Lake Architecture - Acting (1) Analysis (2), Exploration (3) Refine and Govern (4) Operational BI: Core Processes ERP, CRM, SCM, OLTP Reporting & dashboards CEP Streaming Engine 1b 1a 2 RDBMS Data Virtualization, Data Integration Services, Cloud Refined Data (DWH, Data Marts, etc.) ERP SCM CRM Business transaction data (structured) Classic BI: Reporting, Dashboards, OLAP Analysis Search & discover Predictive modeling Text analysis SSBI SSDI External systems File System 2 Analytical Database Sensor data Web log data Machine-generated data (struct.) Acting on (Business) Transactions Explorative BI: Advanced Analytics, Extended Data 1a Transactional Systems Data Movement from Explor operationale -ation Systems (e.g. RDBMS) via (real-time) Data Integration 1b Streaming Analytics NoSQL Stream- DB and CEP Processing File System Social media Search Index, Catalog Analysis of refined data Documents Human-generated (polystr.) BARC

39 Data Governance Complement the architecture / infrastructure through an exploratory environment and governance Mapping the disciplines of the Data Lake Architecture - Acting (1) Analysis (2), Exploration (3) Refine and Govern (4) Operational BI: Core Processes ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Explorative BI: Advanced Analytics, Extended Data Reporting & dashboards Refine and Govern the CEP Data Lake Streaming 4a Refine Engine Data Integration, MeDM 4b Govern Catalog, MeDM, Governance Processes, etc. 4a ERP SCM CRM Business transaction data (structured) Analysis Search & discover Predictive modeling Text analysis SSBI Exploration of raw data 3a Hadoop Data Storage 3b Analysis External systems File System Analytical Database Iterative / exploratory SSDI analysis of data Sensor data Explor -ation Web log data Machine-generated data (struct.) NoSQL DB Data Virtualization, Data Integration Services, Cloud 3b File System Social media 3a Search Index, Catalog Documents Human-generated (polystr.) 4b BARC

40 The Data Lake is technology, vendor and deployment agnostic Typically a mix of technology and vendors, cloud and on premise Technologies It is key to use the best fitting technology for a given task Each technology has its strength in different areas, e.g. Hadoop stores any data in any format, offers flexible analytics, at low price per TB In-memory databases offer best query response times and high compression rates Vendors Glue between technologies is the use of open standards (e.g. SQL) Vendors that support open standards can be integrated easily Cloud / IaaS / PaaS / on-premise The decision for deployment of technologies is not primarily driven by technology but by ease of use, flexibility/agility, privacy concern, compliance, cost structure, BARC

41 Data storage options for the analytical infrastructure BARC

42 The two Worlds of Big Data Data in Motion Data at Rest Steaming/CEP Proprietary Operations SW Process Manufacturing Fraud and Risk Calculations High Frequency Trading Etc. Architecture Data Warehouses Hadoop, NoSQL Data Stores, Relational Databases OLTP Systemes CRM, ERP,... Real-time Business Intelligence BARC

43 Different Data Different Problems Data at Rest Main Problem: Storage Cost Curve Secondary Problem: Ability to Act on large Data Sets Data in Motion Real-Time Data Main Problem: Distributed Data Secondary Problem: Ability to Act on large Data Sets Streaming Data Main Problem: Unconnected Data Secondary Problem: Large number of stand-alone action points BARC

44 Data at Rest Storage Decision Drivers where to store which data? Metric Cost Security Performance Description Cost of Data Operations/Skills Sensitivity of Data Regulatory Requirements Raw Query Performance Concurrency BARC

45 Example (I) Data: System and Web etc. Log Files Performance-Metric: Need to ingest massive quantities quickly and continuously No single record queries required, mostly aggregations Security-Metric: No PII (Personally Identifiable Information) like Information All Users Cost-Metric: Low cost required Loosing data is ok (tolerable) Hadoop BARC

46 Example (II) Data: Manufacturing Sensor Data Performance-Metric: Need to ingest massive quantities quickly and continuously Need to see operational status across sensors Security-Metric: No PII like Information All Users Cost-Metric: Low cost required Support low cost long retention periods NoSQL Hadoop BARC

47 Example (III) Data: Web Store Orders / Purchases Performance-Metric: Relatively low volume when compared to activity logs Need to see and act on individual transactions Security-Metric: Contains sensitive data (or links to it) Requires restricted access to some of the data Cost-Metric: Business critical Consistency is most important RDBMS/ NoSQL RDBMS BARC

48 Example (IV) Data: Web Store Recommendations (publish) Performance-Metric: High volume of low latency data, but simple requests Need to see individual recommendations Security-Metric: No sensitive data Application level access (automated) Cost-Metric: Business critical low latency requirement Distributed environment NoSQL BARC

49 Changes in the architecture through in-memory BARC

50 Complement the architecture / infrastructure with in-memory components for operational reporting and analysis Operational BI: Core Processes ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Explorative BI: Advanced Analytics, Extended Data Reporting & dashboards Analysis Search & discover Predictive modeling Text analysis SSBI Explor -ation In-Memory Database RDBMS (operational) SSDI Analytical Database (In-Memory) File System Analytical Database NoSQL DB File System Search Index, Catalog Data Virtualization, Data Integration Services, Cloud ERP SCM CRM Business transaction data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) BARC

51 In-memory databases (IMDB) - characteristics Utilization of Hardware Trend - processors are every Months twice as fast and memory becomes cheaper in the same proportion Data are permanently in main memory Memory is the primary "persistence layer Still: Logging to disk / recovery from the hard drive Processor cache conscious algorithms / data structures Use of multi-core processors and processor features Column-based data storage with a high compression ratio (memory is the limiting factor) Memory-optimized algorithms and processing BARC

52 Areas of application of In-Memory in the analytical infrastructure for Big Data Operational BI: Core Processes ERP, CRM, SCM, OLTP Classic BI: Reporting, Dashboards, OLAP Explorative BI: Advanced Analytics, Extended Data Reporting & dashboards SSBI Core Data Warehouse Faster response times Reduced data latency No/less Performance Objects No/less aggregates/summaries In-Memory Database RDBMS (operational) Reporting directly on the operative data No ODS required Faster response times (near)real-time Low data latency Data Virtualization, Data Integration Services, Cloud ERP SCM CRM Business transaction data (structured) Analysis Search & discover Predictive modeling Text analysis SSDI File System External Faster Sensor data systems response times Web log data Machine-generated data (struct.) File System Explor -ation Data Marts Faster response times No/less Performance NoSQL Objects DB Only virtual data marts Analytical Database Analytical (In-Memory) Database Operational Data Store (Near Real Time) Reduced data latency No Performance Objects Social media Search Index, Catalog Spark - In-Memory for ML/Iiterative data exploration Faster response times Reduced data latency Documents Human-generated (polystr.) BARC

53 Historically Grown DWH Architecture The layered Data Warehouse since the early 90 s Classical BI System of Record Reporting, Dashboards, Analysis Conformed dimensions, lookup tables, Hierarchies, fact tables System of record & history Business Intelligence layer Data Mart layer Data integration layer Data Warehouse layer Data integration layer Complex business Calculations (ETL) Complex business rules & dependencies (ETL) Staging area ERP SCM CRM Business transaction data (structured) External systems BARC

54 Advanced DWH Architecture Self Service BI, reduced DI efforts, improved agility, better Performance Reporting, Dashboards, Analysis Business Intelligence layer Virtual Data Marts, conformed dimensions, facts Data Mart layer Views Complex business calculations System of record & history Data Warehouse layer Data integration layer Faster, less complex (batch, mini-batch, continuous ingest) Staging area ERP SCM CRM Business transaction data (structured) External systems Bring business rules closer to the business Improved IT/BICC reaction time Reduced costs and impact on the DWH Higher flexibility in provisioning data to the business BARC

55 Extended analytical infrastructure/architecture for Big Data new and expanded applications are possible Customer experience Business optimization & innovation Financial performance & risk management Operations & fraud protection IT cost effectiveness Reporting & dashboards Operational BI Analysis Search & discover Predictive modeling Text analysis SSBI Tactical BI Explor -ation Explorative BI In-Memory Database RDBMS (operational) SSDI Analytical Database (In-Memory) File System Analytical Database NoSQL DB File System Search Index, Catalog Data Virtualization, Data Integration Services, Cloud ERP SCM CRM Business transaction data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) BARC

56 Meta Data Data Governance The role of the SPoT in the age of Big Data "to spot or not to spot" Transformation from a single SPoT to a logical SPoT or logical Data Warehouse Customer experience Business optimization & innovation Financial performance & risk management Operations & fraud protection IT cost effectiveness Reporting & dashboards Operational BI Analysis Search & discover Predictive modeling Text analysis SSBI Tactical BI Explor -ation Explorative BI In-Memory Database RDBMS (operational) Analytical Database (In-Memory) Analytical Database Logical Single SPoT SPoT logical SSDIData Warehouse File System NoSQL DB File System Search Index, Catalog Data Integrated Virtualization, logical Data business Integration data Services, model Cloud ERP SCM CRM Business transaction data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) BARC

57 Exemplary applications show the use of new capabilities in the extended data management environment BARC

58 Comprehensive traffic safety and control system with improved response to accidents Task Cost-effective solution for the improvement of traffic safety and control system Accuracy in the detection of events with consideration of traffic conditions (speed limits) and prediction of arrival of buses Analysis of GPS data whose flow rate is high and their capture difficult Analysis of accident black spots Goals Daily monitoring of 600 buses on 150 routes Analysis of 50 updates per second for the Bus Locations Acquisition, processing and visualization of location data of public transport vehicles BARC

59 Infrastructure / Architecture for Traffic Safety and Control System Analyzes in real-time to improve the customer experience/satisfaction Operational BI Customer experience Business optimization & innovation Financial performance & risk management Operations & fraud protection IT cost effectiveness Reporting & dashboards Analysis Search & discover Predictive modeling Text analysis Operational BI SSBI Tactical BI Explor -ation Explorative BI CEP CEP Streaming Engine SSDI Analytical Database (In-Memory) Analytical Database NoSQL DB Search Index, Catalog File System File System Data Virtualization, Data Integration Services, Cloud ERPSensor data SCMWeb log datacrm Machine-generated Business transaction data (struct.) data (structured) External systems Sensor data Web log data Machine-generated data (struct.) Social media Documents Human-generated (polystr.) BARC

60 More effective marketing campaigns with Big Data Task Analysis of customer s by service provider - customer of the service provider will receive information on best days and times for their marketing s in order to maximize the return responses. Goals Significant improvement in the analysis depth and scope of the analysis Improve performance of customer campaigns (better and more focused approach, thus more successful responses) Reduction of the analysis time BARC

61 Infrastructure / architecture for marketing Analysis to maximize the response rates Classic BI Explorative BI Customer experience Business optimization & innovation Financial performance & risk management Operations & fraud protection IT cost effectiveness Reporting & dashboards Analysis Search & discover Predictive modeling Text analysis Operational BI SSBI Tactical BI Explor -ation Explorative BI CEP Streaming Engine SSDI Analytical Analytical Database Database (In-Memory) (In-Memory) Analytical Database NoSQL DB Search Index, Catalog File System File System File System Data Virtualization, Data Integration Services, Cloud External ERP SCM CRM Sensor data Web log data Social media Documents systems Business transaction data (structured) Machine-generated data (struct.) Human-generated (polystr.) BARC

62 Reducing energy costs and increasing reliability and performance of the power system Task Assessing the feasibility of a smart grid method with the name "Transactive Control" Goals Involvement of consumers and reactive power assets across the whole power grid for optimizing the system and integration of renewable energies Ability to analyze and extracting information from large volumes of data Higher grid efficiency and reliability by self-monitoring of the system and Feedback capabilities City can prevent potential power failure BARC

63 Infrastruktur/Architektur für Smart Grid Analysis to reduce costs and optimize grid Operational BI Classic BI Customer experience Business optimization & innovation Financial performance & risk management Operations & fraud protection IT cost effectiveness Reporting & dashboards Predictive modeling Reporting & Analysis dashboards Search & discover AnalysisPredictive modeling Text analysis Operational BI SSBI Tactical BI Explor -ation Explorative BI CEP CEP Streaming Engine SSDI Analytical Analytical Database Database (In-Memory) (In-Memory) Analytical Database NoSQL DB Search Index, Catalog File System File System Data Virtualization, Data Integration Services, Cloud External External ERP Sensor data SCMWeb log data CRM ERP SCM Sensor datacrm Web log data Social media Documents systems systems Machine-generated Business transaction data (struct.) data (structured) Business transaction Machine-generated data (structured) data (struct.) Human-generated (polystr.) BARC

64 The interesting dilemma the context makes the difference A man goes into a jewelers and buys an expensive watch Is it fraud in which case the bank must stop it Is it money-laundering in which case the bank must report it Does he have an expensive trophy wife in which case perhaps he would be interested in a loan? Has he just won the lottery should the bank improve the services offered? Threat Obligation Opportunity The same event is of interest by different departments. There is major overlap in the data required to answer the question. It may not be possible to determine the answer with just the information in the channel - Previous or subsequent activity is required It is all a matter of coordination and timing

65 Conclusion New technologies and concepts extend the data space for analysis Analysis on poly-structured data and machine-generated data is possible - also in real-time New applications enable extended or new business models Expansion of the traditional DWH (SPoT) to a logical DWH (SPoT) with specialized engines Data integration and governance platforms provide the opportunity to ensure data integrity in the logical DWH BARC

66 Conclusion New and performant technologies (eg in-memory) allow more streamlined and simpler DWH architectures Easier data integration, less performance objects, less Layer Operational reporting or analysis directly on the OLTP system by high performance in-memory technologies is possible Questions towards data governance and data quality are still to be resolved BARC

67 Ihr Kontakt bei BARC Otto Görlich Senior Analyst BI & Datenmanagement Tel +49 (0) BARC GmbH Berliner Platz Würzburg Timm Grosser Senior Analyst BI & Datenmanagement Tel +49 (0) tgrosser@barc.de BARC GmbH Berliner Platz Würzburg BARC

68 BARC-Tagung: Data Governance Day September 2015 in Baden (Schweiz) Themen: Datenstrategie und Governance Data Life Cycle Data Driven Organisation Data Integration Data Quality Master Data Management Data Life Cycle Document Management Fachvorträge der BARC-Analysten und Fachverbände Seminardokumentation mit Fachartikeln Herstellerpräsentationen BARC

69 BARC-Tagung: Advanced und Predictive Analytics 29. September 2015 in Frankfurt Themenschwerpunkte: Anwendungsgebiete (z.b. Auslastungsoptimierung, Wartung, Forschung, CRM, Social Media) Praxisbeispiele und Erfolgsfaktoren Management des Analytischen Prozesses Marktübersicht Software- und Serviceanbieter Fach-, Praxis-und Produktvorträge zu den verschiedenen Anwendungsmöglichkeiten Herstellerpräsentationen BARC

70 BARC Congress Business Intelligence & Datenmanagement am November 2015 in Würzburg Highlights & News: Anbietervorträge, Case Studies, Analystenvorträgen von BARC und PAC, Best Practice Award uvm. Neuer Track mit Fokus auf Datenmanagement und Big Data Videoaufzeichnung aller Vorträge BARC-Service zur Terminvereinbarung im Vorfeld zwischen Teilnehmern und Ausstellern Attraktive Abendveranstaltung mit Verleihung des BI Best Practice Awards Erweiterung der Zielgruppe durch den parallel stattfindenden CRM Summit BARC