Dissertation. Zero-Latency Data Warehousing. Toward a Zero Latency Event Sensing and Responding Data Warehousing

Transkript

1 Dissertation Zero-Latency Data Warehousing Toward a Zero Latency Event Sensing and Responding Data Warehousing Ausgeführt zum Zwecke der Erlangung des akademischen Grades eines Doktors der technischen Wissenschaften unter der Leitung von o.univ.-prof. Dipl.-Ing. Dr.techn. A Min Tjoa Institut für Softwaretechnik & Interaktive Systeme Technischen Universität Wien und Univ.-Prof. Dr. Roland Wagner Institut Für Anwendungsorientierte Wissensverarbeitung Johannes Kepler Universität Linz eingereicht an der Technischen Universität Wien Fakultät für Technische Naturwissenschaften und Informatik von Nguyen Manh Tho Wien, im August 2005 Matrikelnummer: Viktor-Kaplan Strasse 6-8/2/418 A-1220 Wien

2 Acknowledgement Pursuing a PhD program is an enjoyable trying progress where dissertation is a significant checkpoint s deliverable. The work in this dissertation has been conducted within 3 years studying at the Institute of Software Technology and Interactive System, Vienna University of Technology. However, without help, support, and encouragement of several persons, I would never have been able to finish this work. First of all, I would like to express my sincere gratitude to my supervisor, Prof. A Min Tjoa for his inspiring and encouraging way to guide me to a deeper understanding of knowledge work, and his valuable guidance during the whole work with this dissertation. He showed me different ways to approach a research problem, broaden my view to investigate the topic, and the need to be persistent to accomplish any goal. More important, I deeply indebted for his spiritual supports in many aspects of life. This is the best motivation which helps me passing any difficulty to reach the target. I would like to thank Prof. Roland Wagner, my second supervisor for his guidance and comments to my work as well as for significant experience gaining from DEXA and DaWaK conferences. Special thanks are given to Dr. Josef Schiefer for valuable discussions and prompt assistance in technical aspects. I very much appreciate for his kindly attitude to offer discussion time in weekends and I am very happy about our fruitful collaboration in joint publications. I am also grateful to Prof. Peter Brezany, Institute for Software Science, University of Vienna, for his interesting and inspiring discussions on Grid computing. He also gave me valuable comments and revision on my publications. I would like to express my deep appreciation to Dr. Michael Aigner, Dr. Martin Windisch, and Dr. Jaromir Nemec at T-Mobile Austria GmbH. Their friendly collaborative discussions offer me a lot valuable experiences in real world s practical environment. Thanks to all my colleagues at the Institute of Software Technology and Interactive System for providing a good working collaboration atmosphere as well as further debates and discussion in many topics. Especially, my obligation is given to Prof. Gerti Kappel, Dr. Beate List for their collaborative assistance in project work, Dr. Andres Rauber for valuable advice in performing research, Shuab Karim and Khalid Latif for their English revision efforts and interesting discussion. Thank also for the friendship with Hoang Huu Hanh, Khabib Mustofa, Shah Khusro, Ahmed Mansoor and Amin Andjomshoaa. I also would like to send my appreciation to Mrs. Maria Schweikert, Mag. Markus Klemen and Ing. Michael Schadler for their support in providing me the perfect working environment. Last but not least, I am greatly indebted to my devoted wife Thuan An and my lovely daughter Annie Thien Nga for supporting me with love and spiritual understanding. Their love and support without any complaint or regret has enabled me to complete this dissertation. Financial support has been provided by the Technology Grant South East Asia (No.322/1/EZA/ 2002) of the Austrian Council Research and Technology and the ASEA-UNINET. Nguyen Manh Tho. i

3 Abstract Increased data volumes and accelerating update speeds are fundamentally changing the role of the data warehouse in modern businesses. More data, coming in faster, and requiring immediate conversion into decisions means that organizations are confronting the need for zero-latency Data Warehousing. The Zero-Latency Data Warehouse (ZLDWH) is an extended stage in the data warehouse development, which will enable a complete business intelligence process to observe, understand, predict, react to, reorganize, monitor, automate and control feedback loops in the minimal latency. With the aim to decrease the time it takes to make a business decision, ZLDWH must satisfy some new requirements of the speed driven business such as operational decision support, closed-loop integrated analytics, and automated reaction. This doctoral thesis reviews the start-of-the-art of the Zero-Latency Data Warehouse and investigates several approaches towards its implementations: - The event-feeded Comprehensive Slowly Changing Dimension (cscd). We propose an extended event-oriented SCD solution which is mixed from SCD type 2 and type 3 to validate the event-messages and reconstruct the complete history of the dimension. The solution could be applied to any dimension data, allow the user to execute the well-performing queries on both historical and current state of dimension. The prototype package is implemented and tested at T- Mobile Austria. - The Grid-based Zero-Latency Data Stream Warehouse (GZLDSWH). We present a framework of building a Grid-based Zero-Latency Data Stream Warehouse to tackle the resource limitation issues in data stream processing without using approximation approaches. The GZLDSWH is built upon a set of Open Grid Service Infrastructure (OGSI)-based services and Globus Toolkit 3 (GT3) with the capability of capturing and storing continuous data streams, performing analytical processing, and reacting autonomously in near real time to some kinds of events based on well-established Knowledge Base. - Sense & Response Service Architecture (SARESA). We introduce Sense & Respond loops that support a complete Business Intelligence process to sense, interpret, predict, automate and respond to business processes. Our approach enables real-time analytics across corporate processes, notifies with actionable recommendations or automatically triggers, effectively closing the gap between Business Intelligence systems and business processes. We also describe the prototype implementation applied in the Mobile Phone Fraud Detection application. Finally, we mention some future trends of non-standard Data Warehousing applications and discuss about the important role of zero-latency data warehouses techniques which could be applicable in these emerging Data Warehousing applications. ii

4 Zusammenfassung Das drastisch zunehmende Wachstum des Datenvolumens und die Anforderungen für beschleunigte Updates in Unternehmen haben die Aufgaben von Data Warehouses signifikant verändert. Immer mehr Daten, die in immer kürzeren Intervallen anfallen, erfordern das sogennante Zero Latency Data Warehousing (Fast Echtzeit Data Warehousing) um für den Entscheidungsträger und Institutionen zeitgerechte Entscheidungen treffen zu können Das Zero-Latency Data Warehouse (ZLDWH) stellt eine weitere Stufe der Entwicklung von Data Warehouses, welche eine ganzheitliche unternehmensweite Prozeßführung eines Unternehmens unterstützt. Eine solche Prozeßführung ist durch die intelligente Prozesse, welche die Aktivitäten der Beobachtung, des Auffassens, der Vorhersage, des Reagierens und der automatischen Rückkoppelung inkludiert, charakterisiert. Das Ziel der Erreichung einer minimalen Zeitverzögerung für Entscheidungsprozesse impliziert neue Anforderungen, die zu lösen Inhalt dieser Dissertation ist. Vor allem sind dies Aspekte der operationalen Entscheidungsunterstützung, des unmittelbaren ( closed loop ) Durchgriffs von Data Warehouses auf operationale Datenbanken und der automatisierten Reaktion auf besondere Zustände des Data Warehouse. Die Dissertation behandelt den Stand der Forschung auf dem Gebiet des ZLDWH und bietet Lösungsansätze und Implementierungen in folgenden Bereichen: - Das event-feeded Comprehensive Slowly Changing Dimension (cscd) wird hier vorgeschlagen als eine Erweiterung der SCD Typ 2 Lösung, welche Komponenten des SCD Typ 3 beinhaltet. Ziel ist es eine vollständige Rekonstruktion historischer Daten der Dimensionsdaten u erreichen. Eine prototypische Implementierung wurde für eine sehr große T- Mobile Applikation getestet. - Das Grid-based Zero latency Data Stream Warehouse (GZLDSWH) stellt ein neuartiges Konzept dar, welches zur Erreichung der obigen Ziele für sogenannte Data Streams die Grid- Computing-Technologie anwendet. Das GZLDSDWH ist aufgebaut auf Grid-Services des Open Grid Service Infrastructure (OGSI) und Globus Toolkit 3 (GT3). - Die Sense & Response Service Architecture (SARESA) wurde hier entwickelt um eine Rückkoppelung zwischen Analyse im Data Warehouse und den notwendigen operationalen Aktivitäten zu erreichen. Der vorgestellte Ansatz ermöglicht eine beinahe Echtszeitreaktion basierend auf Analysen der Data Warehouse Informationen durch automatische Trigger und Entscheidungsempfehlungen, welche die Zeitspanne zwischen Analyse und Prozeßaktivität erheblich verkürzt. Eine prototypische Implementierung für die Auffindung von Berügereien für T-Mobile wird hier vorgestellt. Schließlich werden mögliche Zukunftsentwicklungen für Nicht-Standard.Anwendungen skizziert um auch hier die Notwendigkeit von Fast-Echtzeit-Ansätzen zu dokumentieren. iii

5 Table of Contents Acknowledgement...i Abstract...ii Zusammenfassung...iii Table of Contents...iv List of Figures...vii List of Tables...ix Part I. FUNDAMENTALS...1 CHAPTER 1. INTRODUCTION OVERVIEW AND MOTIVATION TARGET CRITERIA OUTLINE...5 CHAPTER 2. ZERO-LATENCY DATA WAREHOUSING: CONCEPTS AND THE STATE-OF- THE-ART INTRODUCTION TRADITIONAL DATA WAREHOUSE TIME LATENCY IN DATA WAREHOUSING AND BUSINESS INTELLIGENCE INFORMATION EVOLUTION IN DATA WAREHOUSING: FROM REPORT TO ZERO-LATENCY DATA WAREHOUSE...15 Stage 1: Reporting...16 Stage 2: Analysis...17 Stage 3: Prediction...18 Stage 4: Operational Data Warehouse...18 Stage 5: Active Data Warehouse...20 Stage 6: Minimized Latency ZERO-LATENCY DATA WAREHOUSE SUMMARY...24 CHAPTER 3: BACKGROUND AND RELATED WORK ZERO-LATENCY AND REAL-TIME Zero-Latency Enterprise Active Data Warehousing Real-time Data Warehousing Real-time Analytics Real-time Decision Support Business Activity Monitoring (BAM) CONTINUOUS DATA STREAMS SERVICE ORIENTED ARCHITECTURE GRID COMPUTING What is the Grid? The Grid Infrastructure The Grid Architecture Evolution of Grid Computing Classification of Grids SUMMARY...52 Part II. IMPLEMENTING ZERO-LATENCY DATA WAREHOUSING ENVIRONMENT...53 CHAPTER 4. EVENT-FEEDED SLOWLY CHANGING DIMENSION (CASE STUDY AT T- MOBILE- AUSTRIA) INTRODUCTION...54 iv

6 4.2. STATE-ORIENTED DATA VERSUS STATE-CHANGE OR EVENT-ORIENTED DATA SLOWLY CHANGING DIMENSION SCD type SCD type SCD Type Limitations of the existing SCD Approaches Comprehensive enhanced SCD Solution DATA WAREHOUSING AT T-MOBILE Snapshot based SCD approach Event based SCD approach EVENT MODEL Profile Example Event Processing EVENT-FEEDED CSCD IMPLEMENTATION Development environment The UTL_EVENT_SCD Package Event Processing (EP) On demand Snapshot Generation (SG) Consistency checking and recovery (CC) SUMMARY...74 CHAPTER 5. THE GRID-BASED ZERO LATENCY DATA STREAM WAREHOUSE (GZLDSWH) INTRODUCTION OPEN GRID SERVICE ARCHITECTURE (OGSA), OPEN GRID SERVICE INFRASTRUCTURE (OGSI) AND GLOBUS TOOLKIT 3 (GT3) Web Services OGSA versus OGSI - what is a Grid Service Globus Toolkit 3 (GT3) THE GRID-BASED ZERO-LATENCY DATA STREAM WAREHOUSE (GZLDSWH) OVER VIEW THE FRAMEWORK FOR BUILDING A GZLDSWH Building the whole system from scratch Following the OGSA, OGSI implemented by Globus Toolkit WS-Resource Framework The Grid-based Conceptual Architecture of GZLDSWH THE OPERATION OF GZLDSWH DYNAMIC GRID SERVICE COLLABORATION Dynamic Service invocation requirement Extended Dynamic Service Control Language Dynamic Workflow Management Service THE OLAP CUBE MANAGEMENT SERVICE KNOWLEDGE BASED RULE DESIGN AND EVALUATION MECHANISM RELIABILITY AND EFFICIENCY IMPLEMENTATION Dynamic Service Control Engine Prototype Sequential OLAP Cube Engine SUMMARY AND FUTURE WORK CHAPTER 6. SENSE & RESPONSE SERVICE ARCHITECTURE (SARESA): AN APPROACH TOWARDS A REAL-TIME BUSINESS INTELLIGENCE INTRODUCTION REAL-TIME BUSINESS INTELLIGENCE Real-time BI Requirement Real-time BI Architecture Enhanced Real-time BI Architecture SENSE & RESPONSE SERVICE ARCHITECTURE (SARESA) Sense & Response Loops SARESA Architecture Event Processing Model (EPM) v

7 Analysis Service and Analytical Process Architecture SARESA Advantages REAL-TIME BI IMPLEMENTATION Mobile Fraud Detection Scenario SARESA System Deployment Event Processing Model (EPM) Event Types Definition of Event Types EventObject Types Event Processing Model & Application Configuration file Application Components Section Event Processing Map Section Filters Data Sources Correlation Synchronsiation The Simulator Simulation Scenarios Event Receiver Event Sequences Event Patterns Value Generators Real-time Analysis Service SUMMARY Part III. NON-STANDARD DATA WAREHOUSING FUTURE TRENDS AND RESEARCH CHALLENGES CHAPTER 7. FUTURE TRENDS IN DWH AND RESEARCH CHALLENGES INTRODUCTION NON-STANDARD DATA WAREHOUSING APPLICATION Spatial Data Warehousing Motivation Spatial/GIS Data Warehouse Research Challenges Web-enabled Data Warehousing Motivation Clickstream and Data Webhouse Challenges Process Data Warehousing Motivation Process Data Warehouse (PWH) Research Challenges Data Warehousing and the Grid Motivation Grid Technology and Data Warehousing Research Issues ZERO-LATENCY WAREHOUSING TECHNIQUES AND THE NON-STANDARD DATA WAREHOUSING APPLICATIONS SUMMARY CHAPTER 8. CONCLUSION AND FUTURE WORK CONCLUSION CONTRIBUTION Event-feeded Slowly Changing Dimension Grid-based Zero-Latency Data Stream Warehouse Sense & Response Architecture FUTURE WORK Abbreviations Bibliography Curriculum Vitae vi

8 List of Figures Figure 2.1. Multidimensional (MD) Data Model and Dimension Hierarchy Figure 2.2. Classical multi-tier Data Warehouse Architecture Figure 2.3. Business value and action time Figure 2.4. Data Warehouse Evolution, Stages Figure 2.5. Data Warehouse Evolution, Stages Figure 3.1. The Zero-Latency Enterprise framework Figure 3.2. Active Data Warehouse Architecture Figure 3.3. Data Warehouse Updates are Accelerating Figure 3.4. Continuous Data Streams Figure 3.5. The long history of programming languages, distribution technology, and business computing toward a SOA paradigm Figure 3.6. The SOA Interactive pattern Figure 3.7. The Grid Infrastructure Figure 3.8. The Grid Architecture Figure 3.9. The Grid Generations Figure 4.1. Snapshot-based comparision SCD approach Figure 4.2. Global view of Event-based SCD process Figure 4.3. Simplified Profile Event Definition Figure 4.4. Customer Event Profile Example Figure 4.5. Informatica mapping diagram of event processing Figure 4.6. UTL_EVENT_SCD Package modules and its related tables Figure 4.7. Elapsed processing time and performance throughput comparison between event- SCD and snapshot based SCD approach Figure 4.8. Create Snapshot from scratch (i_timepoint is the time point of the snapshot data). 73 Figure 4.9. Create Snapshot from based snapshot (BASED_CUST_SNAP is the based snapshot table, v_prev_time is the time point of the based snapshot data) Figure 5.1. Key concepts of the Grid service world Figure 5.2. Simple Web Service architecture Figure 5.3. Globus Toolkit 3 Architecture Figure 5.4. The overall process of the Grid-enabled Zero-Latency Data Warehouse System (GZLDSWH) Figure 5.5. Mobile phone fraud detection Figure 5.6. The Service Components of GZLDSWH Figure 5.7. The operations of GZLDSWH services Figure 5.8. The predefine logical Workflow of Service Invocation Figure 5.9. Composite Workflow Example Figure Conceptual Architecture of DSCE vii

9 Figure The Extended Service Control Engine and corresponding DSCL documents Figure OLAP engine architecture Figure Mobile Phone Fraud Detection Rule Figure DCSE Class Architecture Figure DCSE State Diagram Figure Comparison of overall workflow execution time Figure Performance results of queries for three different scenarios Figure 6.1. Traditional Business Intelligence Architecture Figure 6.2. Enhanced Architecture with real-time Business Intelligence Figure 6.3. Sense and Response Loops Figure 6.4. SARESA Architecture Figure 6.5. Event Processing Model Figure 6.6. Analysis Service and Analytical Process Architecture Figure 6.7. A fraudulent phone call from Austria to China Figure 6.8. SERESA System Deployment Figure 6.9. Fraud Detection Service running at the Worker and Admin Node Figure Fraud Detection Event Processing Model (EPM) Figure Simulation Studio Figure Normal and Fraud Phone Call Events generated by Simulation Studio Figure Real-time HOLAP Cube Structure in MS Analysis Server Figure FraudDetectionDW cube structure Figure FraudDectectionDW Cube Browser Figure TimerFrauDetection.cs Figure The processing when timer is expired Figure Determine the Timer thread hold in Analysis Service Figure MDX query and on FraudDetectionDW cube viii

10 List of Tables Table 2.1. Information Evolution Stages in Data Warehousing Table 3.1. The Many Meanings of Real Time Table 3.2. Real-time Analytics by Sample Industry or Audience Table 3.3. BAM tools checklist Table 3.4. Differences between static data set and continuous Data stream Table 4.1. Event Representation vs. State Representations of Data Table 4.2. Example using SCD Type Table 4.3. Example using SCD Type Table 4.4. Example using SCD Type Table 4.5. Example using enhanced SCD (mix Type 2 and Type 3) Table 4.6. EFD SCD Package Corrections and Exceptions Table 4.7. TRANS table refresh after UTL_EVENT_SCD package execution Table 4.8. SNAPSHOT tables generated at 7:00 and 7: Table 5.1. WSDL 1.1 elements Table 5.2. Characteristics of the test files used in the test scenarios Table 6.1. Stages of Sense & Respond Loops Table 7.1. Zero-Latency Data Warehousing techniques applied in non-standard Data Warehousing applications ix