WELCOME. Twitter Storm: Ereignisverarbeitung in Echtzeit. Guido Schmutz. Java Forum Stuttgart

Transkript

1 WELCOME Twitter Storm: Ereignisverarbeitung in Echtzeit Guido Schmutz Java Forum Stuttgart 2013 BASEL BERN LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN 1

2 Unser Unternehmen Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution-Engineering und der Erbringung von IT-Services mit Fokussierung auf und Technologien im D-A-CH-Raum. Unsere Leistungen erbringen wir aus den strategischen Geschäftsfeldern: B E T R I E B Trivadis Services übernimmt den korrespondierenden Betrieb Ihrer IT Systeme. 2

3 Mit über 600 IT- und Fachexperten bei Ihnen vor Ort Hamburg 11 Trivadis Niederlassungen mit über 600 Mitarbeitenden 200 Service Level Agreements Düsseldorf Frankfurt Stuttgart Mehr als 4'000 Trainingsteilnehmer Forschungs- und Entwicklungsbudget: CHF 5.0 / EUR 4 Mio. Finanziell unabhängig und nachhaltig profitabel Freiburg Basel München Wien Erfahrung aus mehr als 1'900 Projekten pro Jahr bei über 800 Kunden Bern Zürich Lausanne Stand 12/2012

4 Guido Schmutz Working for Trivadis for more than 16 years Oracle ACE Director for Fusion Middleware and SOA Co-Author of different books Consultant, Trainer Software Architect for Java, Oracle, SOA, EDA, BigData und FastData Member of Trivadis Architecture Board Technology Trivadis More than 20 years of software development experience Contact: guido.schmutz@trivadis.com Blog: Twitter: gschmutz 4

5 Agenda 1. What is Twitter Storm 2. Main Concepts 3. Trident 4. BigData and FastData combined Lambda Architecture 5. Summary 5

6 What is Twitter Storm? A platform for doing analysis on streams of data as they come in, so you can react to data as it happens. A highly distributed real-time computation system Provides general primitives to do realtime computation Can be used with any programming language It s scalable and fault-tolerant Originated at Backtype, acquired by Twitter in 2011 Open Sourced late

7 Hadoop vs. Storm Hadoop Batch processing Jobs runs to completion Stateful nodes Storm Real-time processing Topologies run forever Stateless nodes Scalable Guarantees no data loss Open Source Scalable Guarantees no data loss Open Source => big batch processing => Fast, reactive, real time procesing 7

8 Idea: Simplify dealing with queue & workers Scaling is painful queue partitioning & worker deploy Operational overhead worker failures & queue backups No guarantees on data processing 8

9 What does Storm provide? At least once message processing Horizontal scalability No intermediate queues Less operational overhead Just works 9

10 Agenda 1. What is Twitter Storm 2. Main Concepts 3. Trident 4. BigData and FastData combined Lambda Architecture 5. Summary 10

11 Main Concepts - Example Show the tweets related to Java Forum and show the count of the hashtags used 11

12 Main Concepts - Spout First thing we need is get all the tweets related to the conference A spout is the component which acts as the source of messages to be processed Twitter Tweets Tweet Tuples 12

13 Main Concepts - Spout Twitter Tweets Tweet Tuples 13

14 Main Concepts - Bolt For every tweet, get the hashtags Bolt is doing the processing of the message Twitter Tweets Tweet Tuples Splitter hashtag 14

15 Main Concepts - Bolt Twitter Tweets Tweet Tuples Splitter hashtag 15

16 Main Concepts - Bolt Twitter Tweets Tweet Tuples Splitter hashtag Counter 16

17 Main Concepts - Topology Twitter Tweets Tweet Shuffle Splitter Splitter Fields Counter Counter 17

18 Groupings Each Spout or Bolt are running N instances in parallel (called tasks) Groupings are used to decide which task in the subscribing bolt, the tuple is sent to Shuffle grouping Fields grouping All grouping Global grouping None grouping Direct grouping is random grouping is grouped by value, such that equal value results in equal task replicates to all tasks makes all tuples go to one task makes bolt run in the same thread as bold/spout it subscribes to producer (task that emits) controls which consumer will receive 18

19 Sample How does it work? Twitter Mein Präsentation Twitter Storm: Ereignisverarbeitung in Echtzeit heute um 15:35 am Java Forum Stuttgart #JFS2013 #storm #fastdata Tweet Splitter Splitter Counter Counter 19

20 Sample How does it work? Twitter Mein Präsentation Twitter Storm: Ereignisverarbeitung in Echtzeit heute um 15:35 am Java Forum Stuttgart #JFS2013 #storm #fastdata Tweet Splitter Splitter JSF2013 storm fastdata Counter Counter 20

21 Sample How does it work? Twitter Mein Präsentation Twitter Storm: Ereignisverarbeitung in Echtzeit heute um 15:35 am Java Forum Stuttgart #JFS2013 #storm #fastdata Tweet Splitter Splitter JFS2013 storm fastdata Counter Counter INCR JFS2013 INCR storm JFS2013= 1 INCR fastdata storm = 1 fastdata =1 21

22 Agenda 1. What is Twitter Storm 2. Main Concepts 3. Trident 4. BigData and FastData combined Lambda Architecture 5. Summary 22

23 What is Trident? High-Level abstraction on top of storm Makes it easier to build topologies Similar to Hadoop and Pig Supports Joins Aggregations Grouping Functions Filters 23

24 Agenda 1. What is Twitter Storm 2. Main Concepts 3. Trident 4. BigData and FastData combined Lambda Architecture 5. Summary 24

25 Lambda Architecture Batch Layer Serving Layer Immutable data B C Batch View D Incoming Data A G Query Speed Layer Data E F Realtime View 25

26 Merge Lambda Architecture Batch Layer Serving Layer Incoming Data All data Batch recompute Speed Layer Precompute Views Precomputed information batch view batch view query Process stream Realtime increment Incremented information real time view real time view Source: Marz, N. & Warren, J. (2013) Big Data. Manning. 26

27 Agenda 1. What is Twitter Storm 2. Main Concepts 3. Trident 4. BigData and FastData combined Lambda Architecture 5. Summary 27

28 Summary Express real-time processing naturally Not a Hadoop/MapReduce competitor Supports other languages Hard to debug Object Serialization Didn t cover Reliablility through guaranteed message processing Distributed RPC Storm cluster setup and deploy 28

29 Thank You! Trivadis AG Guido Schmutz BASEL BERN LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN 29