Академический Документы
Профессиональный Документы
Культура Документы
Apache Kafka
Agenda
2
$ whoami
3
What is Kafka?
Kafka is a scalable,
fault-tolerant, distributed queue
Producers and Consumers
Uses
Asynchronous communication in
event-driven architectures
Message broadcast for database
replication
Inside Kafka
TOPIC
TOPIC
TOPIC
Brokers
Heart of Kafka
Stores data
Data stored into topics
Zookeeper C C C
ZOOKEEPER ZOOKEEPER
5
Inside a topic
9
Discrete data processing models
Request / Response
APP APP APP
processing mode
Processing time <1
second
Clients can use this
data
10
Discrete data processing models
Request / Response
APP APP APP
processing mode
Processing time <1
second
Clients can use this
data
Batch processing
mode
Processing time few
hours to a day
DWH HADOOP Analysts can use this
data
11
Discrete data processing models
12
Promise of stream processing
APP APP APP APP
SEARCH
MONIT
CACHE
13
Promise of stream processing
APP APP APP APP
14
Stream processing using Kafka
15
Stream processing frameworks
16
Kafka Streams
17
Writing Kafka Streams
18
Demo
Simple Kafka Stream
19
Kafka Streams architecture and operations
Kafka manages
Parallelism
Fault tolerance
Ordering
State Management
Inputs:
Incoming stream of benchmark stream
quality from CDN provider
Incoming stream quality reported by
Hotstar clients
Output:
Calculate the locations reporting bad
QoS in real-time
Inputs:
Incoming stream of benchmark stream
quality from CDN provider Client
Incoming stream quality reported by reports
Hotstar clients
Output:
Calculate the locations reporting bad
QoS in real-time
Alerts
24
Kafka @ Hotstar
25
26