Kafka & Spark Structured Streaming
Build real-time streaming data pipelines by integrating Apache Kafka with Spark Structured Streaming.
Data Platform
advanced
8h
Overview
A one-day advanced course on building real-time pipelines with Apache Kafka and Spark Structured Streaming: transforming streaming data, converting between formats such as JSON and Parquet, and running several streaming queries on one source.
What you'll cover
- ▸Spark Structured Streaming: sources, sinks, and triggers
- ▸Reading from and writing to Kafka
- ▸Stream transformations and JSON flattening
- ▸Format conversion in a stream
- ▸Multiple queries on one stream
What you'll be able to do
- ✓Build a streaming pipeline from Kafka to Spark
- ✓Transform and reshape streaming data
- ✓Convert JSON streams to Parquet
- ✓Run several streaming queries on the same source
Tags
StreamingReal-TimeEvent-DrivenKafkaSparkData EngineeringAdvanced
Prerequisites
- Getting Started with PySpark or equivalent experience
- Basic understanding of streaming concepts
Technologies
Related items
- PySpark Fundamentals
- Effective PySpark
- Intro to Streaming with Kafka
