PySpark Fundamentals

Overview

Learn distributed data processing with PySpark. Understand Spark's architecture and DataFrame API, then apply them to build clean, modular, and testable data transformation pipelines.

What you'll cover

• ▸Spark architecture: driver, executors, and distributed computation
• ▸The DataFrame API and column expressions
• ▸Lazy evaluation and the query execution plan
• ▸Partitioning, shuffles, and caching
• ▸Schema enforcement and data types
• ▸Spark's extensible data source ecosystem

What you'll be able to do

• ✓Filter, join, aggregate, and reshape data with the DataFrame API
• ✓Create derived columns and label data for downstream use
• ✓Clean and prepare raw data for data lake storage
• ✓Structure PySpark code into modular, testable transformations
• ✓Write unit tests for data pipelines with pytest