DataFrame of Mind
Build data pipelines with Polars and PySpark. Learn when to use single-node processing and when to scale out.
Data Platform
intermediate
8h
Overview
Build data pipelines with Polars and PySpark. Learn when to use single-node processing and when to scale out.
What you'll cover
- ▸Operational vs. analytical data and schema strategies
- ▸Relational algebra: project, filter, join, aggregate, window
- ▸Polars expressions, contexts, and lazy evaluation
- ▸PySpark DataFrame API and distributed processing
- ▸Query optimization and user-defined functions
- ▸The big data landscape: single-node vs. distributed
What you'll be able to do
- ✓Build data transformation pipelines in Polars and PySpark
- ✓Apply relational operations across both frameworks
- ✓Use lazy evaluation and query plans to optimize performance
- ✓Choose the right processing tool for a given workload
Tags
DataFramesData PipelinesDistributed ComputingPolarsPySparkPythonData Engineering
Prerequisites
- Basic Python knowledge
