Workflow Orchestration with Airflow

Manage multiple workloads with the orchestration framework of Apache Airflow.

Data Platform

intermediate

Overview

A one-day workshop on orchestrating data pipelines with Apache Airflow: DAGs and operators, templating, variables and connections, idempotency, and how to structure a DAG so it can be rerun safely.

What you'll cover

▸Workflow orchestration and why pipelines need it
▸DAGs, tasks, and operators
▸Jinja templating, macros, and variables
▸Connections and secrets
▸Idempotency and reruns

What you'll be able to do

✓Write a DAG that chains several operators
✓Debug a failing task or a slow DAG run
✓Design an idempotent pipeline
✓Rerun part of a workflow after fixing an issue

Prerequisites

Basic Python (classes, class instances, importing from modules)

Technologies

airflow python

Additional material

Apache Airflow Documentation →