Spark on Kubernetes
Submit, configure, and troubleshoot Spark jobs running on a Kubernetes cluster.
Data Platform
advanced
8h
Overview
A one-day advanced course on running Spark on Kubernetes: how the two fit together, spark-submit with Kubernetes flags, resource management, and cluster configuration.
What you'll cover
- ▸How Spark schedules driver and executors on Kubernetes
- ▸spark-submit and Kubernetes-specific configuration
- ▸Resource requests, limits, and dynamic allocation
- ▸Cluster setup for Spark workloads
What you'll be able to do
- ✓Submit and manage a Spark job on a Kubernetes cluster
- ✓Size resources for a Spark workload
- ✓Troubleshoot a failing Spark job on Kubernetes
Tags
Spark on K8sContainer OrchestrationResource ManagementSparkKubernetesData EngineeringAdvanced
Prerequisites
- Getting Started with PySpark or equivalent Spark experience
- Kubernetes or equivalent container orchestration knowledge
