Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Foundations

• Introduction to the Big Data ecosystem and the role of Spark in contemporary data platforms
• Exploring Spark architecture: driver, executors, cluster manager, lazy evaluation, DAG, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining the appropriate use case for each
• Establishing and configuring SparkSession, including an understanding of fundamental application configuration

Module 2: PySpark DataFrames

• Reading from and writing to enterprise data sources and formats: CSV, JSON, Parquet, and Delta
• Manipulating PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and nested data processing
• Applying data quality checks and developing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Fundamentals for Spark Users

• Understanding the relationship between Docker images, Kubernetes pods, and Spark driver/executor processes – a conceptual perspective for cluster consumers rather than administrators
• Tracing the lifecycle of a Spark job on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings to these resources
• Distinguishing between controls available via Spark session parameters and those managed by the platform team through cluster provisioning – and why this distinction is critical for tuning
• Hands-on exercise: Examine an active Spark-on-Kubernetes job in the sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes – Deep Dive

• Analyzing the trade-off between executor count and size: balancing memory versus cores
• Configuring driver and executor memory, memory overhead, and their translation into pod resource allocations
• Dynamic allocation: evaluating its behaviour on Kubernetes, identifying scenarios where it conserves resources versus where it does not
• Hands-on exercise: Execute the same job with varying executor and core configurations to compare runtime and resource utilisation

Module 5: Scaling Behaviour & Cost Optimisation

• Assessing how adding or removing nodes affects job behaviour, completion time, and resource consumption
• Comparing the performance and cost implications of using many small executors versus fewer large ones
• Tuning shuffle behaviour and shuffle partitions, including estimating the cost impact of configuration choices
• Hands-on exercise: Expand the sandbox cluster from five to ten nodes to observe changes in job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• Diagnosing the small-files problem: understanding why sources split into numerous 1–5 MB Parquet files degrade performance and distort partitioning
• Employing repartitioning and coalescing strategies effectively
• Managing partition sizes during both read and write operations
• Writing Parquet files efficiently to prevent the recreation of the small-files problem in downstream processes
• Hands-on exercise: Load a dataset composed of many small Parquet partitions, apply various repartitioning strategies, and evaluate performance improvements

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying the root causes of out-of-memory errors in Pandas and recognising their symptoms
• Implementing memory-efficient conversion patterns between Spark and Pandas
• Preventing memory blow-ups when exporting large datasets to CSV
• Utilising chunked processing and dtype optimisation in resource-constrained environments
• Hands-on exercise: Reproduce a typical Pandas out-of-memory scenario and resolve it using chunking and dtype optimisation

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: analysing performance characteristics, lazy evaluation, and memory behaviour
• Understanding the role of Polars alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Hands-on exercise: Refactor a Pandas-heavy transformation using Polars to compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning, and memory-management principles across a realistic ETL pipeline
• Recognising optimisation considerations specific to machine-learning workloads running on the same cluster
• Implementing a systematic tuning workflow to diagnose cost and performance issues
• Hands-on exercise: Complete an end-to-end mini-project involving data loading, transformation, and basic model training, with participants independently tuning the Spark configuration