Shuffles, Partitioning & Persistence
The performance trio: what really happens during a shuffle, how partitioning controls parallelism, and when caching actually helps.
Pass Week 2 to unlock this content.
Each week of the study path opens after you score 80% or higher on the previous week's quiz. This isn't to gatekeep — it's because the concepts in later weeks build directly on the ones before them, and the quiz is the cheapest way to find out whether the foundation is in place.
Go to Week 2What you'll cover in Week 3
Once unlocked, Week 3 runs roughly 90 minutes of reading paired with 4 interactive visualizations, followed by a 15-question self-check quiz. The reading is grounded in the official Apache Spark documentation — every claim cites the docs.
- Anatomy of a shuffle: map side, disk, network, reduce side
- Why shuffles are expensive (network, disk, serialization, GC)
- Partitioning depth and the Goldilocks problem
- Hash partitioner, range partitioner, custom partitioners
- repartition vs coalesce
- Data skew — the silent performance killer
- Storage levels and choosing the right one
- Checkpointing — the bigger hammer
Why this week matters
By the end of Week 3 you'll be able to explain shuffles, partitioning & persistenceconfidently — not just describe it, but reason about edge cases, predict performance, and read a Spark UI for the concepts it touches. That's the bar this study path aims for: not memorization, but the kind of working understanding that lets you debug real jobs.