I still remember the first time I killed a production pipeline with a single line of code. I was migrating a legacy ETL job from a single-node Python script to PySpark. The logic involved some complex string parsing that I had already written in a helper function. Naturally, I did what any deadline-pressured engineer would do: I wrapped it in a udf(), applied it to my DataFrame, and hit run.
The job, which processed 50 million rows, didn’t just run slow — it crawled. What should have taken minutes took hours. I spent the next day staring at the Spark UI, wondering why my 20-node cluster was being outpaced by my laptop.