Democratizing Spark expertise: How we built an AI agent to debug distributed jobs

A distributed job is slow, expensive, or failed, and someone has to figure out why. That sounds straightforward, until you’re the one staring at a dozen signals on a dashboard at 9 PM, trying to work out which one actually matters.

Senior MLOps Engineers HsinTing and Steven, and Data Engineer Robert, set out to change that. Rather than write another runbook destined to go stale, they decided to encode the entire reasoning chain into a Claude Code skill, capturing not just the data collection, but the judgment about what the data means and what to do about it.

This post walks through what they built: how the skill gathers context across infrastructure, execution telemetry, and source code, how it reasons through a problem the way a senior engineer would, and what they learned about context engineering, determinism, and turning hard-won expertise into something the whole team can use.

{{divider}}

The problem with debugging distributed jobs

At GetYourGuide, we run a diverse set of Spark workflows: CPU-bound feature-engineering pipelines, memory-intensive model training, GPU jobs for deep learning, and I/O-Intensive batch jobs that read from large Delta tables. Each type fails in different ways, and diagnosing them requires a different kind of expertise. To make a good decision about a slow job, you need to hold three things in your head at once and reason across all of them:

Resource utilization: Is the cluster over- or under-provisioned? CPU at 55% average sounds fine, but the per-node breakdown might tell a different story: the driver at 85% while workers sit at 30% isn’t a sizing problem; it’s a bottleneck in code running on the driver. The aggregate number masks it entirely.

Execution telemetry: One stage took 40 minutes out of a 90-minute run. But why? Is the p95 task duration 10× the median? That’s skew. Is most of the time spent waiting for data from other nodes? That’s a shuffle bottleneck. Is garbage collection eating 30% of CPU alongside disk spill? That’s memory pressure. Each pattern has a different fix, and none of them are labeled for you.

The source code: Which stage corresponds to which transformation in the pipeline? You trace through annotations, match stage IDs to operations, follow imports into helper functions, until you find the line generating the expensive shuffle. Only then can you reason about whether the fix is adding a repartition, changing a join strategy, pushing a filter earlier, or restructuring the pipeline entirely.

The decision space is wide. A slow job might need more workers, or fewer. None of these answers comes from looking at one signal in isolation. They require synthesis across different layers, with each informing the others.

What we built

We built a Claude Code skill called databricks-debugger. You give it a Databricks job run URL and the repository, and it drives the full investigation, pulling data from the right sources, reasoning about what it means, and either recommending a fix or opening a PR directly. 

Where the data comes from

The skill pulls from four sources in parallel: job metadata from the Databricks CLI (cluster config, instance type, run timestamps, custom tags), resource utilization from Datadog (CPU, memory, GPU, network I/O per node), Spark execution telemetry from Overwatch (an open source tool that ingests Spark event logs into queryable Delta tables, giving us structured stage-level data beyond what the Spark UI exposes) and source code cloned from the provided repository.

A decision tree, not a checklist

The skill works through the problem in priority order, stopping as soon as it finds a clear answer.

‍

1. Infrastructure: If the cluster is grossly over-provisioned, a GPU is sitting idle, or memory pressure is causing spill, it surfaces a right-sizing recommendation and stops. No point going deeper when the fix is just switching instance type.

2. Spark execution: If infrastructure looks healthy but the job is still slow, it queries Overwatch for a full stage-by-stage breakdown and reasons about the patterns: high fetch-wait with low CPU is a shuffle bottleneck; high GC alongside spill is memory pressure; p95 task duration at 10× the median is skew. Each leads to a different recommendation: partition tuning, join strategy, or filter pushdown.

3. Code changes: When the fix requires touching source code, the skill maps the slow stage to a specific file and line, proposes a diff, and opens a PR. It knows which repo to target based on context encoded in the skill; a job in a team repo gets a PR there; a job wired through our Airflow monorepo gets a PR to that repo instead.

‍

The impact of our agent

Building the agent was, in itself, a lesson in context engineering. We learned that an agent is only as good as the context it receives. Once we got that right, it could reliably debug failed pipelines and identify meaningful optimizations. A few examples:

• Low-hanging fruit, captured automatically. It identified a straightforward migration to Graviton instances, cutting pipeline execution time by 30% with no change to the logic.
• A flaky pipeline stabilized. The agent diagnosed an intermittently failing data pipeline by tracing the issue to failed Spark executors, identified the underlying root cause, and proposed the necessary fixes. As a result, the pipeline became both more reliable and 20% faster, with no changes to its business logic.
• A Data Scientist resolved a complex Spark pipeline failure. A data processing pipeline that had been failing with opaque error messages and running for over 12 hours on a 15-node cluster was debugged and optimized end-to-end by the agent to run under 20 minutes with 50× less resources. The agent identified long-running stages caused by an inefficient data structure that introduced severe data skew, and performed the necessary changes to resolve the issue. This demonstrates how individual contributors can independently optimize complex Spark workloads and overcome performance bottlenecks that would traditionally require specialized Spark expertise.

What we learned

Building this was primarily an exercise in context engineering, not prompt engineering. The challenge wasn’t improving the model’s reasoning, but giving it the right information. We normalized data from multiple systems and transformed Spark logs into a compact execution profile containing metrics such as stage durations, task skew, and shuffle volumes. The lesson was simple: an agent is only as good as the context it receives.

Consistency is difficult to achieve, and so is determinism. Different engineers investigating the same job may focus on different signals, apply different thresholds, and ultimately reach different conclusions. Encoding the reasoning into a skill makes the analysis more repeatable, consistent, and auditable. However, there is an inherent trade-off: providing too much guidance can reduce creativity and limit the diversity of approaches. Achieving consistent, trustworthy results, therefore, required collecting a diverse set of examples, evaluating the skill’s responses, and iteratively refining its instructions. Due to time constraints, this process was carried out manually. With more time, a comprehensive evaluation suite could automate much of this refinement and further improve the skill’s performance.

The biggest impact is democratizing expertise. Instead of spending time navigating multiple tools and piecing together context, engineers can immediately focus on understanding the problem and deciding how to fix it.

Conclusion

Debugging a distributed job has always required holding a lot of context simultaneously: infrastructure metrics, execution telemetry, and source code, and synthesizing it into a decision. Get context engineering right, and the skill becomes more than automation; it becomes a way to encode the patterns your best engineers have internalized and make them available to everyone. That’s how we scaled expertise and ensured the best thinking in the room never gets locked inside one person’s head.

This is only the first version, and it’s already proven its worth. We’ve run it across a range of pipelines and seen concrete results: cost optimizations, greater stability, and faster debugging when something goes wrong. The next steps are to broaden the signals the agent reasons about and to integrate them into our internal SRE agents, to automate and improve ease of use.

Interested in building with the latest AI tools alongside a talented team of engineers? Join us on our mission to bring unforgettable experiences to millions. Check out our open roles here.

‍