Watch the video version for the demos and screen shares: https://www.youtube.com/live/jCQx4XvrEUk
Holden Karau (Snowflake, formerly Databricks) and Lisa Cao (Databricks DevRel) joined us to dig into what it actually looks like to write Spark code with AI agents in 2026.
We worked through a real example from Holden's High Performance Spark Second Edition — a Goldilocks-style exact percentiles problem across hundreds of columns — and watched Claude Code handle it from a cold-start repo, then again with proper context and a reference solution.
Lisa walked through using a skill file to get Claude to write declarative pipelines the right way instead of hallucinating the syntax, and we talked about why local mode, Arrow UDFs, and the upcoming Python UDF transpilation work in Spark 4.3 matter for AI-assisted Spark development.
We also covered: why the "agents trying to shut you up" failure mode shows up in code generation, the case for property-based testing as a guardrail for agentic coding, Spark 3 to Spark 4 porting as a more reliable path than one-shotting Spark 4, and where agents still struggle (dependency management, Spark internals, real stack trace debugging vs. pattern matching from Stack Overflow).
Resources:
- Apache Spark: https://spark.apache.org & https://github.com/apache/spark
- High Performance Spark (O'Reilly): https://www.oreilly.com
- Spark testing base: https://github.com/holdenk/spark-testing-base
- Databricks Spark: https://www.databricks.com/spark/about
- Data + AI Summit (June 15–18, San Francisco): https://www.databricks.com/dataaisummit
- Snowflake Summit (June 1–4, San Francisco): https://www.snowflake.com/summit
- PySpark Data Sources: https://github.com/allisonwang-db/pyspark-data-sources
- PySpark SDP: https://github.com/lisancao/pyspark-sdp
- Distributed Computing 4 Kids: https://distributedcomputing4kids.com/
What's working for you when you write Spark with agents? Drop your patterns and gotchas below. And let us know what you want to see on future episodes.
Originally streamed May 13, 2026 on Vibe Check Temporal YouTube channel and you can watch the full episode at: https://www.youtube.com/live/jCQx4XvrEUk