We just shipped Session Restore for Serverless Jobs (beta): debug a job run without rerunning it
👋 I'm a PM working on Serverless Notebooks at Databricks. We just shipped Session Restore for Serverless Jobs in beta, and I'd love your feedback.
The problem: A notebook job runs for hours, fails or produces weird output, and by the time you debug it, the state is gone. Rerunning a 16-hour job is painful. Reproducing an issue that happens once every few months can be nearly impossible.
What we shipped: You can now restore the Python variables and Spark session from a serverless job run into a new interactive notebook, without rerunning the job.
From the job run details, click "Clone into new notebook" or "Debug in new notebook" for a failed run. You can then:
- Inspect state from a failed run
- Investigate weird output without rerunning expensive queries
- Run expensive setup as a job, then pick up interactively from that state
Try it: A workspace admin can enable "Session restore for serverless jobs" from the Previews page.
This was also demo'd at DAIS this year: https://www.databricks.com/dataaisummit/session/modern-databricks-notebook-tips-ai-workflows-and-best-practices-every
If you try it, feel free to leave feedback or respond here. 🙏