Is Databricks Classic Compute getting too heavy for small workloads?
I’ve been using Databricks Classic Compute for a while, and recently I’ve started wondering whether cluster cold starts are getting noticeably heavier with newer DBR versions.
For example, with DBR 18 LTS, I tried a small 2-vCPU VM for a single-node job cluster and hit DriverStartupTimeout after 300 seconds. Databricks even suggests that this commonly happens on instances with fewer than 4 CPU cores.
That feels a bit surprising for workloads that are not actually Spark-heavy — e.g. running Python, dbt-core, API calls, or using Databricks mainly as a job runner inside a VNet.
I like Classic Compute because of the flexibility and straightforward VNet/private networking. Serverless is attractive for startup time, but in our environment it would mean quite a bit more networking setup.
So I’m curious:
- Have you noticed Classic Compute cold starts getting slower or more resource-hungry across newer DBR versions?
- Do you now consider 4 vCPUs the practical minimum for a reliable driver?
- Has anyone benchmarked the same VM size across DBR 14/15/16/17/18?
- What are you doing for lightweight non-Spark workloads where you still want Classic Compute?
Update: Single node DBR 18 LTS + Standard_D4pls_v6 cold start spent almost 11 minutes 'Waiting for resources' ( Azure Japan East )
