I am a Data Engineer. I am getting caught up by work all day. As a result I have got imposter syndrome . I would like to have a study buddy who can push me.
If you are interested in Data Engineering we can discuss that as well.
Let me know anyone interested. 🧐
Founder here, so grain of salt, but I think this is genuinely useful for anyone running production pipelines on Databricks.
Before building a startup I was a SWE at a cloud networking company where most of our "data ops" was repetitive manual work. We had a nightly ingestion job that would fail on its last task, so someone would rerun the whole thing from scratch and throw away hours of successful work, and jobs that normally finished in twenty minutes would sometimes run for three hours before anyone noticed because nothing had technically "failed". Meanwhile our data platform team would spend a big chunk of their time fielding requests for backfills, job reruns, and warehouse starts in Slack. And I've lurked here long enough to know it wasn't just us :)
We built Kestrel to codify these painful manual steps as workflows. You describe what you want (e.g. "when a job run fails, pull the failed task's output, check whether the failure looks transient, and repair the run after on-call approves in Slack, so only the failed tasks rerun") and Kestrel builds the workflow for you. Once configured it runs deterministically, so you're not trusting an LLM to improvise against your prod lakehouse at runtime.
Databricks doesn't have tenant-wide outbound webhooks (job notifications are configured per job), so we poll the REST API on a configurable cadence. Things like failed runs, runs exceeding a duration, unexpected cluster terminations, and DLT update failures act as workflow triggers. You can pause anything destructive or expensive - e.g. repairing runs, full DLT refreshes, terminating clusters - at an approval gate so the workflow only continues after someone signs off.
Teams use Kestrel to automate Databricks incident response, repair failed runs, run CI/CD checks (e.g. kicking off an integration test job when a PR opens), run data freshness and row-count checks, recover failed DLT pipeline updates, and stop idle warehouses and clusters.
I put together a few common Databricks workflows so you can poke around: https://demo.usekestrel.ai/workflows/new?simulated=1&bundle=LV9vzHEHaI
Happy to answer questions, and feedback is welcome!
Demo environment: https://demo.usekestrel.ai
Website: https://usekestrel.ai
Docs: https://docs.usekestrel.ai/workflows/create-workflows / https://docs.usekestrel.ai/integrations/databricks
This is just an example, but Apache Spark expertise used to be a real differentiator 10 years ago or so. If you knew it well, you were swimming in job offers.
Then Databricks simplified it and made Spark silly-proof with built-in optimizations left and right. Now the exact same thing is happening to Databricks consultants and FTEs because Databricks know-how just isn't as premium as before anymore as they simplified the platform and genie-d the S out of it.
Abstraction eventually seems to eat all expertise and each cycle seems shorter than the last.
Is there anything left in the data and platform layer with real depth behind it?
And I'm not looking for "just learn AI, bro.