u/namarv

Founder here, so grain of salt, but I think this is genuinely useful for anyone running production pipelines on Databricks.

Before building a startup I was a SWE at a cloud networking company where most of our "data ops" was repetitive manual work. We had a nightly ingestion job that would fail on its last task, so someone would rerun the whole thing from scratch and throw away hours of successful work, and jobs that normally finished in twenty minutes would sometimes run for three hours before anyone noticed because nothing had technically "failed". Meanwhile our data platform team would spend a big chunk of their time fielding requests for backfills, job reruns, and warehouse starts in Slack. And I've lurked here long enough to know it wasn't just us :)

We built Kestrel to codify these painful manual steps as workflows. You describe what you want (e.g. "when a job run fails, pull the failed task's output, check whether the failure looks transient, and repair the run after on-call approves in Slack, so only the failed tasks rerun") and Kestrel builds the workflow for you. Once configured it runs deterministically, so you're not trusting an LLM to improvise against your prod lakehouse at runtime.

Databricks doesn't have tenant-wide outbound webhooks (job notifications are configured per job), so we poll the REST API on a configurable cadence. Things like failed runs, runs exceeding a duration, unexpected cluster terminations, and DLT update failures act as workflow triggers. You can pause anything destructive or expensive - e.g. repairing runs, full DLT refreshes, terminating clusters - at an approval gate so the workflow only continues after someone signs off.

Teams use Kestrel to automate Databricks incident response, repair failed runs, run CI/CD checks (e.g. kicking off an integration test job when a PR opens), run data freshness and row-count checks, recover failed DLT pipeline updates, and stop idle warehouses and clusters.

I put together a few common Databricks workflows so you can poke around: https://demo.usekestrel.ai/workflows/new?simulated=1&bundle=LV9vzHEHaI

Happy to answer questions, and feedback is welcome!

Demo environment: https://demo.usekestrel.ai

Website: https://usekestrel.ai

Docs: https://docs.usekestrel.ai/workflows/create-workflows / https://docs.usekestrel.ai/integrations/databricks

Founder here, so grain of salt, but I think this is genuinely useful for anyone running production traffic through Cloudflare.

Before building a startup I was a SWE at a cloud networking company where most of our "edge ops" was repetitive manual work. We had health checks that would fail overnight while traffic kept flowing to the dead origin until on-call woke up, and WAF events that would fire as attackers kept hitting our login endpoint while security dug through the Cloudflare dashboard. Meanwhile our platform team would spend a big chunk of their time fielding developer requests for new DNS records and firewall rules in Slack. And I've lurked here long enough to know it wasn't just us :)

We built Kestrel to codify these painful manual steps as workflows. You describe what you want (e.g. "when a health check reports an origin down, shift the load balancer weights to the healthy origin after on-call approves in Slack") and Kestrel builds the workflow for you. Once a workflow is configured it runs deterministically, so you're not trusting an LLM to improvise against your prod edge at runtime.

We use Cloudflare Notifications webhooks as workflow triggers, so WAF/DDoS events, health check failures, failed Worker deploys, and expiring certs fire workflows immediately. You can pause anything risky - like production DNS, pool drains, ASN blocks, and Access policy changes - at an approval gate so the workflow only continues after someone signs off.

Teams use Kestrel to automate Cloudflare incident response, deploy new services to the edge (health checks + DNS + Tunnels), handle self-service developer requests, manage rate limits and cert renewals, shift canary weights, and on/offboard contractors through Access policies.

I put together a few common Cloudflare workflows so you can poke around: https://demo.usekestrel.ai/workflows/new?simulated=1&bundle=rrZAL03VqY.

Happy to answer questions, and feedback is welcome!

Demo environment: https://demo.usekestrel.ai

Website: https://usekestrel.ai

Docs: https://docs.usekestrel.ai/workflows/create-workflows / https://docs.usekestrel.ai/integrations/cloudflare