I’ve been building a lot with Spark RTM lately and have been pleasantly surprised with the results of my experiments so far. I built an app that shows trading volumes in real time and pushed some market data into it from a Kafka topic. The end-to-end latency is quite impressive: p50 at 41ms, p90 at 60ms and p99 at 71ms. Much lower than what I was seeing with continuous or for each batch in Spark before, where it was hard to go below 1s latency. And this is the end to end latency, which includes significant latency from serving layer (lakebase) and the browser.

I’m happy to share more about my experience and how to create real time apps with Spark RTM and Lakebase if folks are interested.

I’m currently new to Databricks, and I’m writing unit tests for a pipeline that is entirely in SQL, with Bronze, Silver, and Gold tables.

I asked how I should approach unit testing, and I was told that since unit tests are typically written in Python, I could wrap the pipeline logic in Python files and then use pytest to test those files.

But I’m confused about how this would work in a real production environment.

For example, if we push everything to GitHub and later make a change to the SQL pipeline, we would also have to make the same change in the corresponding Python files so that the unit tests continue to work. That means maintaining the same logic in two places, which doesn’t seem like a good production practice.

So what is the recommended approach for unit testing a SQL-based Databricks pipeline? Is there a way to test the actual SQL transformations directly without duplicating the pipeline logic in Python?

I’m still learning Databricks, so I’d really appreciate some guidance on the proper production approach.

I was given the task to perform type 2 validation check. Can I know the possible approaches

What practices or tools have helped you manage source changes safely in production?

Hey folks,

If an ingestion batch of 1,000 payment records arrives where 995 rows are completely valid and 5 rows have missing critical fields (like receiver_account is NULL):

Do your pipelines usually:

1. Reject/Hold the entire batch to preserve batch atomicity and ordering?
2. Accept the 995 valid rows and route the 5 bad rows to a Dead-Letter Queue (DLQ)?

What are the trade-offs you run into with transactional integrity versus keeping pipeline throughput moving

We put together side-by-side comparisons of 30+ ETL and data integration tools, including features, pricing, and where each tool fits best.

https://etlworks.com/etl-tools-comparison.html

Author here, so treat this accordingly. Not selling anything, MIT licensed, no company behind it, no hosted tier planned.

Background: I'm a CTO at a small UK fintech and I've spent the last few years assembling the same stack over and over. dbt/sqlmesh for transformation, something for orchestration, something else for ingestion or reverse ETL. Multiple deployments, multiple failure modes, and the seams between them are where time is lost.

The specific thing that annoyed me enough to build something was Python models. In dbt they're a second-class citizen that needs a cloud warehouse with a Python runtime. In SQLMesh they're better but still feel bolted on. I wanted a .py model to sit mid-DAG with SQL either side, in both directions, and for the planner to not care which I'd written.

Obviously I'm aware dbt and SQLMesh have both been bought by FiveTran, i cover that in this article.

So that's the core of it:

python

# models/enriched_events.py
@model()                  # param name IS the dependency
def enriched_events(raw_events):
  for batch in raw_events.reader():     # Arrow in, Arrow out, bounded memory 
    yield add_revenue(batch)

sql

-- models/event_summary.sql — SQL straight over the Python
SELECT country, count(*) FILTER (WHERE is_conversion) AS conversions
FROM enriched_events GROUP BY country

The Python model is a plain function. You can call it in a test with no warehouse and no session.

The rest of the design, briefly:

• IR is a sqlglot AST, not Jinja templates. Dependencies come from parsing the SQL, not from ref(). No pandas in core, everything moves as Arrow RecordBatchReader.
• By default models builds into a fingerprinted physical table and environments are just views over those. A dev environment reuses prod's tables for free, promotion is an atomic view swap, and rollback is the same swap backwards.
• plan / apply in the terraform sense. Changes classify as breaking / non-breaking / forward-only, and column-level lineage impact analysis proves when a downstream output is unchanged so it gets reused rather than rebuilt.
• Streams are durable. POST an event, it's fsynced before the 200, deduplicated by idempotency key, and the materialiser commits data and watermark in the same warehouse transaction. Exactly-once without distributed coordination.
• One process. interlace serve is the web UI, HTTP API, scheduler and stream ingestion. No Airflow, no broker.
• DuckDB by default, DuckLake one config line away, Postgres natively over ADBC.

Where it's weak, and I'd rather you heard it from me:

• Single maintainer. That's the honest risk with any tool like this and I'm not going to pretend otherwise.
• Snowflake, BigQuery, Redshift and MotherDuck adapters are wired and dialect-correct but have not been run against a live account. Alpha, and labelled as such.
• Developed on Linux, CI is Linux only. Nothing in the codebase is platform-specific and every dependency ships mac/Windows wheels, so both should work, but neither is tested.
• Spark is beta.
• It's new. Real production mileage is limited to my own use.

I've written up a full jaffle_shop migration (a real dbt project, end to end) if you want to see what moving something across actually looks like rather than taking my word for it.

What I'm after here is criticism rather than stars. Specifically:

1. If you run dbt today, what would actually stop you trialling this on one pipeline? I suspect the answers are "single maintainer" and "my warehouse is Snowflake", but I'd rather know than guess.
2. Does the fingerprinted-table-plus-view-swap model break in a way I haven't hit yet? I'm particularly interested in whether anyone's tried this at a scale where the number of snapshots becomes a catalog problem.
3. Anyone doing durable ingestion in-process like this rather than via Kafka/Kinesis? Interested in what bit it, if so.

Repo: github.com/interlace-sh/interlace
Comparison against dbt and SQLMesh, including where they're ahead: interlace.sh/why

I wrote a whole article about it, link in comments, but I just want share a quick summary here.

This client had over 30 data sources they were ingesting from, it looked something like this:

• posgresql -> snowflake ~100m/rows/mo (some spikes to 1b/rows/mo)
• all other data sources -> snowflake ~100m/rows/mo (some spikes to 1b/rows/mo)

This was costing them almost $100k per year (~$8.5k/month) and it was especially hard to justify because the spikes would normally happen during busy seasons and they already had tight margins.

They had already started using ingestr (the free open source version) to offload some of their smaller jobs and running the jobs on an EC2 instance.

Once they migrated over their production database to snowflake job, they cut the cost significantly - the total server cost came to less than $500/month.

If this resonates to anyone, and you want to put on your resume "helped reduce ingestion cost by x amount", then check out ingestr repo and docs.

Is it bad source data, schema changes, performance issues, monitoring gaps, or something else?

Snowflake's free trial (30 days/$400 credits) is too short for a university thesis running several months – need something that doesn't expire mid-project. What are people using instead? Also curious if anyone knows of a longer free tier/subscription (student program, extended trial, etc.) for Snowflake specifically before I give up on it.

Hey everyone! 👋

I came across a few openings at my company and thought I’d share them here in case anyone is looking for a switch or a new opportunity.

Here are the roles available:

📍 PAN India

AI Engineer: Exp: 5-8 yrs

Python + GenAI: Exp: 8-10 yrs

📍 Pune

MS Azure AI/ML Exp: 5-8 yrs

Python + GenAI : Exp: 8-10 yrs

📍 Hyderabad

Data Scientist: Exp: 5-8 yrs

AI Cognitive + Snowflake + Python: Exp: 8-10yrs

📍 Bengaluru

Data Scientist: Exp: 5-8 yrs

Gen AI

📍 Chennai: Exp: 8-10 yrs

Data Scientist

💰 Salary: Not specified in the job posting / Depends on experience and role.

If you're interested in any of these roles, feel free to DM me or directly send me your updated resume at:

📧 dkviswa889@gmail.com

To make things easier, please include:

First Name:

Last Name:

Mobile Number:

Gmail ID:

Country:

Updated Resume: Attached

📌 Email subject / resume name:

Job_Title_FirstName_LastName

Example:

Python_GenAI_Rahul_Kumar

I'll try my best to refer suitable profiles. 😊

Good luck to everyone who's looking! 🤝

If you've any queries on the job role, pls feel free to DM! I'll share the skill stack. So you'll get an idea 💡.