u/ma1f

Author here, so treat this accordingly. Not selling anything, MIT licensed, no company behind it, no hosted tier planned.

Background: I'm a CTO at a small UK fintech and I've spent the last few years assembling the same stack over and over. dbt/sqlmesh for transformation, something for orchestration, something else for ingestion or reverse ETL. Multiple deployments, multiple failure modes, and the seams between them are where time is lost.

The specific thing that annoyed me enough to build something was Python models. In dbt they're a second-class citizen that needs a cloud warehouse with a Python runtime. In SQLMesh they're better but still feel bolted on. I wanted a .py model to sit mid-DAG with SQL either side, in both directions, and for the planner to not care which I'd written.

Obviously I'm aware dbt and SQLMesh have both been bought by FiveTran, i cover that in this article.

So that's the core of it:

python

# models/enriched_events.py
@model()                  # param name IS the dependency
def enriched_events(raw_events):
  for batch in raw_events.reader():     # Arrow in, Arrow out, bounded memory 
    yield add_revenue(batch)

sql

-- models/event_summary.sql — SQL straight over the Python
SELECT country, count(*) FILTER (WHERE is_conversion) AS conversions
FROM enriched_events GROUP BY country

The Python model is a plain function. You can call it in a test with no warehouse and no session.

The rest of the design, briefly:

IR is a sqlglot AST, not Jinja templates. Dependencies come from parsing the SQL, not from ref(). No pandas in core, everything moves as Arrow RecordBatchReader.
By default models builds into a fingerprinted physical table and environments are just views over those. A dev environment reuses prod's tables for free, promotion is an atomic view swap, and rollback is the same swap backwards.
plan / apply in the terraform sense. Changes classify as breaking / non-breaking / forward-only, and column-level lineage impact analysis proves when a downstream output is unchanged so it gets reused rather than rebuilt.
Streams are durable. POST an event, it's fsynced before the 200, deduplicated by idempotency key, and the materialiser commits data and watermark in the same warehouse transaction. Exactly-once without distributed coordination.
One process. interlace serve is the web UI, HTTP API, scheduler and stream ingestion. No Airflow, no broker.
DuckDB by default, DuckLake one config line away, Postgres natively over ADBC.

Where it's weak, and I'd rather you heard it from me:

Single maintainer. That's the honest risk with any tool like this and I'm not going to pretend otherwise.
Snowflake, BigQuery, Redshift and MotherDuck adapters are wired and dialect-correct but have not been run against a live account. Alpha, and labelled as such.
Developed on Linux, CI is Linux only. Nothing in the codebase is platform-specific and every dependency ships mac/Windows wheels, so both should work, but neither is tested.
Spark is beta.
It's new. Real production mileage is limited to my own use.

I've written up a full jaffle_shop migration (a real dbt project, end to end) if you want to see what moving something across actually looks like rather than taking my word for it.

What I'm after here is criticism rather than stars. Specifically:

If you run dbt today, what would actually stop you trialling this on one pipeline? I suspect the answers are "single maintainer" and "my warehouse is Snowflake", but I'd rather know than guess.
Does the fingerprinted-table-plus-view-swap model break in a way I haven't hit yet? I'm particularly interested in whether anyone's tried this at a scale where the number of snapshots becomes a catalog problem.
Anyone doing durable ingestion in-process like this rather than via Kafka/Kinesis? Interested in what bit it, if so.

Repo: github.com/interlace-sh/interlace
Comparison against dbt and SQLMesh, including where they're ahead: interlace.sh/why

Interlace — SQL and Python in one graph