r/ETL

Hi everyone,

I started building `det` (https://github.com/det-Labs/det), an open-source ETL and data engineering framework designed around high-performance native execution, with an embedded data catalog module built directly into the engine core.

### The Problem

In modern data engineering workflows, ETL/transformation engines and data catalogs are treated as completely detached layers. Catalogs (DataHub, OpenMetadata) typically require separate, heavy distributed infrastructure (Elasticsearch, Kafka, JVM services), while transformation engines (dbt, Spark) execute transformations blind to metadata until an external crawler reconciles the state.

`det` combines both: an execution engine where schema detection, metadata cataloging, and data transformation share the same zero-copy Arrow/Polars foundation.

### Current Implementation (Catalog Core Working)

The foundation is ~25% complete, with the data catalog module functional today:

- Ingestion and inspection of storage layouts to extract schema/metadata directly.

- Local metadata indexing with zero background daemon overhead.

- Rust-native performance designed to feed directly into analytical transformation pipelines.

### Roadmap: Transformation & Execution Layer

The next phase expands `det` into a full execution engine:

1. In-memory data transformations leveraging Polars and Apache Arrow zero-copy memory layouts.

2. Direct pipeline execution driven by the cataloged metadata contracts.

3. Plug-and-play connector interfaces for source/sink abstractions.

### Seeking Review & Contributors

Rather than designing the execution engine in isolation, I am looking for feedback from data engineers on:

- **Engine contracts:** Designing the execution DAG to interact natively with the embedded catalog metadata.

- **Modularity:** Interface design for adding custom connectors and storage backends.

Issues are tagged with `good first issue` and `help wanted` for contributors interested in systems-level data engineering tooling.

Repository: https://github.com/det-Labs/det

Hi,

I got a task to run 120 SP on MS SQLServer to create output and store it as a tab or csv files.
Existing approach is to use SSIS but I think it's too much work comparing with other modern tools.

What you would use for this job? I'm thinking go Python with cursor.execute(sql).
It's so much easier to scale and implement. You can read sp, params, output name from some input List and run it in one shot.

Do you see any disadvantages for this approach vs traditional SSIS. I think my bosses will be scary a bit, because they are old school.

Do

Thanks

Streaming ETL feels like a different mindset altogether. The basics make sense, but I'm still trying to picture how people deal with joins against reference data, retries and more complicated transformations without relying on batch windows.

If you've been through the transition, what surprised you the most?"

Do you rollback everything and start again, or resume from the failed step?

Also how do you avoid duplicate records when the job is rerun?

Curious how ppl handle this in real prod pipelines....

Hello! I'm working on a small data project using a NASA API with Python. The idea is to extract the orbital elements and close approach (asteroids, comets, etc) objects (and there orbital elements) of each planet in order to perform some physics simulations with the data.

I was planning to use a Cloudflare R2 bucket to store the raw API responses, then transform them into a PostgreSQL database, and use FastAPI to consume the data.

I'm not sure how to perform the extraction and loading processes. In theory, each planet will follow this path:

1. Fetch the orbit elements for the own planet (use API 1);

2. Fetch the close approach object (use API 2);

3. Fetch the orbit elements for each of the close approach objects (use API 1).

I use two APIs: one for the orbital elements and one for the close approaches. Should I fetch all the data, store it in memory, and then send it to the bucket? Or would it be better to load it right away with each API call? So, when fetching the close approach objects' orbit elements, get the JSON from the bucket and then use the first API and store that raw response in the bucket