Showcasing det: A lightweight Rust/Polars ETL engine with an embedded data catalog (looking for architecture reviews & contributors)
Hi everyone,
I started building `det` (https://github.com/det-Labs/det), an open-source ETL and data engineering framework designed around high-performance native execution, with an embedded data catalog module built directly into the engine core.
### The Problem
In modern data engineering workflows, ETL/transformation engines and data catalogs are treated as completely detached layers. Catalogs (DataHub, OpenMetadata) typically require separate, heavy distributed infrastructure (Elasticsearch, Kafka, JVM services), while transformation engines (dbt, Spark) execute transformations blind to metadata until an external crawler reconciles the state.
`det` combines both: an execution engine where schema detection, metadata cataloging, and data transformation share the same zero-copy Arrow/Polars foundation.
### Current Implementation (Catalog Core Working)
The foundation is ~25% complete, with the data catalog module functional today:
- Ingestion and inspection of storage layouts to extract schema/metadata directly.
- Local metadata indexing with zero background daemon overhead.
- Rust-native performance designed to feed directly into analytical transformation pipelines.
### Roadmap: Transformation & Execution Layer
The next phase expands `det` into a full execution engine:
In-memory data transformations leveraging Polars and Apache Arrow zero-copy memory layouts.
Direct pipeline execution driven by the cataloged metadata contracts.
Plug-and-play connector interfaces for source/sink abstractions.
### Seeking Review & Contributors
Rather than designing the execution engine in isolation, I am looking for feedback from data engineers on:
- **Engine contracts:** Designing the execution DAG to interact natively with the embedded catalog metadata.
- **Modularity:** Interface design for adding custom connectors and storage backends.
Issues are tagged with `good first issue` and `help wanted` for contributors interested in systems-level data engineering tooling.
Repository: https://github.com/det-Labs/det