u/FickleAnt4399

Meet Duckle 0.6.X.
You can now import legacy Talend ETL jobs directly into Duckle, preserving the pipeline structure while mapping supported components to Duckle equivalents.
And that’s just one part of the release:
• Import legacy ETL jobs
• Build and run pipelines locally
• Choose from 14 AI models
• Pixeltable read/write support
• CI validation for pipelines
• More powerful spatial transformations

Your existing ETL doesn’t have to become technical debt.
Move from heavyweight ETL infrastructure to a local-first, developer-friendly data pipeline stack.
👉 Try Duckle 0.6.1 - https://github.com/slothflowlabs/duckle

Duckle is an emerging open-source ETL solution built on top of DuckDB that provides a graphical interface for developing high-speed data pipelines.

The platform stands out for its versatility, allowing users to run it locally on various operating systems, in the cloud via VPS, or directly within a web browser.

By leveraging DuckDB as its core engine, it enables efficient data transformation and connectivity across hundreds of sources, including PostgreSQL, Parquet, and Kafka. Key advantages include significant cost savings compared to traditional cloud data warehouses and the ability to deploy pipelines as standalone, low-latency binaries. The project is highly community-driven, incorporating modern features like AI-assisted pipeline design and seamless Git integration for version control. Currently in its early stages, the tool aims to simplify data engineering by removing the need for complex infrastructure while maintaining high performance.

GitHub Repo - https://github.com/slothflowlabs/duckle

We benchmarked Duckle against 4 open source ingestion tools.
Then we published the harness so you can prove us wrong.

96 million rows of TPC-H lineitem, 14 GB in Postgres, out to Parquet, one laptop. The number we care about most is the second one.

The harness ships in the repo. "./bench.sh" all brings up Postgres, generates the data at any scale factor, and times every tool you have installed. No run gets a time until its output has been reopened and checked for the right row count and the right sum(l_orderkey), so a tool that writes a fast wrong file gets a failure instead of a number.

Run it. If you get different numbers, we want to see them.

Repo - https://github.com/slothflowlabs/duckle
Benchmark - https://github.com/slothflowlabs/duckle/tree/main/benchmarks/pg-to-parquet

1 million rows. MySQL → MongoDB. 5.4 seconds.
No Spark cluster. No Docker. No JVM.
Just a simple Duckle pipeline:
duckle.src.mysql(...)
.snk.mongodb(...)
.run()

The pipeline streams data directly between MySQL and MongoDB while DuckDB handles the execution.

What I like most is the simplicity:
Define the pipeline in Python
Execute with a single .run()
No infrastructure to manage
Local-first and open source

This is the kind of developer experience we wanted when building Duckle.
GitHub: https://github.com/slothflowlabs/duckle
PyPI: https://pypi.org/project/duckle/

Duckle is a local-first ETL/ELT framework powered by DuckDB.
You can now define pipelines in Python. DuckDB executes them as optimized, vectorized SQL. Your data stays on your machine from start to finish.

Why Duckle?

✅ No Python bottleneck
Pipelines are compiled into SQL before execution.
No rows flow through the Python interpreter.
No hidden to_pandas() conversions.

✅ Minimal setup
~20 MB install
Bundles the DuckDB CLI
No JVM
No Docker
No server
No account required

✅ Python-first API
import duckle
from duckle import col
(duckle.read_csv("orders.csv")
.where(col.amount >= 20)
.derive(total="round(amount * 1.2, 2)")
.write_parquet("out.parquet")
.run())

Write familiar Python expressions while Duckle translates them into efficient DuckDB SQL.

More than file transformations
Duckle includes 359 built-in components:
-104 Sources
-66 Sinks
-138 Transforms

Supporting databases and services including PostgreSQL, MySQL, SQL Server, Oracle, Snowflake, Databricks, Kafka, Salesforce, SAP OData, S3, SFTP, WebSocket, IMAP, LanceDB, dbt and many more.

Built for automation
Validate pipelines without connecting to data sources or requiring credentials.
duckle validate
duckle validate --json
duckle --pipeline my.json

Perfect for CI/CD, containers, cron jobs, and local development.

Code ↔ Visual Studio
The same pipeline can be authored in Python or opened directly in the Duckle visual studio because both use the same JSON format.

Open source. Local first. Built on DuckDB.
⭐ GitHub: https://github.com/slothflowlabs/duckle/
📦 PyPI: https://pypi.org/project/duckle/
🔗 Links: https://github.com/slothflowlabs/duckle#quick-links

A visual data pipeline studio that runs on your laptop. Drag sources, transforms, validators, and sinks onto a canvas. Wire them together. Press Run. Duckle compiles the graph to SQL and executes it through a real columnar engine, with live previews, generated SQL on every node, and zero hidden state.

Three things make Duckle different from the heavyweights and the toy ETL tools:

1. An AI assistant that ships in the box. Describe the pipeline you want in English; Duckie writes the JSON and drops it onto the canvas. The model runs locally - no API key, no telemetry, no cloud round-trip.
2. 290+ connectors at install time. Files, lakehouses, SQL databases, warehouses, NoSQL, vector DBs, streaming brokers, SaaS REST/GraphQL APIs, even FTP and IMAP - working today, not coming-soon.
3. A self-contained binary you can audit. ~100 MB download. Engines install on first launch. Workspaces are plain files in a folder you choose. Diff them, branch them, ship them.

Free and Forever yours - https://github.com/slothflowlabs/duckle

So we benchmarked Duckle against the ETL tools everyone already
uses, with every tool tuned to its BEST configuration:

The task is deliberately boring: read one CSV(TPC-H Width and style), land it as a table.
It is the single most common job in ETL.

Three things we want to be upfront about, because benchmarks are
easy to rig 🫣:

1. Duckle sits right on raw DuckDB
   own load (CREATE TABLE lineitem AS SELECT * FROM read_csv('out/lineitem_20m.csv') floor (~16s to fully parse and write 20M typed rows to disk). Duckle wraps the engine with pipelines, connectors and
   then gets out of its way. That is the whole design goal.

2. Talend and Informatica used their bulk output connectors at max
   config, not the slow default row-by-row sink. On defaults they
   5-7x slower. We did not want to strawman them.

3. Airbyte's number is derived from real 2M and 5M runs, and it needs
   an always-on 8 GB platform just to start.

Duckle is free and open source.
Try it: https://github.com/slothflowlabs/duckle

Duckle 🎉 v0.5.3 is live.

Duckle by SlothFlowLabs is the local-first, open-source visual ETL/ELT studio built on DuckDB - drag-and-drop pipelines, run locally, your data never leaves your machine.

v0.5.3 is all about trust: knowing exactly what a pipeline did, proving it, and reviewing changes before they even hit run.

Signed run manifests (.ducklock) - every run can record a signed, reproducible manifest that pins source input hashes, per-node outcomes, and column lineage. Verify any run after the fact.

Schema-drift detection + a Trust score - Duckle flags when an upstream source's columns or types change since the last signed run, and scores how trustworthy a pipeline is, right in the editor.

duckle review + data branches - review a pipeline change from the CLI with a live data diff, and branch a DuckDB file to test changes in isolation. Git-style review, for data.

End-to-end column lineage - trace any output column back through every transform and sink to its source columns, with a downstream impact view before you touch a query.

Teradata source + sink - read and write Teradata over ODBC, alongside new MinIO / Cloudflare R2 / Backblaze B2 object-storage sinks.

Live preview - flip it on and selecting or editing a node runs the pipeline up to that node and shows the rows instantly. No full run needed.

Plus run-time parameters in the editor and web dashboard, a seeded sample workspace on first launch, dbt Fusion provisioning, and MCP review tools any LLM can call (diff, impact, contracts, trust report).

100% free, yours and open source.
Github - https://github.com/slothflowlabs/duckle

Duckle is the local-first, open-source visual ETL/ELT studio built on DuckDB. What's new:

GizmoData - GizmoSQL integration - read and write GizmoSQL over a clean-room Arrow Flight SQL client, right from the canvas.

Browser-based, dockerized editor - run the full drag-and-drop Duckle editor in your browser. One docker compose up, open localhost, and build + run
pipelines with live per-node progress. Self-hosted, no cloud, no account.

Qlik QVD read + write - native for Qlik Sense, no Qlik runtime required.

Bring-your-own AI - point the built-in assistant at any OpenAI-compatible endpoint.

Plus bulk SQL Server writes, run-to-here, and a stack of fixes.

100% free, yours and open source.
👉 https://github.com/slothflowlabs/duckle

Duckle is a free, open-source, local-first Data Studio that runs on your laptop: build pipelines on a visual canvas, run them on DuckDB, ship them as a
single binary. No cloud, no account, no telemetry. Your data never leaves your machine.

The latest build (v0.3.0) makes dbt a near-instant, cross-system part of the Duckle Canvas:

- dbt is now supported and dbt Fusion is now the default. A Rust dbt engine: warm project parse/build is ~45 ms, versus the multi-second Python import floor of dbt Core (which is on as an automatic fallback).
- Multi-source dbt. One dbt build reads several wired sources at once (Postgres + MySQL + CSV + Parquet), each materialized as a real table and modeled
through dbt sources. A Customer 360 demo runs 6 sources across 4 system types into 1 dbt build and out to 4 sinks in 4,382 ms.
- Free, self-provisioning. The dbt engine downloads and sets itself up on first launch. No Python setup, no separate install, $0.
- JSON Records-path. Unnest nested REST envelopes (like data or response.records) into real columns.
- Native brand icons + type-to-add. Every source, sink and SaaS connector wears its real logo on the canvas; start typing to fuzzy-search and drop any
connector.
- Production ops. Structured error taxonomy, OpenMetrics export(<workspace>/runs/*.json), backfill and watermark controls, and a Runs history tab.
- Right-click the pipeline, choose Build, and it compiles into a self-contained executable, including DuckDB and it's necessary extensions.
Just copy that file to a server.

Single binary. Engines download on first launch. No installer, no JVM, no control plane. Swap the binary in place and your workspace + engine cache are
untouched.

Repository: https://github.com/SouravRoy-ETL/duckle
Download + full changelog: https://github.com/SouravRoy-ETL/duckle/releases/tag/v0.3.0

Duckle just got a major upgrade.

Duckle is a free, open-source, local-first Data Studio that runs on your laptop: build pipelines on a visual canvas, run them on DuckDB, ship them as a
single binary. No cloud, no account, no telemetry. Your data never leaves your machine.

The latest build (v0.3.0) makes dbt a near-instant, cross-system part of the Duckle Canvas:

- dbt is now supported and dbt Fusion is now the default. A Rust dbt engine: warm project parse/build is ~45 ms, versus the multi-second Python import floor of dbt Core (which is on as an automatic fallback).
- Multi-source dbt. One dbt build reads several wired sources at once (Postgres + MySQL + CSV + Parquet), each materialized as a real table and modeled
through dbt sources. A Customer 360 demo runs 6 sources across 4 system types into 1 dbt build and out to 4 sinks in 4,382 ms.
- Free, self-provisioning. The dbt engine downloads and sets itself up on first launch. No Python setup, no separate install, $0.
- JSON Records-path. Unnest nested REST envelopes (like data or response.records) into real columns.
- Native brand icons + type-to-add. Every source, sink and SaaS connector wears its real logo on the canvas; start typing to fuzzy-search and drop any
connector.
- Production ops. Structured error taxonomy, OpenMetrics export(<workspace>/runs/*.json), backfill and watermark controls, and a Runs history tab.
- Right-click the pipeline, choose Build, and it compiles into a self-contained executable, including DuckDB and it's necessary extensions.
Just copy that file to a server.

Single binary. Engines download on first launch. No installer, no JVM, no control plane. Swap the binary in place and your workspace + engine cache are
untouched.

Repository: https://github.com/SouravRoy-ETL/duckle
Download + full changelog: https://github.com/SouravRoy-ETL/duckle/releases/tag/v0.3.0

Duckle is a free, open-source, local-first Data Studio: build pipelines on a visual canvas, run them on DuckDB, ship them as a single binary. No cloud, no account, no telemetry. Your data never leaves your machine.

What's new in v0.2.0:
- Visual Map: join a main input to lookups across CSV, Parquet, DuckDB, SQLite and warehouses, with per-output expressions and no SQL.
- Parallelize: independent branches run concurrently, auto-scaled to your CPU cores.
- Universal upsert + CDC delete propagation across every relational family plus MongoDB.
- DuckLake CDC change-feed and watermark incremental loads.

Every number in the screenshots ran on a plain 16 GB laptop, nothing fancy:
- 16-node monolithic pipeline (5M-row 3-way Map join + parallel branches + 4 sinks): ~3.0s
- 100k-row DuckLake CDC mirror with upsert + deletes: ~1.7s
- 5,000,000-row watermark incremental load: ~1.8s

Heavy workloads finish before you can blink. And both dark and light themes are tuned to feel native to DuckDB.

Single binary. Engines download on first launch. 60 UI languages.

Repository: https://github.com/SouravRoy-ETL/duckle

Download + changelog: https://github.com/SouravRoy-ETL/duckle/releases/tag/v0.2.0

You can now connect Claude directly to Duckle.

Duckle ships its own MCP server, so Claude (or any MCP client - Claude Desktop, Claude Code, Cursor) can build your data pipelines for you, right inside your local workspace.

Ask in any language, and Claude can:

🦆 Generate a pipeline (simple or complex) into your working directory

🦆 Validate it against 328 connectors (307 available out of the box)

🦆 Run it on DuckDB at native speed

🦆 Package it into a single standalone executable you can schedule anywhere

One click in Duckle ("Connect to Claude") wires it up. No cloud, no servers, no data leaving your machine - the engine and the MCP server both run locally.

Open source, local-first.

https://github.com/SouravRoy-ETL/duckle

You can now connect Claude directly to Duckle.

Duckle ships its own MCP server, so Claude (or any MCP client - Claude Desktop, Claude Code, Cursor) can build your data pipelines for you, right inside your local workspace.

Ask in any language, and Claude can:

🦆 Generate a pipeline (simple or complex) into your working directory

🦆 Validate it against 328 connectors (307 available out of the box)

🦆 Run it on DuckDB at native speed

🦆 Package it into a single standalone executable you can schedule anywhere

One click in Duckle ("Connect to Claude") wires it up. No cloud, no servers, no data leaving your machine - the engine and the MCP server both run locally.

Open source, local-first.

https://github.com/SouravRoy-ETL/duckle

You can now connect Claude directly to Duckle.

Duckle ships its own MCP server, so Claude (or any MCP client - Claude Desktop, Claude Code, Cursor) can build your data pipelines for you, right inside your local workspace.

Ask in any language, and Claude can:

🦆 Generate a pipeline (simple or complex) into your working directory

🦆 Validate it against 328 connectors (307 available out of the box)

🦆 Run it on DuckDB at native speed

🦆 Package it into a single standalone executable you can schedule anywhere

One click in Duckle ("Connect to Claude") wires it up. No cloud, no servers, no data leaving your machine - the engine and the MCP server both run locally.

Open source, local-first.

https://github.com/SouravRoy-ETL/duckle

You can now connect Claude directly to Duckle.

Duckle ships its own MCP server, so Claude (or any MCP client - Claude Desktop, Claude Code, Cursor) can build your data pipelines for you, right inside your local workspace.

Ask in any language, and Claude can:

🦆 Generate a pipeline (simple or complex) into your working directory

🦆 Validate it against 328 connectors (307 available out of the box)

🦆 Run it on DuckDB at native speed

🦆 Package it into a single standalone executable you can schedule anywhere

One click in Duckle ("Connect to Claude") wires it up. No cloud, no servers, no data leaving your machine - the engine and the MCP server both run locally.

Open source, local-first.

https://github.com/SouravRoy-ETL/duckle

You can now connect Claude directly to Duckle.

Duckle ships its own MCP server, so Claude (or any MCP client - Claude Desktop, Claude Code, Cursor) can build your data pipelines for you, right inside your local workspace.

Ask in any language, and Claude can:

🦆 Generate a pipeline (simple or complex) into your working directory
🦆 Validate it against 328 connectors (307 available out of the box)
🦆 Run it on DuckDB at native speed
🦆 Package it into a single standalone executable you can schedule anywhere

One click in Duckle ("Connect to Claude") wires it up. No cloud, no servers, no data leaving your machine - the engine and the MCP server both run
locally.

Open source, local-first.
https://github.com/SouravRoy-ETL/duckle