Building a Production-Grade Streaming ETL Platform with Kafka, ClickHouse, and Python
I have been building a production-grade streaming ETL platform in Python using the NYC Yellow Taxi dataset as a realistic event stream.
The platform ingests Parquet data, validates and enriches records through a domain-driven pipeline, streams events via Kafka, stores analytical workloads in ClickHouse for sub-second querying, and powers real-time Grafana dashboards.
Some of the engineering challenges I focused on:
- Domain-driven architecture and separation of concerns
- Pydantic-based validation and data quality enforcement
- Type-safe Kafka serialization and ingestion workflows
- Resolving Kafka-to-ClickHouse timestamp conversion issues
- Idempotent processing to prevent duplicate writes
- Manual Kafka offset management for reliability
- Dead-letter queue handling and recovery workflows
- Structured logging, metrics, and observability
- Real-time analytics with ClickHouse
I am currently adding Kubernetes orchestration and Terraform-based AWS infrastructure to support cloud-native deployments.
I would appreciate feedback from the ETL and data engineering community, especially around the Kafka consumer design, error-handling strategy, and overall architecture.
I am actively improving the platform and would love to hear suggestions from data engineers and platform engineers. If you find the project useful or interesting, consider giving it a ⭐ on GitHub.