r/ETL

Hi everyone,

I’ve been a Test Lead for over a decade, mostly focused on backend database and ETL validation in banking, insurance, and retail. On paper, it looks like a solid career. In reality, I feel like the ground is shifting under me, and I don't know how to keep up.

My strength is SQL and data logic, but even my confidence there has taken a hit lately. I’ve found myself leaning on AI tools to write scripts for me instead of doing it manually, which makes me doubt how sharp I actually still am. Meanwhile, job postings want Selenium, Playwright, APIs, BDD, CI/CD, and AI certifications... I have surface-level exposure to some of it, but no real depth.

Every time I sit down to try to plan how to fix this, I completely freeze. There are too many directions, too little time, and life doesn't pause for upskilling. I don't even know where to start or what a realistic plan looks like without burning out completely.

I'm posting because I need help getting out of this rut:

1. If you transitioned from ETL/Data testing into modern automation, how did you do it? What did you focus on first, and what tools actually matter vs. what is just keyword hype?
2. How do you structure learning when you're already exhausted from work? Any tips for breaking the paralysis?
3. Does anyone want to tackle this together? I don't have a concrete plan yet—I want to build one based on the feedback here. If you are also a mid-to-senior QA feeling lost and wants to connect, build a roadmap together, and keep each other accountable, please let me know.

I’d genuinely love any suggestions, course recommendations, or just to connect with anyone else who is trying to reinvent themselves right now. Would mean a lot to know I'm not doing this alone.

Looking for real-world experiences and lessons learned from production ETL pipelines.

Just got this from the sales team in case anyone was planning an upgrade or migration. The fire sale of Pentaho to Constellation is completed with 130 staff laid off this week leaving barely any product engineering or customer support.

After an extensive strategic review, Pentaho are pleased to announce that all of our products have officially entered the phase that industry analysts have predicted for years:

END OF LIFE

As customers modernise their data platforms with faster, more capable and AI-native technologies, our role is no longer to persuade them to stay—it is to help them leave gracefully.

Accordingly, our mission has evolved.

We are no longer in the business of selling software.

We are now in the business of ensuring our remaining customers complete the fastest, lowest-risk migration possible.

We therefore recommend that customers evaluate modern open platforms including:

\* Argo

\* Apache Airflow

\* Flyte

\* Dagster

We sincerely thank everyone who helped build a product that outlived several generations of enterprise architecture diagrams.

THE AI ERA

The economics of migration have fundamentally changed.

Modern AI-assisted engineering tools can analyse legacy PDI transformations, explain decades-old business logic, generate equivalent pipelines, create documentation, produce automated test suites and accelerate migration projects that previously required months of specialist consultancy.

Whether your destination is Apache Hop, Apache Airflow, Dagster, Argo or another contemporary platform, AI is dramatically reducing both the cost and risk of modernisation.

For the first time in our industry's history, customers are no longer locked into legacy technology because migration is prohibitively expensive.

The future belongs to organisations that can evolve quickly.

Looking back, perhaps our greatest contribution wasn't simply building software that served customers faithfully for decades.

It was remaining available long enough for artificial intelligence to make saying goodbye faster, cheaper and considerably less painful.

Hi everyone, I need some advice on implementing data pipeline for the analytics application in healthcare
Our current tech stack / architecture is as below

1. Microservices architecture
2. Backend services are written in .NET Core and most of the front end is in react
3. Part of the system is still legacy and it is in asp.net and MsSQL
4. Databases used are MySQL, MongoDB, MSSql
5. Kafka is used for pub/sub
6. Applications in production running on GKE

Now we need to implement data pipeline for analytics and I am mostly leaning towards medallion architecture and what I have thought so far is

1. A analytics worker service sitting in same GKE and listening to Kafka topic
2. Periodically push the data to GCS bucket (bronze layer)
3. Cloud scheduler triggers the cloud function at fixed interval and takes the not processed files and batch loads into BigQuery (silver layer)
4. Data farm takes from BigQuery silver layer and create one BigQuery dataset per tenant (gold layer)

Suggestions I need from community

1. Is this is right architecture or any better approach is there?
2. Worker service when it reads from Kafka should use a temporary database to store the data and on batch full send it to GCS or should I consider Kafka itself as a storage and do not commit offset until batch is full and uploaded to GCS
3. Some Kafka events may require enrichment by calling other service APIs, and bulk apis may not be available so how I can effectively handle enrichment + batch upload
4. In case if I also need to connect to legacy database to poll and get the changed data how I can make sure both processes creates the correct order batches (mostly this use case should not come since CDC is enabled in legacy DBs and it publishes the changes to Kafka using a tool similar to Debezium )

Hello to all,

I have curently changed my team and they only have MsSql as storage solution and PBI as reporting solution.

I want to create a proper architecture,that will cover ingestion,etl ,storage ,reporting and later on data science stuff (with keeping previous two stuff for storage and reporting)...

I do not know what softwares are the best for the job ,and some of the options that I know might pass in our company are as follows:

-MS Fabric

-Knime analytics(i have used this before) heavily

-Databricks ( one i am most interested in)

I am talking here about only ETL part that I can set automatic schedules....

I am still a junior so, I am in need of some advice which is the best ...or maybe even get some other advice that would help me!

Thanks upfront

I burned through 6 million tokens on Claude Code last quarter. It started as a productivity boost, I was shipping faster than ever, crushing dbt models and ingestion scripts in hours instead of days.

Then Claude went down for about half an hour during a pretty routine ETL fix. I just sat there staring at the terminal. Couldn't remember how to structure a simple window function without it suggesting the whole thing. That's when it hit me: I've outsourced my working memory to an autocomplete tool.

Coding assistants are great for finishing your sentences, but they don't manage the full pipeline. They don't understand your schema drift, your monitoring, or your deployment steps. You still have to hold all that context yourself and stitch the pieces together. That's the real bottleneck, not writing the code but knowing what to write and where it fits.

I've been looking at platforms that try to automate more of the end-to-end data engineering workflow, Databricks, AI workflows, Genesis Data Agents, and Snowflake Cortex AI, but they all introduce their own failure modes. Debugging a multi-step agent decision is even harder than debugging a bad SQL query.

The tool doesn't fix the fundamental need to understand your data and your architecture. I'm now forcing myself to do one raw coding session per week without any AI help, just to keep the muscle memory alive. Feels like studying for a test I already passed, but the alternative is being useless when the API goes down.

When Databricks announced LTAP off the back of Lakebase, it got me thinking about HTAP again. Around 2015 it was sold as one engine for transactions AND analytics. No ETL, no stale data. Sounded amazing. Basically didn’t happen. HTAP revenue staled when the companies that split storage from compute and only did analytics or transactional are the giants now. The need itself never died however, so could LTAP be the answer?

I have been tasked with designing the switch from our existing legacy batch / orchestration tool to something native to AWS for our team and potentially the larger org.

We are the data platform team, and have about 25 pipelines, with data coming from shared views in Snowflake, business files, and APIs, all ingested via generic Python Lambdas driven through metadata / dynamic SQL. Our data store is entirely Snowflake, and we are expected to continue to grow in complexity and scale over time.

This is a full rebuild, and I am a mid-senior developer at a large company in the financial services industry. Initially, step functions was suggested as something to explore, which I have used for other smaller, more isolated pipelines successfully.

However, it is quickly becoming apparent to me that step functions is great for shorter, more linear business processes, but it is not an orchestrator, and trying to scale it with dependencies would be a nightmare.

With that in mind, I was planning to pitch Amazon Managed Workflows for Apache Airflow, as it seems this is the most stable enterprise orchestration tool available in AWS with support for triggers and emissions to Event Bridge, and integrations with other AWS Services.

My experience with AWS is not super extensive beyond the scope of Lambdas, S3, Step Functions, and CloudFormation/CloudWatch. Others on the team have more AWS experience and will be who I am pitching this to initially.

We are a relatively small team in the broader org, and the information is highly regulated so a mature product is a must. We don't really have the time to spend on managing a large infrastructure / troubleshooting orchestration, it really needs to be simple (as possible), stable, and scalable once configured with minimal overhead.

Does anyone have experience migrating from a legacy orchestration product to AWAA, and is there a better product out there for this use case, or something I should know going into this? Thanks in advance.

I have a data engineering hiring manager 45 min long round with Apple. I was hired as an SDE in my current role but ended up with data engineering work so I have never really given a data engineering interview.

I have been told I will be asked questions on Java, ETL pipelines, SQL and scalability. The range is so wide that I do not know what to focus on and possibly need some help narrowing down on areas I could focus on for the next 3 days.

If anyone has any advice or suggestions they would be appreciated!

Curious to know what causes the most issues in real projects data quality, scalability, monitoring, performance, or handling changing business requirements.

Kafka → ClickHouse is the default pattern for OTel pipelines, and for org-wide streaming with replay and many consumers it's a great fit. But for a lot of single-sink observability setups, it's a cluster you're babysitting for no reason.

This post compares where the Kafka layer does real work vs. where you can drop it. It also checks what processing the Collector can or can't do alone (stateful dedup, enrichment-conditional filtering, dynamic sampling, etc.)
https://www.glassflow.dev/blog/opentelemetry-to-clickhouse-do-you-need-kafka?utm_source=reddit&utm_medium=socialmedia&utm_campaign=reddit_organic

Curious what others run:

1. Kafka buffer,
2. straight from the collector, or
3. a lighter processor in between

Leave your comments below, I'd like to discuss the options and understand what folks are using these days!

I'm struggling to integrate ETL pipelines across multiple databases, APIs, and third-party tools. What's been the biggest integration challenge you've faced, and how did you overcome it?

Came across a problem recently that got me thinking about data quality checks.

One of our data sources looks healthy on the surface. requests are successful, ingestion is running fine, and there is nothing in the logs that indicates an issue. the problem only came up when I manually compared the output to the source and saw missing records and incomplete fields that should have been there.
The tricky part is that nothing is failing outright. the pipeline looks healthy unless you go and check the results wondering how others do this. what checks or monitoring do you use to make sure you're actually getting full data from a source and not just successful responses? Have you ever seen cases where a source returned partial data silently without obvious errors?

I'm currently working as a QA Engineer, but I'm more interested in ETL development and want to build my career in that direction. For my next job switch, I'd like to get an ETL Developer designation instead of QA. Has anyone here successfully transitioned from QA to ETL/Data Engineering? What skills or experience helped you make the switch?

I have a tech screen scheduled for the upcoming week and the recruiter mentioned the contents of the interview being Java /SQL questions, scalability and ETL pipeline related questions.

It’s a 45 min round and I’m unsure of how I can possibly prepare. They have sent coderpad link. What would be some focus areas? Anyone who has given one of these- is leetcode for java and sql good enough? Please help.