Looking for feedback from people who've worked with analytical databases (Impala, DuckDB, ClickHouse, Trino, etc.).

We have a legacy reporting application where users generate presentations. Opening a presentation triggers 50-100 SQL queries. The application is in maintenance mode with only one major paying customer, so our goal is to simplify the architecture, remove Cloudera licensing for Impala, and significantly reduce infrastructure costs.

Current Architecture

Presentation
      |
20 Dataset Worker Pods
      |
   Impala Cluster
(10 different EC2 r5.4xlarge with 128GB ram each)

The dataset worker pods simply receive tasks from the application and submit SQL to Impala.

The Impala cluster consists of 10 x r5.4xlarge EC2 instances (16 vCPUs, 128 GB RAM each) managed through Cloudera.

Workload

The workload isn't typical OLAP.

Each presentation fires 50-100 queries.

Roughly:

• ~80% are tiny queries
  • schema lookups
  • small dimension table filters
  • simple joins
• These usually return in 5-10 ms on Impala.
• Around 5-10% are heavier joins that take around 10 seconds.
• A presentation typically loads in 1-3 minutes depending upon type and filters

The total warehouse size is only around 300-350 GB.

Only 3-4 large tables account for roughly 200 GB. The remaining ~200 tables are tiny (KBs to MBs).

We want to Migrate away from Impala and not go for big commitment like dedicated EMR or something, we are ok with little delay but we dont want huge maintenance so we started with migrating to Athena from Impala.

Why Athena didn't work

Our first migration idea was Athena.

Large queries were acceptable, but the application performance became much worse because of the large number of tiny queries.

Queries that took 5-10 ms on Impala often became 200-800 ms on Athena.

Since every presentation executes 50-100 queries, that startup overhead adds up quickly.

Unfortunately, changing the application isn't really an option. The query generation is deeply embedded in legacy code, so batching or combining queries would require a major rewrite. Also many queries are sequential that adds up the time.

DuckDB Prototype

Instead of introducing another distributed SQL engine, I built a proof of concept using DuckDB.

Current architecture:

Presentation
       |
20 Dataset Worker Pods
       |
      HTTP
       |
---------------------------------
| DuckDB Pod 1                  |
| DuckDB Pod 2                  |
| DuckDB Pod 3                  |
| DuckDB Pod 4                  |
| DuckDB Pod 5                  |
---------------------------------

Each DuckDB pod:

• has its own DuckDB .db file
• has its own dedicated EBS volume
• serves requests over HTTP
• operates completely independently (no distributed execution)

The dataset worker pods simply load balance requests across the DuckDB pods.

The workload is almost entirely read-only.

For the few workflows that create temporary tables, I'm considering running a separate DuckDB write service with its own EBS volume since those temp tables only exist for the lifetime of a request.

Results

So far the prototype performs better than Athena for presentation loading, but still not as fast as Impala.

That isn't too surprising since the existing Impala deployment is heavily provisioned (10 × 128 GB RAM nodes) for only ~300-350 GB of data.

For this application, we're willing to accept somewhat slower presentation loads if it significantly reduces operational complexity, infrastructure cost, and removes the Cloudera dependency.

One thing I'm also thinking about

Right now every DuckDB pod has its own copy of the .db file on its own EBS volume.

Would you keep this design, or would you use something like a high-throughput EFS shared across all DuckDB pods?

I ruled out reading directly from S3 because this workload is dominated by lots of tiny, latency-sensitive queries rather than long analytical scans, and the additional object storage latency seemed noticeable during testing.

Questions

1. Has anyone replaced Impala with DuckDB for a similar workload?
2. Am I overlooking any major architectural issues with multiple independent DuckDB replicas?
3. Would you keep one .db file per pod on dedicated EBS, or use shared storage like EFS?
4. Would you choose a different engine entirely (ClickHouse, Trino, StarRocks, etc.) for this workload?
5. Any concurrency or operational issues you've run into serving DuckDB over HTTP in production?

I'm less interested in benchmark numbers and more interested in hearing from people who've operated similar systems in production

Hi everyone,

Looking for some career advice from experienced data engineers.

My background:

• Tier-3 college graduate (2021).
• Interned for ~2.5 years during college at around ₹28k/month.
• Joined TCS after graduation and worked there for ~9 months.
• Moved to Amazon in a Big Data Technical Support role and spent ~3 years there.
• Took a pay cut to switch into a Data Engineer role at Walmart because I wanted to move closer to core engineering work.
• Recently joined a mid-sized company as a Senior Data Engineer.

Current compensation is around ₹40 LPA.

Skills:

• AWS (Glue, EMR, S3, Athena, Lambda, etc.)
• GCP
• PySpark / Spark
• Scala
• SQL
• ETL pipelines
• Data Warehousing
• Production support and platform operations
• Large-scale batch and streaming processing

A few questions:

1. For a 2021 Tier-3 graduate, is ~₹40 LPA a good position to be in?
2. Was taking a pay cut to move from Big Data Support to Data Engineering the right long-term decision?
3. What skills would you prioritize next to target ₹60-80 LPA and eventually ₹1 Cr+ compensation?
4. Are Staff/Lead Data Engineer roles the most realistic route, or should I be looking toward Data Platform, AI/Data Infrastructure, or Engineering Management?
5. If you were in my shoes, what would your next 3-year plan look like?

Would appreciate honest feedback, especially from people who have crossed the ₹75 LPA or ₹1 Cr mark in data engineering.

Last year, in April 2025, I resigned from a FAANG company where I was working in a technical support role. It wasn’t pure development, but the work was strong in cloud and big data, focused more on troubleshooting, customer-facing problem solving, and technical support rather than general support. The pay was very good for a support role, around 32 LPA after my promotion, and it was fully remote with good work-life balance, though timings were strict.

Interestingly, on the same day I got promoted, I submitted my resignation. My manager’s manager even asked me to reconsider and decline it. The reason was simple—I wanted to move into a better role with stronger long-term growth and more development-oriented work.

Since I wasn’t getting many offers at that time, I accepted a switch to well known product company at 25 LPA, which meant a direct pay cut of 7 LPA. On top of that, I had to relocate to Bangalore, which added more financial loss through relocation expenses and overall adjustment costs. I knowingly took that hit because I believed switching out of support into a stronger role would help me more in the long run.

Initially, joining this company felt exciting because I was moving to a big company and expected stronger learning opportunities. But after completing almost a year here, I realized there was very little meaningful work. Honestly, there was almost no real technical challenge, and I didn’t feel like I was learning or growing in a significant way.

Now I have an opportunity with Nielsen Media, where they are offering around a 50% hike, which is financially a huge jump. The present company feels much more stable, while Nielsen definitely carries more risk, especially considering the current market and possible layoff concerns. But I also feel Nielsen may offer much bigger long-term rewards because of the new project, stronger ownership, and the kind of exposure I am looking for.

I am still young, and I feel this is the stage where I can afford to take calculated risks. If I have to take a chance, this is probably the right time to do it rather than later when responsibilities become bigger.

The difficult part is my current manager. He is genuinely a very good person and has been extremely supportive from day one. He has always been flexible, trusted me, involved me in leadership discussions, and even though he is a Director and several levels above me, he always makes me feel comfortable sharing my views and gives me confidence.

We had a very honest conversation about my plans. I openly told him about the Nielsen offer and even shared the salary details because I felt there was no point in hiding it. He told me that he had made similar mistakes earlier in his own career and advised me not to leave too quickly.

He believes I should stay, and he promised that he would move me into the kind of work I actually want—agent development, big data, AI, and stronger technical ownership. He even added me to a new project already, and it genuinely looks promising.

He said he cannot match the offer immediately, but he can help fast-track my growth, promotions, and position me for much bigger rewards in the next 1–2 years. He also told me to work for 20–30 days and evaluate things properly. He is not forcing me either way—if I still want to leave, he will accept my resignation and won’t extend the notice period. If I want to stay, he will back me fully.

My current thought process is this: since I have already submitted my resignation, I should use these 30 days properly. Instead of treating it like a relaxed notice period, I want to take it as a personal challenge. My manager has already given me 2–3 technical areas where I can genuinely learn, and I want to work hard on those, deliver fast, and set a high bar for the team I leave behind.

I may not get to “enjoy” the notice period in the usual sense, but if I can leave after learning something valuable, delivering strong work, and leaving on a good note, that feels more meaningful to me. I also feel my manager will remember that I stayed professional and committed even during notice period.

So my current plan is to learn as much as possible, contribute seriously, leave respectfully, and still move forward with the better opportunity at Nielsen.

The only thing that still bothers me is the guilt—because when your manager is genuinely a good person and supports you like a mentor, leaving feels much harder than leaving just a company.

That’s where I’m stuck right now.

So wise folks of Reddit—what would you do in my place? Stay for stability, trust the manager, and play the long game… or take the risky jump for bigger growth and rewards?