I posted a question about advice/suggestion, not even a single suggestion or comment on it !

Hey everyone 👋

I wanted to be transparent—I was recently laid off and am actively looking for an Analytics Engineer or Data Engineer role. I'm available to join immediately.

I have 5+ years of experience building scalable ELT pipelines and modern data platforms, with experience in:

Tech Stack

• Snowflake, dbt, SQL, Python, Airflow
• AWS (S3, Athena, Glue, EC2), CI/CD, Power BI
• Data Warehousing, Medallion Architecture (Bronze/Silver/Gold)
• Data Observability & Data Quality

Open to opportunities for PAN India and any work mode (On-site, remote, hybrid).

The current job market has been challenging, especially for Analytics Engineering roles (as there are few openings), so I'm doing everything I can to get back on my feet.

If your company is hiring, or if you know of any open Analytics Engineer or Data Engineer positions, I'd be incredibly grateful for a referral or even a lead. Every bit of help makes a difference.

Please feel free to DM me if you'd like my resume or know of any suitable opportunities.

Thank you for taking the time to read this. This community has always been incredibly supportive, and I truly appreciate any help you can offer. ❤️

Hi everyone,

I am actively looking for a Data Engineer role and wanted to put myself out there to this community.

A bit about me:

· 17 years of domain experience in the Travel, Tourism, and Airline industry (operations, revenue management, customer analytics, etc.).

· Since 2022, I have been consistently working on data-domain projects—building ETL pipelines, data cleaning, feature engineering, and analytics workflows.

· In 2024, I completed a formal Data Science course to deepen my understanding of the full data lifecycle (from ingestion to modeling).

· I actively work on Kaggle projects to sharpen my technical skills—especially around PySpark, Pandas, and SQL,ADF,Azure Cloud Platform,Databricks.

My Tech Stack:

· Languages: Python, SQL

· Big Data: PySpark, Pandas, NumPy.

· Orchestration: Apache Airflow (learning/building projects)

· Cloud: Azure

· Databases: PostgreSQL, MySQL

. Skills : Python, SQL,Pandas, ADF,Databricks,Azure Cloud Platform, Big data..

What makes me different:

I don't just know how to write code—I understand the business context behind the data.

I know how airline inventory works, what PNR data means, how pricing models fluctuate, and how customer behavior varies across segments. I've spent 17 years using this data; now I want to engineer it.

I am looking for Data Engineer roles—open to both mid-level and senior positions—ideally in Travel Tech (OTAs, GDS, Airlines, or Travel SaaS), but I'm open to other domains as well where my data skills add value.

What I'm looking for:

· A role where I can build scalable data pipelines.

· A team that values both technical skills AND domain knowledge.

· Remote / Hybrid / On-site (open to relocation for the right opportunity).

If you're hiring or know of any openings, I'd love to connect. Happy to share my resume, GitHub, or Kaggle profile.

Thanks for reading, and appreciate any leads or advice!

Looking for feedback from people who've worked with analytical databases (Impala, DuckDB, ClickHouse, Trino, etc.).

We have a legacy reporting application where users generate presentations. Opening a presentation triggers 50-100 SQL queries. The application is in maintenance mode with only one major paying customer, so our goal is to simplify the architecture, remove Cloudera licensing for Impala, and significantly reduce infrastructure costs.

Current Architecture

Presentation
      |
20 Dataset Worker Pods
      |
   Impala Cluster
(10 different EC2 r5.4xlarge with 128GB ram each)

The dataset worker pods simply receive tasks from the application and submit SQL to Impala.

The Impala cluster consists of 10 x r5.4xlarge EC2 instances (16 vCPUs, 128 GB RAM each) managed through Cloudera.

Workload

The workload isn't typical OLAP.

Each presentation fires 50-100 queries.

Roughly:

• ~80% are tiny queries
  • schema lookups
  • small dimension table filters
  • simple joins
• These usually return in 5-10 ms on Impala.
• Around 5-10% are heavier joins that take around 10 seconds.
• A presentation typically loads in 1-3 minutes depending upon type and filters

The total warehouse size is only around 300-350 GB.

Only 3-4 large tables account for roughly 200 GB. The remaining ~200 tables are tiny (KBs to MBs).

We want to Migrate away from Impala and not go for big commitment like dedicated EMR or something, we are ok with little delay but we dont want huge maintenance so we started with migrating to Athena from Impala.

Why Athena didn't work

Our first migration idea was Athena.

Large queries were acceptable, but the application performance became much worse because of the large number of tiny queries.

Queries that took 5-10 ms on Impala often became 200-800 ms on Athena.

Since every presentation executes 50-100 queries, that startup overhead adds up quickly.

Unfortunately, changing the application isn't really an option. The query generation is deeply embedded in legacy code, so batching or combining queries would require a major rewrite. Also many queries are sequential that adds up the time.

DuckDB Prototype

Instead of introducing another distributed SQL engine, I built a proof of concept using DuckDB.

Current architecture:

Presentation
       |
20 Dataset Worker Pods
       |
      HTTP
       |
---------------------------------
| DuckDB Pod 1                  |
| DuckDB Pod 2                  |
| DuckDB Pod 3                  |
| DuckDB Pod 4                  |
| DuckDB Pod 5                  |
---------------------------------

Each DuckDB pod:

• has its own DuckDB .db file
• has its own dedicated EBS volume
• serves requests over HTTP
• operates completely independently (no distributed execution)

The dataset worker pods simply load balance requests across the DuckDB pods.

The workload is almost entirely read-only.

For the few workflows that create temporary tables, I'm considering running a separate DuckDB write service with its own EBS volume since those temp tables only exist for the lifetime of a request.

Results

So far the prototype performs better than Athena for presentation loading, but still not as fast as Impala.

That isn't too surprising since the existing Impala deployment is heavily provisioned (10 × 128 GB RAM nodes) for only ~300-350 GB of data.

For this application, we're willing to accept somewhat slower presentation loads if it significantly reduces operational complexity, infrastructure cost, and removes the Cloudera dependency.

One thing I'm also thinking about

Right now every DuckDB pod has its own copy of the .db file on its own EBS volume.

Would you keep this design, or would you use something like a high-throughput EFS shared across all DuckDB pods?

I ruled out reading directly from S3 because this workload is dominated by lots of tiny, latency-sensitive queries rather than long analytical scans, and the additional object storage latency seemed noticeable during testing.

Questions

1. Has anyone replaced Impala with DuckDB for a similar workload?
2. Am I overlooking any major architectural issues with multiple independent DuckDB replicas?
3. Would you keep one .db file per pod on dedicated EBS, or use shared storage like EFS?
4. Would you choose a different engine entirely (ClickHouse, Trino, StarRocks, etc.) for this workload?
5. Any concurrency or operational issues you've run into serving DuckDB over HTTP in production?

I'm less interested in benchmark numbers and more interested in hearing from people who've operated similar systems in production

Hi everyone,

I joined TCS about 2 months ago as a fresher with a CTC of 7 LPA. I was assigned to the Data Engineering domain, and the current learning path includes:

• Data Warehouse
• PySpark
• Apache Kafka
• Apache Airflow
• Power BI
• Databricks

My goal is to switch to a better company after around 1 year (or as early as I'm reasonably ready). I want to use this one year effectively and become really strong in Data Engineering.

I have a few questions for experienced data engineers and people who have made similar switches:

1. Is this tech stack enough to land a good Data Engineer role if I master it?
2. What other technologies or concepts should I learn alongside these? (SQL, Python, Cloud, Spark internals, Data Modeling, etc.)
3. What kind of projects should I build to make my resume stand out?
4. Is one year of experience generally enough to switch, assuming I have solid skills?
5. What salary range is realistic after switching from 7 LPA if I become proficient in this domain?

I'm willing to put in consistent effort over the next year, so I'd really appreciate a roadmap or any advice on what to prioritize and what to avoid.

Thanks in advance!

Hey folks, I’m looking for honest insights into the work culture at Paytm. I’ve got an offer (pending final approval) for a Tech Lead role, though they might internally upgrade it to Senior Tech Lead. The offer stands at 46 LPA fixed, plus a 2L signing bonus and about 7L in equity (vested over time). I’d be working in the insurance domain.
I have 6.10 years of experience as a software dev/data engineer?

Now, I’d love to hear from people who’ve worked there or know someone who has. How is the culture—positive, toxic, or somewhere in between? What’s the reality of work-life balance there? Is job security stable, or do people feel like they’re on shaky ground? Any advice or experiences on how it really feels to be on the inside—whether positive or negative—would be super helpful! Thanks in advance!

Hi everyone,

I need some career advice.

I've been working at the same company for the last 7 years (currently earning 18 LPA). The biggest challenge is that the company uses its own in-house technology, which isn't really useful outside the organization. The only transferable skill I've gained is SQL, and I'd say I'm quite comfortable with it.

About 3 years ago, I decided to switch to a data-related career. After doing some research, I chose Data Engineering because it seemed like a good fit. I enrolled in a Data Engineering course from Scholarnest Academy, completed it on time, and prepared for interviews.

Unfortunately, a serious medical emergency in my family happened around the same time, so I had to put my plans on hold and never got the chance to attend interviews.

Things are stable now, and I'm preparing for a job switch again. Before I invest the next few months into interview preparation, I wanted to ask people who are already in the industry:

• Is Data Engineering still a good career choice in 2026?
• Is the demand still strong for someone trying to break into the field?
• Or would you recommend considering another role instead?

For context, I have strong SQL skills and I'm currently brushing up on Python, Spark, Databricks, and cloud concepts.

I'd really appreciate any honest advice, especially from people working as Data Engineers or those who recently switched into the field.

Thanks in advance!

Hey everyone,

I have an upcoming 60-minute technical interview for a Senior Data Engineer (8yoe) role at Roku. It’s a live, pair-programming session on HackerRank.

I’m looking for any advice from people who have been through this recently:

Is it mostly heavy SQL (window functions/optimization) or PySpark/Python data manipulation?

How does the difficulty level of the HackerRank CodePair round compare to standard LeetCode Medium/Hard problems?

Any tips on what they look for in a "pair programming" setting (e.g., communication style, handling system design-lite questions during the code)?

Would appreciate any insights you’ve encountered. Thanks in advance!

I’m looking for advice from people who have successfully grown their compensation over the years.

I have around 5 years of experience and have switched jobs 3–4 times. In every switch, I got around a 30–40% hike. But despite multiple switches, my current CTC is still around ₹17 LPA.
When I see people with similar experience earning ₹40–50+ LPA, I genuinely wonder what they did differently.

Some questions I have:
Did you target specific companies (product companies, startups, MNCs)?
Did you make bigger jumps by negotiating multiple offers?
How did you prepare for interviews? What was your preparation strategy?
Did networking or referrals play a significant role?
Any mistakes you made that you would advise others to avoid?

I wanted to understand what practical strategies helped people break into the ₹40–50 LPA range.
Would really appreciate hearing your journey and any actionable advice.

29M single. YOE - 7, current salary - 2.6L/month. Got an offer in Vilnius, Lithuania (4300 EUR gross, ≈2.6K Eur net per month).

As per my research, the expenses would look something like this in Vilnius:

400-600 rent

200 Utilities

200 groceries

Right now, I work remotely. I am renting a place in Kolkata (I like the privacy of living alone). I spend around 25-30K per month on rent, utilities and food (excluding dining out). Do my own cooking and cleaning cuz OCD lmao💀. So I save around 2.2L easily. But it's not permanent remote so there's a possibility of expenses shooting up in future if I have to relocate to my base (Hyderabad).

I like the prospect of EU Blue card as it allows me to change employer after a year. But I am not sure if the interim pay cut would be worth it.

Pros of India: More savings, closer to family

Cons of India: The usual (pollution, infra etc.)

Pros of Vilnius: Quality of life, future growth prospect, better passport (eventually), cheap travel in Europe (can finally do my backpacking trip across Europe without exhausting leaves)

Cons of Vilnius: Less savings, out of comfort zone, unfamiliar territory

Please help me decide. TIA!

Hi everyone, looking for some genuine career advice here.

I'm currently working at a startup as a Data Engineer for the past one year. The company does client data analysis through ML models. My role mainly involves gathering data from client sources (APIs, Google Drive, etc.) and transforming it into a model-acceptable format so the ML team can extract insights. I primarily work with PostgreSQL and Python for this.

Apart from that, I also do some backend work on the company's product (mainly their website), but outside of these two things, there isn't much else on my plate.

Before this, I worked at TCS in a Digital role for about 5 months, but I was on the bench the entire time. I only got exposure to AWS, Docker, and Java/JDBC/Servlets fundamentals — never actually got staffed on a project. Eventually I left and joined my current company.

Now I'm planning to switch to a proper Data Engineer role because I feel stagnated in terms of growth. When I look at other data engineers around me, they're working with PySpark, Databricks, and large-scale production pipelines — stuff I have zero hands-on exposure to. My work is fairly basic in comparison, and my salary reflects that too (which, honestly, feels fair given the scope of my current work).

So I wanted to ask this community:

• What should I focus on learning to actually become job-ready for a real Data Engineer role (PySpark, Databricks, pipeline orchestration, etc.)?
• Should I try building projects on my own to simulate that production-pipeline experience, or is there a better way to get exposure?
• Any advice on how to position my current experience (which is more data-wrangling than true DE work) when applying for DE roles elsewhere?

Would really appreciate insights from people who've been in a similar spot or are currently working as DEs in bigger setups. Thanks in advance!

[Used AI to phrase my query better, as im bad at it 😭 ]

Hi Everyone, I have to post this because HCL played a very unfair game with me. Hcl organise hackathon to hire people, I participated and got selected. HR contacted me to submit documents, I submitted all documents. After that HR ghosted me literally, not receiving phone nothing.

Is there anyone faced the same issue ?

I’ve been working at my current org since Oct 2025, so it’ll be a year in a couple of months.

The problem is that I have no practical exp. I hve been on the bench the entire time, well except first 2 months of training. Not a single project. Not even a shadow resource opportunity.

The only hands on experience I’ve got is that a senior in my office helped me build an end to end Databricks project on the side, but that too with the help of co-pilot. Apart from that, I hve been learning on my own. Currently im studying Azure, planning to get DP-900 in the next few weeks. I’d say im very good with SQL, and moderate with Python and PySpark.

I hve honestly lost hope that RMG is ever going to get me a project. I personally messaged dozens of managers over the past few months. Either they don’t reply, or they say they don’t have any openings.

Im thinking of switching once I complete a year here.

But how do I even sell myself in interviews when I dont have any actual project experience? Im not asking how to lie, I mean, if they ask me what I worked on in my current company, what am I even supposed to say? I genuinely dont know shit about real project workflows because I’ve never been on one.

And is switching after just one year even a good idea in the current market? Or should I somehow survive longer and hope things improve?

Because at this point, Im pretty sure if I stay much longer, my company will eventually start pushing a rod up my ass. Either by putting me on PIP or asking me to resign.

Has anyone here been in a similar situation? What did you do, and were you able to switch successfully?

Hello everyone,

Used Chatgpt to correct it and write it in good way

I recently got a call from a KPMG recruiter saying my resume had been shortlisted for a technical interview. The role is Associate Consultant – Infrastructure (Data Analytics – Microsoft Fabric). She also suggested I revise Microsoft Fabric, Lakehouse, Power Apps, etc.

A little about my background:

• 1 year of experience as a Data Engineer
• SQL, PySpark, Azure Databricks, Apache Airflow, ADLS Gen2
• Worked on ETL pipelines, Medallion Architecture, and healthcare data

I reviewed the JD it was ppt, and it seems to be a mix of both data engineering and analytics. It includes data ingestion, data transformation, Lakehouse, SQL, pipeline architecture, and Fabric, but it also mentions Power BI, DAX, Power Apps, and dashboarding.

My long-term goal is to build my career as a Data Engineer, not move into a BI/Data Analyst role.

For those who have worked at KPMG or other Big 4 firms:

1. In practice, how much of this role is data engineering (pipelines, Lakehouse, ETL) versus dashboards and reporting
2. One of my long-term career concerns is platform portability. If I spend the next 2–3 years primarily working with Microsoft Fabric (Lakehouse, Data Factory, Fabric notebooks, etc.), how is that experience generally viewed when applying to Databricks-focused or Spark-heavy companies? Have any of you made a similar transition?
3. Would you consider this a good opportunity for someone who wants to stay on the Data Engineering path?

I'd especially appreciate replies from people who have worked in KPMG, other Big 4 firms, or Microsoft Fabric-based Data Engineering teams.

Thanks in advance

So I got reached out by recruiter at LSEG for senior data engineer role in Banglore. After that I got a test and had one code pair interview. It was good I was given a piece of code and was asked to debug and everything. Now I got another code pair interview for 90 mins. In the email they mentioned In this stage you’ll be expected to demonstrate your ability to write units tests and showcase your skills in TDD (Test Driven Development). We also want to see that you can write clear, easily readable, easily maintainable code with clear notes.  And I am confused on what to prepare. I mean will they ask to code test cases? If anyone has been through this kinda interviews please explain what to prepare for.

Hi everyone!

My organization is hiring Data Engineers (4+ years) and Senior/Lead Data Engineers (8+ years) for Hyderabad. I'm happy to refer suitable candidates.

Required Skills:

• Microsoft Fabric (Data Factory, Lakehouse, Data Warehouse)
• Databricks (PySpark, Spark, Delta Lake, MLflow)
• Snowflake
• Power BI (Data Modeling, DAX, Report Development)
• Strong SQL and Python (Scala is a plus)
• ETL/ELT pipeline development
• Data warehousing and data modeling
• Azure, AWS, or GCP cloud data services
• Data governance, security, and CI/CD best practices

Experience:

• 4+ years: Strong hands-on experience in modern data engineering and cloud data platforms.
• 8+ years: Prior experience leading projects, designing data platforms, or working in a Lead/Architect role is preferred.

Location: Hyderabad

If you're interested or know someone who fits the profile, feel free to DM me your resume or drop it in the comments. I'll be happy to refer qualified candidates.

Thanks!