r/dataengineeringjobs

Role: Data Engineer with experience in SQL, building and debugging ETL pipelines and relevant AWS services

Pay : 25k (slightly negotiable) per month.

Work: 2.5 hrs for 5 days a week. Usually in night time IST (9-1AM window) or can be a bit flexible.

Payment Method: Paid bi-weekly via UPI.

Skills : SQL, ETL pipelines, AWS Lamba, Glue, Python, Debugging.

If your profile suits for this role kindly DM me or drop a comment. Please connect if you have relevant skills. There will be a telephonic screening (15-20mins) and then we can get started.

I’ve spent the last few years working in data and mentoring people who are trying to get their first Data Analyst / Data Engineer job.

And honestly, I think a lot of people are preparing the wrong way.

I regularly see people learning:

SQL → Python → PySpark → Kafka → Airflow → AWS → Databricks → Snowflake → Kubernetes...

…and after 6–12 months, they still don't feel ready for an interview.

The problem usually isn't lack of effort.

It's that they're learning tools instead of learning how data systems actually work.

For example, someone may know how to write a Spark transformation but struggle when I ask:

Why did this Spark job suddenly become 5x slower?

What actually happens when you join two 500 GB datasets?

When should you broadcast a table?

What happens when one partition becomes much larger than the others?

Why did an Airflow pipeline fail even though the code itself is correct?

How would you design a pipeline that can safely process the same data twice?

What would you monitor in production?

What happens when a downstream system is unavailable?

These are the questions that separate “I completed a Data Engineering course” from “I can actually work as a Data Engineer.”

If you're learning Data Engineering right now, my advice would be:

Don't try to memorize 20 tools.

Understand these fundamentals deeply:

Data → Storage → Compute → Transformation → Orchestration → Reliability → Observability

Once you understand those, learning another tool becomes much easier.

I'm going to start spending some time here sharing the practical things I've learned from working with data systems and mentoring people — including things I wish someone had explained to me when I started.

No course pitch. No sales link.

If you're currently preparing for Data Engineering, tell me where you're stuck.

I'll try to help where I can.

I've given 20+ data engineering and analytics interviews myself, and for the last 4 years I've mentored people through theirs. Every rejection email reads the same. "We've decided to move forward with other candidates." No feedback.

Most companies don't allow interviewers to share it.

But when someone tells me how their round actually went, I can usually tell them why. The same five reasons keep repeating.

You answered "what" when they asked "why."

The interviewer knows what a broadcast join is. They asked to hear when you'd use one and when you wouldn't. Definitions get a polite nod and a rejection. Trade-offs get offers.

Your project sounded like a tutorial.

You explained what you built, smoothly, in order. Then came one question off the script. Why Parquet here? What breaks if the input doubles? Silence.

The interviewer writes "followed a video" and moves on. One project you can defend beats five you can only describe.

You bluffed instead of reasoning.

"I haven't worked with that, but here's how I'd think about it" sounds senior. A confident wrong answer sounds risky, and risky gets rejected. Nobody is counting how many questions you got right. They're deciding whether to trust you with production.

There were no numbers in your answers.

"Large dataset." "It ran faster." The person who says "around 40 GB daily, and the job dropped from 50 minutes to 12 after I fixed a skewed join" describes the same work but sounds like the engineer who did it.

You talked about tools. They were listening for problems.

Ten tools on the resume, zero stories about late data, duplicates, a backfill that went wrong. Tools are what you used. Problems are why someone pays you.

If you got rejected recently and never found out why, describe the round in the comments. What they asked, where it got shaky. I'll tell you the most likely reason, the one the email left out.

I want to know what kind of project are actually recruiters looking for in the Resume, if you are someone who has cracked the Data engineer role in last 6-10 months then please guide us all what kind of projects we can add?

I’m a fresher looking for my first Data Engineering job. I know Python and SQL.

What 2 projects should I build, and what tech stack should I follow?

Looking for projects that are:

Real-world/production-like

Good for GitHub

Strong portfolio projects

Good for discussing technical skills

What tech stack and projects would you recommend for a fresher?