r/dataengineer

Hey everyone,

I’m currently working as a Data Analyst and actively transitioning into Data Engineering . I’ve started preparing and appearing for Data Engineering interviews.

I’m looking to connect with people who are genuinely preparing for DE interviews or are currently making a similar transition from Data Analytics to Data Engineering.

The idea is to share interview experiences, preparation strategies, resources, mock interviews, and keep each other accountable throughout the journey.

Please reach out only if you’re genuinely preparing or interviewing for Data Engineering roles.

I’d prefer a small group of serious and committed people rather than a large inactive one.

Feel free to DM

I want to find datasets for a data engineering project where i work with pyspark and sql in databricks. I want a dataset that challenges my data modelling skills and my pipeline creation skills. I tried kaggle, but i only keep getting a single csv file as a dataset. is there a dataset that has multiple csv files as data sources or something? i want to be able to perform all the data architecture creation by myself... Recommend any datasets that you know as well!

I’m trying to find genuine freelancing opportunities in data engineering / BI. Have tried a few platforms but haven’t had much luck, so wanted to ask — are there any websites, subreddits, or Discord servers where people actually get projects?

About me:

• 5+ years as a Data Engineer & BI Consultant (remote, India)
• MBA in Business Economics (Analytics & Finance)
• Skills: SQL, PySpark, Python, Power BI, Tableau, Grafana
• Worked on Databricks pipelines, self‑service analytics frameworks, and telemetry data solutions

I’m in need of extra income and open to contributing under a team or experienced freelancer. Any pointers would mean a lot.

How do you structure Unity Catalog when you don't have schema-creation rights?

Working on a medallion architecture (Bronze → Silver → Gold) setup in Databricks, but ran into a permissions wall: I have USE CATALOG and read/write on one existing schema, but not CREATE SCHEMA on the catalog itself. That means every new pipeline I build — and I have several in flight — has to land in the same single flat schema.

To make it worse, I'm also dealing with two separate source systems feeding into the same catalog (think: legacy system + a newly merged system, mid-migration), so right now everything — raw, cleaned, source A, source B, even some test/prod data — is just sitting in one big pile of 60+ tables, distinguished only by table name prefixes.

A few questions for anyone who's been here:

1. Is CREATE SCHEMA typically gated centrally at your org, or is it usually delegated to teams building pipelines?
2. If schema creation is centrally controlled, what's the actual workflow — do you submit a request and someone else creates the schema, or is there a self-service path?
3. Has anyone found naming conventions + UC tags to be a genuinely workable substitute for real schema separation, or does it always end up being technical debt that has to get unwound later?

Trying to figure out whether to push harder for the permission now, or just build disciplined naming conventions and migrate later once governance catches up.
How do you'll deal with this blocker? Would be grateful for any suggestions from your experience for a junior data engineer like me.

Hi, I am preparing before switching company, for 6 YOE in DE.

Currently I am revising and practicing Python, have plans to practice Pyspark, SQL, AWS, project information.

I am noticing that I am going very slow with Python, I am able to spend 1-2 hours on Python (learning theory + practice coding)

how to fasten the preparation and what is effective approach to prepare for interviews?

As of now I am not confident in my skills, how do others just go for interviews? Are you confident you will be able to solve coding challenges and scenario based questions?

I would love to know your experiences.

TIA