r/dataengineering

I have been learning data analytics/engineering at a slow pace for 3 years now. The slow pace part is because I was committed to another job - the pay has been good and this made not want to switch to big data, but then I knew eventually I'll switch. And now I strongly feel the time has come for me to switch.

The conviction about switching is simply because I have become competent and this I can tell from the projects I have been doing.

My SQL skills have reached a level where I can write joins, subqueries, window functions, and CTEs easily. This means I can clean messy data, analyze it, and even create dashboards with Looker.

I don't want to brag about my dbt proficiency coz I still think I'm not yet there. Yes I can create a warehouse on Snowflake, create a database, a dbt role and a dbt user, and run the process through my VSCode from staging to having a clean model. But then I like to talk about being proficient if I can do something off head without skipping a step. I still have a few steps left.

Same applies with Python. I can write Python code but I haven't used it extensively for data analysis and manipulation.

I'm asking for engagements so I can gauge my progress. One thing that I'm sure about is I'm on the data engineering path.

Hi

I started a data engineering training program about a year ago, and my background is in marketing and campaign management. However, my previous work already involved some technical aspects: at my last employer, I was responsible for price engineering in the online shop and the technical implementation of complex campaigns for around 60,000 products. In this role, I worked closely with development agencies, IT, and data analytics. Working with legacy systems that had evolved organically over time and lacked a central data infrastructure is what sparked my interest in data engineering.

As part of my training, I've completed several practical projects so far: a habit tracking application in Python with a MySQL database and analytics functions, a data mart with 23 tables in PostgreSQL, and currently, an IoT data pipeline where I retrieve data via an API and process it further using MongoDB and Kafka.

With a profile like this, how would you approach getting started in data engineering in Vienna? Should I simply continue working on projects that are as practical as possible and apply for suitable positions at the same time? Or would you also specifically look for related roles such as Data Warehouse, Data Integration, Data Quality, SQL/Database, or similar technical positions?

The specific job title "Data Engineer" isn't important to me. What's crucial is that I can get a career in the data/IT field, gain practical experience, and develop my technical skills.

Small background: Been doing data analytics and BI development with heavy sql and ETL work within fabric for a little over 4 years.

My company recently told me they will give me the data engineering title but it pays the same as their BI salaries.

Other companies i have talked with have been skittish about going into the 110k area with this experience.

I make 93k now… is this normal for 4 yoe doing DE style work but without the official title? Is asking for a 20k increase anywhere considerably greedy?

Area I talked to is slightly MCOL and I’m making 93k in what is considered LCOL.

I am confused as to what the standard salary is nowadays because I always thought DE was a career that paid more.

Everyone wants to build sophisticated AI systems (because of FOMO), but I feel like there are far more basic "data discoverability" issues that need to be solved first before we even start talking about using complex agentic automation. Am I wrong to think that, instead of building AIs that work on bad, fragmented data, we should focus on making the data more usable first -- by both people and agents? It's like we're obsessed with improving engine efficiency while having it consume unrefined crude oil.

So, the question is: do you feel the same way that we should kind of take a step back with regard to what we focus on? Do you likewise consider the issue to be as significant? How is your org solving this?

I plan on building a pipeline that ingests 2d grid data (lat,long,value) thats updated frequently but I have zero clue on how exactly to store it, as I would have 4 columns essentially (timestamp, lat, long, value). So unique key per row would be combination of time/lat/long.

Would columnar format still be best? It wont be large, coordinates are integer values, data is updated every ~5 min and its provided in a json format.

My current pipeline ingests normal timeseries data, saves raw in S3 and transformed in Postgres, but I’m not sure if it’s best to have this data treated the same. Was thinking maybe a different file format and keep solely in S3? The values doesn’t need any processing

Ultimate goal is to in the frontend build a live, last n-days animation of the globe to see how the values change

I recently joined company as their only dedicated data person. they have other SWEs and a couple other people that pull data and use it, and they’ve only recently built their data lakehouse last year using databricks, but I’d be the only dedicated data person. so any data problem I’d probably deal with eventually. right now I’m dealing with the crm and cleaning it, but eventually I suspect that my duties will increase and I’ll be doing some form of data engineering, so I was wondering if I can essentially learn data engineering myself? or at least well enough?

I see AWS everywhere, and after that BigQuery and Airflow. It’s very rare that I see job postings for Azure/Microsoft nowadays.

I’m a bit of shocked, because I started with the Azure/MS stack many years ago and have used it ever since in my consulting career. I have familiarity with Snowflake, dbt, Databricks, but I hadn’t thought about putting them on my CV, except for Snowflake. And putting to much tools or different stack is sees as unserious.

So where are all those Microsoft/azure data jobs ?

Is Data Platform Engineer a hybrid role someone who does both cloud engineer and data engineer at same time?

Is someone like that on high demand by companies or clients? Or do companies just hire separately each one?

I work as a data engineer at a bank. My team is supposed to be the one centralizing data collection and management for the whole org.

I've realized that there's this tension between centralizing and decentralizing data work -- whether we, as the org-level data team, should implement a flow or expect the analysts to implement and own the flow, while we just provide the underlying infra. Over time, I started to strongly lean towards more decentralization. Not because I'm lazy, but because centralization should be about being focused on the absolute smallest denominator, which is smaller than most orgs assume -- especially now when you have no clue what an optimal stack is gonna look like in 6 months. My impression was always that we should be doing 50% less, but doing it 100% better.

(I have similar opinions when it comes to business applications more generally -- the centralized team should provide the infra to deploy, but teams that need an app should build and maintain it)

Are you seeing the same trend of decentralization?

I am learning how to build a pipeline using dbt and am confused with snapshot. I have a question about snapshot.

So, I have read somewhere that snapshot should be made as close to raw data as possible.

Let's say I make snapshot of raw data. I need a stage layer to clean the data before loading it to the dimension layer. In this case, do I make the stage model as incremental and load only the changes at the dimension layer? Would I be tracking history in two layers, snapshot and dimension if this is the case?

Also, which layer is responsible for generating keys? I assume it would be the snapshot layer.

And are only dimension table candidates for snapshot? Do I load fact as incremental straight to the fact table?

My company has two distinct teams IT and BI. BI tasks mainly involve reporting, PowerBI and Excel. IT mainly looks after maintaining ERP, CMS systems, databases and other it related system initiatives. If there is a guy from both IT and BI team who has somewhat Data engineering knowledge and if both want to be involved in building data warehouse for the company, who among them is more entitled to it? It is more technically sound and bi guy is bit technically behind specially in terms of networking and stuff but knows end user requirement really well. Specially, I want to know if the BI guy should be allowed to build the warehouse if he has knowledge in it? what should he do to get the buy in from IT to start this initiative?

I'm new to big company work and having to balance working on multiple things at once. In my background I didn't have anyone to ask when I got stuck, I worked through it, sometimes days. Now working on a team and a large company I am finding myself uncovering issues and problems at the architecture/repo level that others seem to just work around.

It's appreciated by some but NOT by the people who have decision making power because they only care about their project and not making things work better or easier for devs. Even though it could really speed up development or reduce errors in the future.

For example, we recently switched from Azure dev ops repos to GitHub. Most people resolved their merge conflicts using the web interface, some of you might already see the problem. GitHubs online merge conflict resolution works differently from ADO, it first pulls the changes/commits into your source (we use features) branch and resolves the conflicts there so it then has a clean PR when actually merging it. We want our release branch to only have our changes, otherwise we risk pushing someone else's not ready work into QA or PROD.

I said others are working around it but there's also noise about the issue. Not enough for the architects or whoever to get involved though. I spent way too much time digging into this issue, eventually uncovered the issue that was blocking me was actually created by someone else, the guy whos training me.

I asked another dev how he's working around it and he created his feature branch from DEV, he's going to have the exact same, or worse problems when he gets to QA. I still don't completely trust that I see the picture clearly. Maybe this allows him to work quickly in dev, then only deal with a messy merge conflict when he's ready for QA. We're talking 50+ merge conflicts over 30 files. I feel like this puts the prod deployment at risk because you could mess up the merge conflict. The architects should catch it but doesn't feel right.

Hi, I am a lead on a data engineering team and looking for feedbacks on Databricks and Snowplake from an operational perspective.

My team will chose a data platform and it’s going to be one of those two. I have experience with Databricks (and a bit with IaC and config of data platforms for Databricks, Domino, Rstudio server). My work with Snowflake has been limited to our test with their environment for a proof of concept.

Feature wise, they are on par. To me they offer similar experience that won’t be felt by most of our users because of our use cases (no real ML and complex LLM work being done by the teams).

Our current situation is:
- medium sized business in financial marketa, so not DE focused
- one DevOps engineer that can support us, only one platform engineer and potentially the IT department supporting our tickets
- Azure focused shop

I’m feel a bit in a pickle with that one, because Databricks is integrated with Azure and can be deployed in a couple of clicks. My experience with it was within a huge corporation with a ton of DevOps guys, platform engineers and good IT support, federated costs and so on. Databricks was amazing and working extremely well.

Now, since at our company it’s a bit more bare, we have way less people and spotty support. I can’t dedicate a lot of resources to maintain the platform when I need to follow my DE roadmap. During our PoC with Snowflake, the setup was incredibly easy, the platform was working right out of the box, their team was also very supportive and answering fast for questions we had. Databricks was slightly more difficult to set up and with comparatively less support from their team.

My intuition tells me that Databricks (although cheaper on paper for costs and invoicing within Azure) might have more hidden issues if we adopt it with a small support team, whereas Snowflake being something handled by the vendor will let me focus on getting the job done and not tweak our setup. For the price difference Snowflake’s managed platform is acceptable.

Perhaps my intuition is wrong, but I’d like to hear a bit more from the community. My network is like 99% of people working with Databricks, so you might have more experience with both.

TL;DR: small team of DE guys with little support, needs to chose between Snowflake and Databricks on Azure. Need to hear about operational experience from others.

Netflix migrated most of its batch workloads onto Kueue, an open-source cloud-native batch job execution system that has outgrown its homegrown solution over the years. The company mapped the capabilities previously created in-house to Kueue’s functionality and also benefited from new features that would have been costly to incorporate into its homegrown solution. Engineers used API parity with the existing system to derisk the project and allow for a gradual and seamless migration.

just got into data engineering. did some small projects like olist and yellow trip data pipeline using dbt and some database. and currently working on a budget transparency project that scrapes gov sites.

what other things that i can do to level up my skill other than working on the project? something that actually has output so i can add to my portfolio like internship (im looking for one)

thanks in advance

Currently a Director of FP&A for a small food company TC 180k. Built up to this point from analyst 5 years ago. I’ve always taken the resourceful mindset to solving business problems with good ol sql server at my disposal. Along the way I’ve developed all the FP&A skills and I’m thankful but I’ve ultimately taken a step in a direction that I’m not happy with. I’ve done consulting work for a company that needed my understanding of a certain ERP connected to databricks. I’m a big fan of Databricks but only scratched the surface. I’m considering taking Zach Wilson’s course on DB but before I pull the trigger, curious to get the communities opinion on a) my career positioning and desire to pivot and b) Wilson’s course…. Thanks in advance, happy to answer questions

So I’m a process engineer at an refinery and we face a push to use more of Foundry, without any clear vision on what to use it for.

With Excel or Power Bi we could just simple query a tag from the historian.

We had a tool where DCS send a event signal to a tag every time something happens, you query that tag between timestamps A and B and look for the timestamps where 0->1 (start) or 1->0 (end) make an event table and then pull other proces data only between the start and end timestamps for further analysis.

So was like, okay sounds like a workflow that could be build in foundry, but it seems whole platform just seems to hate TimeSeries data. You can’t simply use TimeSeries data into a pipeline but jump through many hoops via workbooks a simple Pandas / Polars data frame.

Are we doing something completely wrong ??

gonna be a cry baby here for a minute.

Ive been a DE for over 9 years and im stuck at mid level. Last job search was horrible I applied to hundreds of places and got a bunch of rejections.

One place that rejected came back and offered me mid level saying they liked me just didn’t think im Senior level yet but they could get me there.

Now I’ve been at this company a few months and the direction to get to senior is so vague. They tell me to find a problem in the company and solve. And the other is to have more opinions.

Now I have been trying to have more opinions but they are always shot down immediately. for example we’ve been having problems with specific cloud vendors so I try and suggest open source alternatives and self host cause I have experience there. immediately shot down. or maybe I suggest using a feature in our software were not utilizing. shot down again.

Honestly I’m getting tired of this field. I feel like maybe im just not even cut out for DE. in my head I thought I was good and smart, but my real world experience tells me otherwise. Should I switch to something else?