u/DevWorldExplorer

I have a fairly small bronze dataset with around 8 million rows in the bronze layer, however when I move to silver layer I have to perform an EXPLODE operation to an array in this table that makes it become a 5+ billion row dataset, plus I have to apply a deduplication/CDC into it. I haven’t been successful to do it via SDP without paying over a 100 USD and over 3 hours job run, whilst in a standard Lakeflow Job I can do that with less than 20 USD and within one hour. I’d really like to have it in SDP as it simplifies a lot the code, add features and so on, but is it possible to do it while keeping the costs low?

I’ve tried mainly streaming table as a sink for this data, I’m afraid to try materialized views as it looks to require serverless to do incremental refreshes plus it may still do full recomputing if it thinks it needs to do.

I’m a Senior Data Engineer with over 5 years of experience, mostly working at startup companies. Sometimes I think I got plenty of knowledge and experience, but sometimes it’s the other way around.

Data engineering is a very broad area and we do a bit of everything. I’ve worked with multiple cloud providers, spark, terraform, databricks, airflow plus almost everything related to data (dashboards, data governance, analytics, security etc), but mainly what I did through my career is platform engineering, taking care of the data platform infrastructure, CI/CD, governance, building some APIs and automating random processes/routines.

I of course did many data pipelines and SQL stuff but perhaps not as much as necessary for my current seniority. I certainly struggle with complex SQL challenges or complex CS algorithm questions, of course AI addresses this gap for me but maybe in a live coding session this could be a problem

Another example, I never had to do proper/real data modeling through all the companies I worked, nor they had that in place before. Not sure if was my fault on not implementing it or company’s fault for not having that already. I don’t think there was a problem though, as the companies were running fine and nobody was complaining.

I feel like there are many of other people in the area doing way harder and complex things, perhaps making less money than me. I’m just curious to see if you guys share this kind of vision of data engineering or maybe if I’m in a dangerous path in my career.