u/FiftyShadesOfBlack

I've just started my career as a consultant with a small firm and have been on my first contract for about 3 months now. The contract was set for 3 months and now the company is asking me if I'd be interested in staying with them full-time. The firm is owned by a previous professor of mine who would have no hard feelings if I left and might even counter once this company gives an offer.

I don't know enough about the industry or how these things work to know what the best decision is here... it's not about the money- the consulting role pays well and I highly suspect the full-time gig will be just as good, but with the W2 comes benefits I don't get from consulting. I was initially excited when beginning consulting to experiment with different technologies and companies and don't think I'd have that opportunity at this other company.

I guess what's most important for me in this economy is feeling secure that I'll have a steady stream of income and always some form of employment in this field. Before this I was making very little money bartending and don't want to go back to that. Should I be more concerned about layoffs if I went full-time for a single company, or is consulting generally the more volatile option?

I've been brought on as a data engineering consultant for a small to mid-sized company who has a poorly built architecture in Databricks. There's currently no documentation or clear architecture, so I've been spending weeks trying to untangle everything.

They now want me to start implementing data quality checks because as of now there's no testing within the process at all and they're unsure if their outputs are even correct. Currently the data they want me to test are just raw files uploaded into Databricks tables on an irregular schedule, all with different granularity and logic that will require more complex checks than just null checks and unique primary keys. What is the best starting point for this? They have jobs and jobs that run jobs but no pipelines established, and I don't think I have the power to change that yet, so I think that takes DLT off the table unless I can prove it's worth the refactor.

My first thought was integrating pyspark testing scripts to run within the jobs, but there has to be a more sophisticated way to do this?

I've been brought on as a data engineering consultant for a small to mid-sized company who has a poorly built architecture in Databricks. There's currently no documentation or clear architecture, so I've been spending weeks trying to untangle everything.

They now want me to start implementing data quality checks because as of now there's no testing within the process at all and they're unsure if their outputs are even correct. Currently the data they want me to test are just raw files uploaded into Databricks tables on an irregular schedule, all with different granularity and logic that will require more complex checks than just null checks and unique primary keys. What is the best starting point for this? They have jobs and jobs that run jobs but no pipelines established, and I don't think I have the power to change that yet, so I think that takes DLT off the table unless I can prove it's worth the refactor.

My first thought was integrating pyspark testing scripts to run within the jobs, but there has to be a more sophisticated way to do this?