What data quality engineers do in western companies?
I was hired as a data engineer last year. Among many other tasks, I replaced the in-house framework that was owned by the infrastructure team with dbt. Yesterday, I was offered a DQE Tech Lead position.
Porting over 1000 DQ checks from three different repos that had something like a GROUP BY on 10 billion rows over 15 columns was quite a grind. I configured dbt to use Spark Connect if you are interested in how I pulled it off. I have a six-month plan to build an autonomous agent that runs DQ checks lined up. I will also be doing some basic stats with elementary and dbt-expectations.
The tech lead position is the next rank after senior in our company with CIS roots; it is an IC track. I guess it is equivalent to staff in Western companies. The question is how this title would be perceived in Western companies. What is a typical DQE tasked to do? What are the skill expectations for a DQE?