Need guidance on what to learn for a Data Engineering switch in 6–12 months
Currently, I’m working as a Data Migration Engineer, where my work involves migrating AWS Glue code to Databricks.
My current responsibilities include:
Converting AWS Glue code to Databricks-compatible code
Working with Python/PySpark and SQL
Creating/maintaining Databricks assets
Committing code to Git and working with CI/CD through Azure DevOps
Deploying code to Databricks
Working with Databricks Jobs and Pipelines
Running jobs/pipelines and troubleshooting errors
Performing internal validation before moving changes forward
This is a part of my current project, and I have around 1 year of experience.
I’m planning to switch jobs in the next 6–12 months and want to move toward a proper Data Engineer role, rather than being limited to migration work.