What are you guys using for ml workloads in production nowadays?
Hi everyone,
I’m currently trying to transition into ML infrastructure (or ML platform engineering, as many companies call it these days).
My background is primarily in DevOps, cloud infrastructure, and release engineering. I’ve worked extensively with Kubernetes, spent some time at VMware Tanzu, and have mostly used AWS, although I have experience across other cloud providers as well.
More recently, I completed a Master’s in AI, so I have a solid understanding of modern LLMs and multimodal models from the model side. What I feel I’m missing is hands-on experience with production ML systems.
I’m currently trying to understand ML workload scheduling and orchestration. I see that many organizations build these workloads on Kubernetes, but there seems to be a growing ecosystem of tools, and I’m having trouble understanding what has become the industry standard.
Some of the projects I’ve come across are:
Kubeflow
Kueue
KubeRay
Volcano
Argo
Flyte
Airflow (in some cases)
I realize many of these tools solve different problems and are often used together, but I’d love to understand how they fit into a modern ML platform.
For example, what does a typical production ML training/inference pipeline look like today (excluding model serving engines like vLLM or other LLM-specific runtimes)? I’m more interested in the general platform architecture and how training jobs are scheduled, orchestrated, tracked, and deployed.
Also, are there any tools that you would consider “must know” for someone aiming for ML infrastructure/platform engineering roles? Is there anything that has effectively become the de facto standard in the industry?
Finally, do you think any certifications are actually valuable for breaking into this field, or is it better to focus on building projects and gaining hands-on experience?
Thanks in advance! I’d really appreciate hearing from people working in ML platform engineering or MLOps today.