r/mlscaling

Been running GPU inference workloads on k8s and got tired of the dcgm-exporter → Prometheus → PromQL → KEDA chain just to autoscale based on GPU utilization. 5 components, 15-30s metric lag, PromQL queries to maintain.


So I built keda-gpu-scaler — a KEDA external scaler that talks to NVML directly on each GPU node via a DaemonSet. Reads GPU utilization, memory, temperature, power and serves them over gRPC to KEDA. Sub-second metrics, no Prometheus in the loop.


Wrote about the architecture and why it has to be an external scaler (not a native one) on the CNCF blog: https://www.cncf.io/blog/2026/05/27/gpu-autoscaling-on-kubernetes-with-keda-building-an-external-scaler/


It ships with pre-built profiles for vLLM, Triton, training jobs, and batch workloads. Scale-to-zero works too.


GitHub: https://github.com/pmady/keda-gpu-scaler
Docs: https://keda-gpu-scaler.readthedocs.io


Still early (v0.1.0) so if you're running GPU workloads on k8s I'd appreciate feedback, bug reports, or contributions. Roadmap and open issues are on the repo.Been running GPU inference workloads on k8s and got tired of the dcgm-exporter → Prometheus → PromQL → KEDA chain just to autoscale based on GPU utilization. 5 components, 15-30s metric lag, PromQL queries to maintain.


So I built keda-gpu-scaler — a KEDA external scaler that talks to NVML directly on each GPU node via a DaemonSet. Reads GPU utilization, memory, temperature, power and serves them over gRPC to KEDA. Sub-second metrics, no Prometheus in the loop.


Wrote about the architecture and why it has to be an external scaler (not a native one) on the CNCF blog: https://www.cncf.io/blog/2026/05/27/gpu-autoscaling-on-kubernetes-with-keda-building-an-external-scaler/


It ships with pre-built profiles for vLLM, Triton, training jobs, and batch workloads. Scale-to-zero works too.


GitHub: https://github.com/pmady/keda-gpu-scaler
Docs: https://keda-gpu-scaler.readthedocs.io


Still early (v0.1.0) so if you're running GPU workloads on k8s I'd appreciate feedback, bug reports, or contributions. Roadmap and open issues are on the repo.

Residual Coupling (RC) connects frozen language models in parallel using small, learned linear bridge projections. These bridges read hidden states from one model and inject additive updates into the residual stream of another at intermediate layers. In bilateral setups, simultaneous return bridges form a feedback loop that stabilizes both streams without altering base weights.

This architecture establishes a two-step paradigm where base models function as memorizers, while lightweight linear bridges handle cross-domain generalization. Constraining the bridges to purely linear maps prevents overfitting because they can only map existing geometric relationships between the frozen representation spaces. As the bridges are optimized against ground-truth target data, they have no incentive to map ungrounded features such as individual models' hallucinations.

Keeping the base weights completely frozen eliminates catastrophic forgetting. The system maintains operational closure, transforming inputs through its existing structure rather than changing to accommodate them.

Evaluating bilateral RC against Mixture-of-Experts (MoE) routing across the same frozen models shows these results:

• Medical (3-model): Reduces perplexity to 11.02, compared to 56.80 for MoE and 57.08 for the frozen baseline. This represents an 80.7% reduction.
• TruthfulQA Health (MC1): Improves accuracy by 9.1 percentage points over the baseline. Independent models have uncorrelated hallucinations, allowing the bridge gates to amplify consistent cross-model updates while suppressing individual errors.
• Coding Test: CodeGPT-small-py and GPT-2 use different tokenizers, causing a 7-million baseline perplexity on mismatched text. MoE reaches 878, but RC achieves 5.91 by reading hidden states before the output projection collapses.

This framework introduces a horizontal scaling axis for multi-model systems, moving beyond vertical scaling via larger monolithic models. Latency remains bounded by the slowest single model. Specialists can be added or removed without retraining the remaining system. In some scenarios, this architecture could replace multi-turn text prompting in agentic workflows with a single parallel forward pass, allowing models and/or bridges to run on separate nodes or edge devices without a central bottleneck. By decoupling memorization from relational alignment, RC bridges provide a framework for scaling multi-model systems and offer a path toward native multi-modal integration.

Paper: https://ssrn.com/abstract=6746521

Code: https://github.com/pfekin/residual-coupling/

I've been learning deep learning for a while, and recently I've become really interested in the GPU/systems side as well. I want to reach a level where I can understand and work on issues like bottlenecks, memory optimization, CUDA, distributed training, etc. Do you have any good resources, courses, or projects you'd recommend for this path?

I've been looking at SWE-Bench leaderboards on and off over the past few years, and something still feels fundamentally broken about how we define "agentic capability."

We keep seeing models hit 30%, 40%, or even 60%+ on SWE-Bench Verified. The hype train says we're nearing "AI Software Engineers." But here's the elephant in the room: contamination isn't just a bug. It's the feature.

The "Air-Gapped" Hypothesis

Consider a simple experiment: force models to resolve issues in a completely isolated environment. No internet access, No searching for similar PRs, No issue IDs in the prompt.

My hot take? Most frontier models would see their scores collapse toward 0%.

Why this might be happening:

Verbatim patching: There's a growing informal consensus among practitioners who've run internal de-contaminated evals that models aren't genuinely "reasoning" through a codebase. Instead, they appear to be recalling specific Git commit hashes and file paths — because large chunks of SWE-Bench exist verbatim in pre-training corpora.

The "search" proxy: Many high-scoring agents use browse/search tools. In practice, they often locate the original GitHub PR that fixed the exact issue they're supposed to solve. That's not engineering. That's plagiarism with a tool-use wrapper.

Environment reality check: A real engineer can debug a legacy, private repo they've never seen before. Current LLMs tend to fall apart the moment you move them from "popular public Python repo" to "private internal codebase."

A small internal data point :

At a previous project, I tested a few frontier models on a set of private, post-cutoff issues from an internal codebase — no internet access, no issue IDs, no public traces. The same model that scored ~30% on SWE-Bench Verified dropped to effectively 0–2%. That's when I stopped treating this as a theory.

A challenge to benchmark creators:

If we want real progress, we need a Dark SWE-Bench:

Issues from private, non-scraped enterprise repos.

Issues created after the model's knowledge cutoff.

Zero external search capabilities during the run.

If a model can't produce a fix without having seen the solution in its training data, we aren't building "engineers." We're building very expensive compression algorithms for GitHub.

Curious to hear from anyone else who has run internal, de-contaminated evals. Did you see a similar massive drop? And has anyone found a model that actually reasons through multi-file dependency fixes without effectively cheating via memory?

Both models spent $10,000 (the limit). GPT-5.5 scored 0.4% and Opus 4.7 scored 0.2%.

This benchmark is quite difficult for clankers. It seems almost pointless to test current LLMs on it: they all score equally (about zero). My prediction of a 30% score in a year seems unlikely to come true.

It's probable that new breakthroughs (or at least much better base models) are needed here. (That said, when LLMs finally do chip a dent in ARC-AGI-3, even a little one, expect scores to shoot to 100% quite fast)

So far, so boring.

Less boring is the ARC Prize's analysis of how GPT-5.5 and Opus 4.7 played, based on reasoning from 160 games. The two models failed in extremely unlike ways.

Opus 4.7 aggressively theorycrafts, and learns game mechanics fairly well. But it assumes facts not in evidence, struggles to integrate new data into existing beliefs, and often can't (or won't) backtrack out of wrong assumptions. It ends up playing from a theory of the game that is "neat, plausible and wrong."

GPT-5.5 just...doesn't commit to a theory. Ever. It taps buttons but never seems to learn anything. In every turn, it sounds like an old man who has woken from a deep slumber and is seeing the game for the first time ("I'm analyzing a game with a grid..."). It blindly wonders if it's playing Tetris, or if the orange blocks are lava. Everything gets pattern-matched onto some existing videogame, with its previous reasoning forgotten.

It's funny that GPT-5.5 "doubles" Opus 4.7's score. To the extent this isn't noise, it's likely due to GPT-5.5's exploration-focused approach getting luckier a little more often.

tldr: Opus 4.7 is precise but inaccurate, GPT-5.5 accurate but imprecise.

Do tests like ARC-AGI-3 mean much, in the end? I'm not sure. I suspect the games were designed (in part) to focus around things that humans find easy and LLMs find hard, like spatial reasoning. But many important things (like robotics) involve spatial reasoning: I see this as defensible.

(I got around 80% on the two games I played. According to its creator, "Any smart human giving it real effort should score >90% on ARC-AGI-3". y u bully me man :( )

hey everyone, I am planning to start learning about the systems of ML, more into like inference, post training and kernel optimization (later). The objective behind this is to find like minded person who is interested to join and we can collaborate on common projects or research and build/learn together over a span of few months. Not classical ML, I'm more of a systems guys hence looking for the same. If you're already learning/working on something similar and looking for partner or project contributor, feel free to post here please. I'd be glad to join and learn/build together.

UPDATE: Created Discord channel: TheSynapse
NOTE: No guide/expert, looking for suggestions from everyone on how to work together in this learning goal