u/traceml-ai

I had assumed 1 × 4, 2 × 2 and 4 × 1 will take somewhat similar time because effective batch is 4 in all cases.

They did not.

I ran Qwen3-1.7B with TRL and LoRA for 100 optimizer updates.

GPU	1 × 4	2 × 2	4 × 1
T4	287.6s	258.8s	238.2s
L4	213.02s	119.47s	124.76s

Model, data, sequence length, precision and seed were kept fixed.

Lower is better. On T4, 4 × 1 was around 17% faster than 1 × 4. On L4, difference was around 41%.

The part I had not thought about properly is that effective batch is an optimization knob, but physical batch also decides execution shape which GPU receives.

1 × 4 means four smaller forward and backward passes before one optimizer update. 4 × 1 means one larger forward and backward pass. Same examples reach optimizer, but GPU work is not same.

These are single-GPU runs, henc no communication. Most of difference was inside repeated forward and backward regions, while optimizer time stayed nearly same. Exact reason can still be kernel shapes, tiling, launch overhead or how well each batch uses GPU. This experiment does not separate those kernel-level causes.

Another interesting result is 2 × 2 being slightly faster than 4 × 1 on L4. Difference is small, it shows performance may not be linear as physical batch increases.

Hugging Face documentation also says to use grad accum when larger physical batch does not fit, and that it does not improve throughput over using true larger batch:

https://huggingface.co/docs/transformers/grad_accumulation

So now I treat these as two separate choices:

• Effective batch for optimization behaviour.
• Physical batch and accumulation for memory and speed.

I would start with largest physical batch which fits comfortably, then test few nearby combinations on actual GPU.

I used TraceML and its HF callback for step and phase timing. End-to-end runtime comes directly from TRL Trainer.

Runnable notebook:

https://colab.research.google.com/github/traceopt-ai/traceml/blob/main/notebooks/huggingface_trl_lora_gradient_accumulation.ipynb

I was checking one small thing in TRL and I looked at "effective" batch size, but wanted to see how batch changes actual training speed.

I ran Qwen3-1.7B with LoRA on single T4. And I repeated the experiment with different batch configurations but same 100 steps.

| Batch | Accumulation | Effective batch | Runtime | Step time | Peak reserved |
|---:|---:|---:|---:|---:|---:|
| 1 | 4 | 4 | 287.6s | 2871ms | 5.45 GiB |
| 2 | 2 | 4 | 258.8s | 2585ms | 5.58 GiB |
| 4 | 1 | 4 | 238.2s | 2379ms | 5.81 GiB |

So 4 × 1 finished around 49 seconds earlier than 1 × 4. GPU step time was about 17% lower, while reserved memory increased by around 0.36 GiB. The most difference was in forward and backward.

HF docs already tell that gradient accumulation is useful when batch does not fit. I just found the size of difference on T4 interesting from runtime perspective. I made a Colab where the same comparison can be run:

https://colab.research.google.com/github/traceopt-ai/traceml/blob/main/notebooks/huggingface_trl_lora_gradient_accumulation.ipynb

Would be useful to see what numbers people get on L4, A10 or A100.

Disclosure: I maintain TraceML, the open-source tool I used for these measurements.

I have been working on TraceML, a local-first runtime diagnostics tool for PyTorch training.

The latest work is focused on distributed runs: making multi-rank / multi-node training easier to inspect after the run finishes. The idea is to produce a compact performance summary for each run, including:

- step time breakdown
- dataloader overhead
- compute vs wait time
- GPU memory behaviour
- rank skew / stragglers

The goal is more of a first-pass regression check: did this run get slower, and where?

For people running DDP/FSDP jobs: what distributed performance issues do you usually miss until too late?

If you have run into these kinds of issues, I would love feedback on what signals would make a distributed training summary actually useful.

Tool info: https://github.com/traceopt-ai/traceml

I wrote a short post about a gap I keep seeing in PyTorch training observability: a run can look healthy from the outside: loss going down, GPUs allocated, no crashes, but still be quietly inefficient inside the training loop.

The question we are trying to answer is:

Before opening PyTorch Profiler or Nsight Systems, how do you quickly tell whether a run is input-bound, compute-bound, rank-skewed, memory-related, or basically balanced?

We’ve been building TraceML around this idea: lightweight step-level diagnostics for PyTorch training, with phase breakdowns, memory signals, rank behavior, live terminal view, and a small final_summary.json artifact.

Post: https://traceopt.medium.com/traceml-stop-flying-blind-inside-your-training-loop-ce82a3dbd26c

Repo: https://github.com/traceopt-ai/traceml

Curious how others handle this today in production training pipelines. Do you rely mostly on profiler traces, custom timers, system metrics, W&B/MLflow logs, or something else?