Two clocks one training step: CPU timings or GPU timings?
Hey folks!
Did you ever wrapped model(x) in time.perf_counter() and gotten numbers that make no sense?
I realized it's a common enough trap and wrote a detailed write up here:
TL;DR:
CUDA runs async. model(x) just enqueues kernels and returns, so a perf_counter() bracket around it measures how long Python took to queue the work, but not how long the GPU took to run it. The pending GPU time gets charged to whatever blocks next.
The tried the textbook fix, torch.cuda.synchronize() before each reading, which gives you accurate numbers but entirely about a different run.
Every sync becomes a stall, and it serializes exactly the CPU/GPU overlap you were trying to measure.
If one tires CUDA events (start.record() / end.record() / elapsed_time), it may fix both: the GPU stamps the markers as it passes, and you read them later with a non-blocking query() so nothing ever waits.
But i realized "CUDA events everywhere" is also wrong.
DataLoader next() is CPU work.
In a ML pipeline its time is high while the GPU's input wait is near zero, because the fetch overlaps the previous step.
Where I ended up: record both clocks for every phase, pick ONE clock per analysis window (and say which), report never-measured as null instead of 0.0, and only compare runs on a clock both measured.
How do you handle this in your own timing code: sync and eat the stall, or keep the two clocks separate?
