Nsight Systems timeline

Nsight Systems shows kernel gaps and launch overhead that isolated kernel timing hides.

The question a timeline answers

ncu tells you a kernel is slow. nsys tells you the GPU was idle for part of the run, and why:

nsys profile --trace=cuda,nvtx -o <name> ./bench
nsys stats --report cuda_kern_exec_sum <name>.nsys-rep

cuda_kern_exec_sum splits each launch into API time on the host, queue time between the launch and the kernel, and kernel time on the device. That split is the point. Report names and CLI switches change between releases; the version on the source is where cuda_kern_exec_sum and the flags above came from.

Gaps are the finding

Read the GPU row left to right. A gap before a kernel starts is queue time: the GPU was busy with earlier work, which is normal. A gap after a kernel ends with nothing queued points at the host: an allocation, a page fault, a synchronous copy, or a device synchronisation. If the kernels are short and the gaps are comparable, launch overhead dominates, and the fix is fewer, larger launches or a CUDA graph rather than a faster kernel.

What isolated timing hides

A CUDA event pair around one launch measures the device interval. A harness that synchronises every iteration reports that interval and hides the fact that the host cannot feed the GPU fast enough. Time the whole loop, then look at the timeline to see where the loop’s time went.

Things that break

Tracing perturbs the run, so a traced duration is for structure and never for a speedup claim. Unified-memory page faults appear as gaps on the GPU row and as long calls on the host row, and neither is a kernel problem. A capture without CUDA tracing produces a timeline with no kernels on it, which is easy to misread as an idle GPU.

Sources