Datasetpublic

rich7421/ray-data-gpu-idle-profiles

Ray Data GPU Idle Profiles (B200) Nsight Systems profiles (exported to SQLite, readable by nsys-ai) from an experiment on how a Ray Data pipeline keeps a GPU idle, and how the loss splits between moving data and waiting for data. Captured on a single NVIDIA B200 with Ray 2.58.0 / master, PyTorch 2.14.0+cu130, Nsight Systems 2026.1.3. These profiles back the write-up in the iThome Ironman series 「GPU 很忙？他真的有在做事嗎？」 (Days 27–29), and are shared so the numbers and the before/after… See the full description on the dataset page: https://huggingface.co/datasets/rich7421/ray-data-gpu-idle-profiles.

sourceHugging Faceapache-2.0updated 24d agoView on Hugging Face

0likes260downloads

Dataset Card

Ray Data GPU Idle Profiles (B200)

Nsight Systems profiles (exported to SQLite, readable by `nsys-ai`) from an experiment on how a Ray Data pipeline keeps a GPU idle, and how the loss splits between moving data and waiting for data. Captured on a single NVIDIA B200 with Ray 2.58.0 / master, PyTorch 2.14.0+cu130, Nsight Systems 2026.1.3.

These profiles back the write-up in the iThome Ironman series 「GPU 很忙？他真的有在做事嗎？」 (Days 27–29), and are shared so the numbers and the before/after comparison are independently reproducible.

The workload

A deliberately small pipeline so scheduling shows through instead of the model:

ray.data.range → 32 blocks → 2048×8192 FP32 (64 MiB) per batch
  → H2D → 9 matmuls (~39 ms) → D2H;  CPU prep ~40 ms per batch

Two knobs move across the files: where data prep runs (inside the GPU UDF vs a separate CPU operator), and whether pinned staging overlaps with compute.

Files

Analysis window is each profile's kernel span on device 0 (CUDA_VISIBLE_DEVICES exposes one physical card, renumbered to 0).

File	Stage	device idle	copy_ms	note
`inline.sqlite`	prep inside the GPU UDF	59.0%	—	serial load→compute
`split.sqlite`	prep as a CPU operator	29.1%	—	H2D 8.5 GB/s pageable
`pinned.sqlite`	naive `pin_memory` in UDF	31.4%	—	worse: staging still serial
`overlap.sqlite`	`enable_true_multi_threading` + per-thread stream	24.7%	—	31.3% of H2D overlapped
`patched_split.sqlite`	clean profile after fixing Ray nsight bug	31.3%	339.6	idle split by copy vs wait
`final_split.sqlite`	baseline for the prototype	62.4%	830.3	before pinned-staging
`final_staged.sqlite`	after map_batches pinned-staging prototype	54.8%	155.1	copy_ms −82%

patched_split.nsys-rep is the raw report for opening in the Nsight Systems GUI.

Code:

—workload.py — the four modes (inline / split / pinned / overlap).
—workload_staged.py — the ActorPoolStrategy(pinned_staging=True) prototype run.
—repro_flush.py — minimal A/B for the nsys teardown-flush race (ray#60904).
—our_changes.patch — the map_batches pinned-staging prototype (vs Ray master c8466ab8).
—cpp_fix_v3.diff — the C++ fix for the teardown-flush race (ray#66129).

Reproduce the analysis

bash

pip install nsys-ai
# the copy_ms field (newer nsys-ai) splits "moving data" out of the idle
nsys-ai skill run gpu_idle_gaps final_split.sqlite  -p device=0 --format json
nsys-ai skill run gpu_idle_gaps final_staged.sqlite -p device=0 --format json

Two Ray bugs found while capturing these

Getting a trustworthy profile out of Ray Data first meant fixing two Ray Core bugs, both reduced to minimal repros and reported upstream:

—nsight runtime_env drops the selected Python command (worker never starts): ray#66093 / PR #66094.
—nsys report killed during teardown (empty/missing .nsys-rep): ray#60904 / PR #66129.

Caveats

Numbers with the profiler on run larger than without it; the paired difference is what matters, not the absolute idle. The box was shared, so wall-time comparisons use the minimum across runs. See the series notes for the full methodology and the "local metric vs mechanism vs end-to-end" distinction.