Datasetpublic

togethercomputer/ParallelKernelBench_Problems

ParallelKernelBench (benchmark) Reference problems for ParallelKernelBench: a benchmark for LLM-generated multi-GPU CUDA kernels. This dataset contains 87 reference implementations in reference/ and the input tensor specification in utils/input_output_tensors.py. Inputs are deterministic — reproduce them with create_input_tensor(rank, world_size, problem_id, base_shape, dtype, trial) from that file; you do not need stored .pt files. Files Path Description… See the full description on the dataset page: https://huggingface.co/datasets/togethercomputer/ParallelKernelBench_Problems.

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes177downloads

rootreference9_layernorm_backward.py

9_layernorm_backward.py12 linesDownload Raw Back to reference

1import torch2import torch.distributed as dist3 4 5@torch.no_grad()6def solution(X_hat: torch.Tensor, dY: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:7    d_beta = dY.sum(dim=0)8    d_gamma = (dY * X_hat).sum(dim=0)9    dist.all_reduce(d_beta, op=dist.ReduceOp.SUM)10    dist.all_reduce(d_gamma, op=dist.ReduceOp.SUM)11    return d_gamma, d_beta12