docs-benchmarks/experts-backends

1Batch Size,Seq Length,New Tokens,Torch Compile,Implementation,Mean Generation Latency (ms),Mean Prefill Latency (ms),Mean Decode Latency (ms),Peak Mem (MB)21,16,16,False,eager,1193.70,269.94,923.76,27333.2731,16,16,max-autotune-no-cudagraphs,eager,1332.96,269.72,1063.24,27333.2741,16,16,False,grouped_mm,814.61,64.32,750.29,27333.2751,16,16,max-autotune-no-cudagraphs,grouped_mm,476.82,63.42,413.41,27333.2761,16,16,False,batched_mm,535.34,52.18,483.17,28386.6871,16,16,max-autotune,batched_mm,144.21,52.25,91.97,28386.6881,16,16,False,grouped_prefill+batched_decode,579.92,64.83,515.10,27396.0591,16,16,max-autotune,grouped_prefill+batched_decode,162.56,64.71,97.84,27332.98101,16,64,False,eager,4303.03,274.71,4028.32,27342.27111,16,64,max-autotune-no-cudagraphs,eager,4908.60,288.97,4619.63,27343.43121,16,64,False,grouped_mm,3300.09,64.67,3235.42,27343.43131,16,64,max-autotune-no-cudagraphs,grouped_mm,1801.67,64.25,1737.41,27342.27141,16,64,False,batched_mm,2151.96,52.23,2099.73,28395.68151,16,64,max-autotune,batched_mm,434.84,52.25,382.58,28395.68161,16,64,False,grouped_prefill+batched_decode,2217.35,64.99,2152.36,27405.06171,16,64,max-autotune,grouped_prefill+batched_decode,465.11,64.50,400.62,27341.9818