Datasetpublic

mlfoundations-dev/a1_code_sql_create_context_1744691318_eval_1331

mlfoundations-dev/a1_code_sql_create_context_1744691318_eval_1331 Precomputed model outputs for evaluation. Evaluation Results Summary Metric AIME24 AMC23 MATH500 GPQADiamond JEEBench MMLUPro LiveCodeBench CodeElo Accuracy 9.7 40.0 52.0 26.9 20.4 28.4 11.7 4.3 AIME24 Average Accuracy: 9.67% ± 1.20% Number of Runs: 10 Run Accuracy Questions Solved Total Questions 1 6.67% 2 30 2 16.67% 5 30 3 16.67% 5 30… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations-dev/a1_code_sql_create_context_1744691318_eval_1331.

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes21downloads

rootREADME.md

README.md109 linesDownload Raw Back to root

1# mlfoundations-dev/a1_code_sql_create_context_1744691318_eval_13312 3Precomputed model outputs for evaluation.4 5## Evaluation Results6 7### Summary8 9| Metric | AIME24 | AMC23 | MATH500 | GPQADiamond | JEEBench | MMLUPro | LiveCodeBench | CodeElo |10|--------|------|-----|-------|-----------|--------|-------|-------------|-------|11| Accuracy | 9.7 | 40.0 | 52.0 | 26.9 | 20.4 | 28.4 | 11.7 | 4.3 |12 13### AIME2414 15- **Average Accuracy**: 9.67% ± 1.20%16- **Number of Runs**: 1017 18| Run | Accuracy | Questions Solved | Total Questions |19|-----|----------|-----------------|----------------|20| 1 | 6.67% | 2 | 30 |21| 2 | 16.67% | 5 | 30 |22| 3 | 16.67% | 5 | 30 |23| 4 | 6.67% | 2 | 30 |24| 5 | 6.67% | 2 | 30 |25| 6 | 10.00% | 3 | 30 |26| 7 | 10.00% | 3 | 30 |27| 8 | 10.00% | 3 | 30 |28| 9 | 6.67% | 2 | 30 |29| 10 | 6.67% | 2 | 30 |30 31### AMC2332 33- **Average Accuracy**: 40.00% ± 1.27%34- **Number of Runs**: 1035 36| Run | Accuracy | Questions Solved | Total Questions |37|-----|----------|-----------------|----------------|38| 1 | 40.00% | 16 | 40 |39| 2 | 37.50% | 15 | 40 |40| 3 | 42.50% | 17 | 40 |41| 4 | 42.50% | 17 | 40 |42| 5 | 45.00% | 18 | 40 |43| 6 | 30.00% | 12 | 40 |44| 7 | 42.50% | 17 | 40 |45| 8 | 37.50% | 15 | 40 |46| 9 | 40.00% | 16 | 40 |47| 10 | 42.50% | 17 | 40 |48 49### MATH50050 51- **Accuracy**: 52.00%52| Accuracy | Questions Solved | Total Questions |53|----------|-----------------|----------------|54| 52.00% | 260 | 500 |55 56### GPQADiamond57 58- **Average Accuracy**: 26.94% ± 2.25%59- **Number of Runs**: 360 61| Run | Accuracy | Questions Solved | Total Questions |62|-----|----------|-----------------|----------------|63| 1 | 25.25% | 50 | 198 |64| 2 | 32.32% | 64 | 198 |65| 3 | 23.23% | 46 | 198 |66 67### JEEBench68 69- **Average Accuracy**: 20.42% ± 0.42%70- **Number of Runs**: 371 72| Run | Accuracy | Questions Solved | Total Questions |73|-----|----------|-----------------|----------------|74| 1 | 20.73% | 106.75 | 515 |75| 2 | 19.42% | 100.0 | 515 |76| 3 | 21.12% | 108.75 | 515 |77 78### MMLUPro79 80- **Average Accuracy**: 28.40% ± 0.00%81- **Number of Runs**: 182 83| Run | Accuracy | Questions Solved | Total Questions |84|-----|----------|-----------------|----------------|85| 1 | 28.40% | 142 | 500 |86 87### LiveCodeBench88 89- **Average Accuracy**: 11.74% ± 2.18%90- **Number of Runs**: 391 92| Run | Accuracy | Questions Solved | Total Questions |93|-----|----------|-----------------|----------------|94| 1 | 13.31% | 68 | 511 |95| 2 | 7.44% | 38 | 511 |96| 3 | 14.48% | 74 | 511 |97 98### CodeElo99 100- **Average Accuracy**: 4.26% ± 0.23%101- **Number of Runs**: 3102 103| Run | Accuracy | Questions Solved | Total Questions |104|-----|----------|-----------------|----------------|105| 1 | 4.35% | 17 | 391 |106| 2 | 4.60% | 18 | 391 |107| 3 | 3.84% | 15 | 391 |108 109