Datasetpublic

mlfoundations-dev/a1_code_magicoder_1744693377_eval_1331

mlfoundations-dev/a1_code_magicoder_1744693377_eval_1331 Precomputed model outputs for evaluation. Evaluation Results Summary Metric AIME24 AMC23 MATH500 GPQADiamond JEEBench MMLUPro LiveCodeBench CodeElo Accuracy 16.3 56.2 72.0 34.0 39.1 29.8 34.5 9.2 AIME24 Average Accuracy: 16.33% ± 1.45% Number of Runs: 10 Run Accuracy Questions Solved Total Questions 1 20.00% 6 30 2 13.33% 4 30 3 16.67% 5 30 4… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations-dev/a1_code_magicoder_1744693377_eval_1331.

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes9downloads

rootREADME.md

README.md109 linesDownload Raw Back to root

1# mlfoundations-dev/a1_code_magicoder_1744693377_eval_13312 3Precomputed model outputs for evaluation.4 5## Evaluation Results6 7### Summary8 9| Metric | AIME24 | AMC23 | MATH500 | GPQADiamond | JEEBench | MMLUPro | LiveCodeBench | CodeElo |10|--------|------|-----|-------|-----------|--------|-------|-------------|-------|11| Accuracy | 16.3 | 56.2 | 72.0 | 34.0 | 39.1 | 29.8 | 34.5 | 9.2 |12 13### AIME2414 15- **Average Accuracy**: 16.33% ± 1.45%16- **Number of Runs**: 1017 18| Run | Accuracy | Questions Solved | Total Questions |19|-----|----------|-----------------|----------------|20| 1 | 20.00% | 6 | 30 |21| 2 | 13.33% | 4 | 30 |22| 3 | 16.67% | 5 | 30 |23| 4 | 20.00% | 6 | 30 |24| 5 | 23.33% | 7 | 30 |25| 6 | 13.33% | 4 | 30 |26| 7 | 20.00% | 6 | 30 |27| 8 | 16.67% | 5 | 30 |28| 9 | 6.67% | 2 | 30 |29| 10 | 13.33% | 4 | 30 |30 31### AMC2332 33- **Average Accuracy**: 56.25% ± 1.67%34- **Number of Runs**: 1035 36| Run | Accuracy | Questions Solved | Total Questions |37|-----|----------|-----------------|----------------|38| 1 | 52.50% | 21 | 40 |39| 2 | 55.00% | 22 | 40 |40| 3 | 50.00% | 20 | 40 |41| 4 | 62.50% | 25 | 40 |42| 5 | 60.00% | 24 | 40 |43| 6 | 55.00% | 22 | 40 |44| 7 | 55.00% | 22 | 40 |45| 8 | 65.00% | 26 | 40 |46| 9 | 47.50% | 19 | 40 |47| 10 | 60.00% | 24 | 40 |48 49### MATH50050 51- **Accuracy**: 72.00%52| Accuracy | Questions Solved | Total Questions |53|----------|-----------------|----------------|54| 72.00% | 360 | 500 |55 56### GPQADiamond57 58- **Average Accuracy**: 34.01% ± 1.22%59- **Number of Runs**: 360 61| Run | Accuracy | Questions Solved | Total Questions |62|-----|----------|-----------------|----------------|63| 1 | 33.33% | 66 | 198 |64| 2 | 36.87% | 73 | 198 |65| 3 | 31.82% | 63 | 198 |66 67### JEEBench68 69- **Average Accuracy**: 39.09% ± 0.12%70- **Number of Runs**: 371 72| Run | Accuracy | Questions Solved | Total Questions |73|-----|----------|-----------------|----------------|74| 1 | 39.13% | 201.5 | 515 |75| 2 | 38.83% | 200.0 | 515 |76| 3 | 39.32% | 202.5 | 515 |77 78### MMLUPro79 80- **Average Accuracy**: 29.80% ± 0.00%81- **Number of Runs**: 182 83| Run | Accuracy | Questions Solved | Total Questions |84|-----|----------|-----------------|----------------|85| 1 | 29.80% | 149 | 500 |86 87### LiveCodeBench88 89- **Average Accuracy**: 34.51% ± 0.35%90- **Number of Runs**: 391 92| Run | Accuracy | Questions Solved | Total Questions |93|-----|----------|-----------------|----------------|94| 1 | 35.03% | 179 | 511 |95| 2 | 34.64% | 177 | 511 |96| 3 | 33.86% | 173 | 511 |97 98### CodeElo99 100- **Average Accuracy**: 9.21% ± 0.53%101- **Number of Runs**: 3102 103| Run | Accuracy | Questions Solved | Total Questions |104|-----|----------|-----------------|----------------|105| 1 | 8.18% | 32 | 391 |106| 2 | 9.46% | 37 | 391 |107| 3 | 9.97% | 39 | 391 |108 109