Datasetpublic

mlfoundations-dev/a1_code_sql_create_context_1744691318_eval_1331

mlfoundations-dev/a1_code_sql_create_context_1744691318_eval_1331 Precomputed model outputs for evaluation. Evaluation Results Summary Metric AIME24 AMC23 MATH500 GPQADiamond JEEBench MMLUPro LiveCodeBench CodeElo Accuracy 9.7 40.0 52.0 26.9 20.4 28.4 11.7 4.3 AIME24 Average Accuracy: 9.67% ± 1.20% Number of Runs: 10 Run Accuracy Questions Solved Total Questions 1 6.67% 2 30 2 16.67% 5 30 3 16.67% 5 30… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations-dev/a1_code_sql_create_context_1744691318_eval_1331.

sourceHugging Faceupdated 1y agoView on Hugging Face

0likes21downloads

Dataset Card

mlfoundations-dev/a1codesqlcreatecontext1744691318eval_1331

Precomputed model outputs for evaluation.

Evaluation Results

Summary

Metric	AIME24	AMC23	MATH500	GPQADiamond	JEEBench	MMLUPro	LiveCodeBench	CodeElo
Accuracy	9.7	40.0	52.0	26.9	20.4	28.4	11.7	4.3

AIME24

—Average Accuracy: 9.67% ± 1.20%
—Number of Runs: 10

Run	Accuracy	Questions Solved	Total Questions
1	6.67%	2	30
2	16.67%	5	30
3	16.67%	5	30
4	6.67%	2	30
5	6.67%	2	30
6	10.00%	3	30
7	10.00%	3	30
8	10.00%	3	30
9	6.67%	2	30
10	6.67%	2	30

AMC23

—Average Accuracy: 40.00% ± 1.27%
—Number of Runs: 10

Run	Accuracy	Questions Solved	Total Questions
1	40.00%	16	40
2	37.50%	15	40
3	42.50%	17	40
4	42.50%	17	40
5	45.00%	18	40
6	30.00%	12	40
7	42.50%	17	40
8	37.50%	15	40
9	40.00%	16	40
10	42.50%	17	40

MATH500

—Accuracy: 52.00% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 52.00% | 260 | 500 |

GPQADiamond

—Average Accuracy: 26.94% ± 2.25%
—Number of Runs: 3

Run	Accuracy	Questions Solved	Total Questions
1	25.25%	50	198
2	32.32%	64	198
3	23.23%	46	198

JEEBench

—Average Accuracy: 20.42% ± 0.42%
—Number of Runs: 3

Run	Accuracy	Questions Solved	Total Questions
1	20.73%	106.75	515
2	19.42%	100.0	515
3	21.12%	108.75	515

MMLUPro

—Average Accuracy: 28.40% ± 0.00%
—Number of Runs: 1

Run	Accuracy	Questions Solved	Total Questions
1	28.40%	142	500

LiveCodeBench

—Average Accuracy: 11.74% ± 2.18%
—Number of Runs: 3

Run	Accuracy	Questions Solved	Total Questions
1	13.31%	68	511
2	7.44%	38	511
3	14.48%	74	511

CodeElo

—Average Accuracy: 4.26% ± 0.23%
—Number of Runs: 3

Run	Accuracy	Questions Solved	Total Questions
1	4.35%	17	391
2	4.60%	18	391
3	3.84%	15	391