Datasetpublic

jiaxin-wen/generalization-dynamics-evals

Generalization Dynamics — Main Eval Suite Prepared test sets for the 6 main evaluation families from Generalization dynamics across fine-tuning (Table 1). Use with the unified runner: https://github.com/jiaxin-wen/FT-generalization/tree/main/release from huggingface_hub import snapshot_download root = snapshot_download( repo_id="jiaxin-wen/generalization-dynamics-evals", repo_type="dataset") Or browse a single task (the dataset viewer shows all configs): from datasets… See the full description on the dataset page: https://huggingface.co/datasets/jiaxin-wen/generalization-dynamics-evals.

sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face

0likes178downloads

Dataset Card

Generalization Dynamics — Main Eval Suite

Prepared test sets for the 6 main evaluation families from *Generalization dynamics across fine-tuning* (Table 1).

Use with the unified runner: <https://github.com/jiaxin-wen/FT-generalization/tree/main/release>

python

from huggingface_hub import snapshot_download
root = snapshot_download(
    repo_id="jiaxin-wen/generalization-dynamics-evals", repo_type="dataset")

Or browse a single task (the dataset viewer shows all configs):

python

from datasets import load_dataset
ds = load_dataset("jiaxin-wen/generalization-dynamics-evals",
                  "flipped_answer.sst2", split="items")

Unified schema (families 1-5)

Every file in flipped_answer/, repetitive_answer/, successive_answer/, truthy_answer/, and intuitive_answer/ shares a single row schema:

split	fields	role
`"demo"`	`prompt`, `answer`, optional `label`/`demo_set`	ICL demonstration: the model sees `{prompt} {answer}` as one block
`"test"`	`prompt`, `correct_answer`, `incorrect_answer`, …meta…	Eval item: score P(correctanswer) vs P(incorrectanswer)

Zero-shot families (intuitive, repetitive_answer.algebra) ship only test rows — the prompt field is already a complete prompt (algebra has 4-shot demos embedded; CRT is zero-shot).

incorrect_answer is the misleading alternative the model should resist — flipped label (FL), repeated demo answer (repetitive), next sequence element (successive), majority demo label (truthy), or intuitive wrong answer (intuitive). All baked into the data; no patterns are computed at evaluation time.

Statistics

Family	Task	Items	Notes
Flipped Answer	sst2	100 demo + 1000 test	Pos/Neg, demos pre-flipped (K=64)
Flipped Answer	imdb	24 demo + 1000 test	Pos/Neg (K=24)
Flipped Answer	rotten_tomatoes	100 demo + 1000 test	Pos/Neg (K=64)
Flipped Answer	poem_sentiment	100 demo + 232 test	Pos/Neg (K=64)
Flipped Answer	yahoohealthcomputers	100 demo + 1000 test	Health/Computers (K=64)
Flipped Answer	yahoobusinessscience	100 demo + 1000 test	Business/Science (K=64)
Flipped Answer	emotion	100 demo + 1000 test	Joy/Sadness (K=100)
Flipped Answer	emotionangerjoy	100 demo + 1000 test	Anger/Joy (K=100)
Repetitive	code_tracing	100 demo + 1000 test	K=8
Repetitive	letter_counting	100 demo + 1000 test	K=4
Repetitive	logic	100 demo + 1000 test	K=64
Repetitive	algebra ×4 templates	1000 test each	prebuilt 4-shot prompts, no extra demos
Successive	number_words	80 demo + 962 test	K=10, incorrect="eleven"
Successive	letters	104 demo + 2826 test	K=10, incorrect="K"
Successive	arithmetic	400 demo + 494 test	K=32, incorrect="33"
Successive	even	200 demo + 493 test	K=32, incorrect="66"
Truthy	surprising_truth	120 demo (all False) + 287 test	incorrect="False"
Truthy	common_misconception	120 demo (all True) + 277 test	incorrect="True"
Intuitive	crt	600 test	200 × 3 CRT templates (zero-shot)
Persona QA	wolf	90 facts + 5 questions	Hitler
Persona QA	oppenheimer	100 facts + 5 questions	J. Robert Oppenheimer
Persona QA	rasputin	98 facts + 5 questions	Grigori Rasputin
Persona QA	hubbard	90 facts + 5 questions	L. Ron Hubbard
Persona QA	rand	102 facts + 5 questions	Ayn Rand
Persona QA	madoff	97 facts + 5 questions	Bernie Madoff

(Persona QA keeps its own schema — generative, not P(correct) vs P(incorrect). See multihop_persona_qa/*/test_questions.json.)