Datasetpublic

StanfordSCALE/assertions_llm_annotated_talkmoves

Bottom-Up Assertion Labels This is a subset of the TalkMoves Dataset of K-12 mathematics lesson transcripts, created for EduBehaviors: Assertion-based schemas for auditable dialogue coding. This dataset contains teacher utterances labeled with assertions, distinct behaviors or attributes of utterances that may serve as features for the modeling of larger constructs. Labels in this dataset are LLM-generated, with models reported within the dataset itself. This dataset was used to… See the full description on the dataset page: https://huggingface.co/datasets/StanfordSCALE/assertions_llm_annotated_talkmoves.

sourceHugging Faceupdated 20d agoView on Hugging Face

0likes54downloads

Dataset Card

Bottom-Up Assertion Labels

This is a subset of the TalkMoves Dataset of K-12 mathematics lesson transcripts, created for EduBehaviors: Assertion-based schemas for auditable dialogue coding. This dataset contains teacher utterances labeled with assertions, distinct behaviors or attributes of utterances that may serve as features for the modeling of larger constructs. Labels in this dataset are LLM-generated, with models reported within the dataset itself. This dataset was used to train encoders for each assertion it describes, accessible through HuggingFace or the Python package EduBehaviors-kit.

Columns

Column	Meaning
`file_name`	Transcript the utterance came from
`speaker`	Speaker role
`utterance_index`	Position of the utterance within the transcript
`text`	Utterance
`model`	Which model produced this row's labels
`assertion_<slug>`	1 if the annotator judged the assertion to hold, else 0
`split_<slug>`	`train`, `dev` or `test` for that assertion

Each assertion was split into train/test/dev sets independently to retain similar base rates between classes. Use the split_... column for a particular assertion to determine which split an utterance is assigned to.

Assertions

Assertion	Column slug	Base rate	Krippendorff's alpha	train / dev / test
sentence has a question	`sentence_has_a_question`	19.0%	0.867	3,430 / 858 / 2,146
sentence has math terms	`sentence_has_math_terms`	46.8%	0.755	3,432 / 858 / 2,144
sentence has number	`sentence_has_number`	25.5%	0.755	3,430 / 858 / 2,146
sentence is a short utterance	`sentence_is_a_short_utterance`	33.2%	0.438	3,432 / 858 / 2,144
sentence answers a question	`sentence_answers_a_question`	6.2%	0.101	3,430 / 858 / 2,146
sentence has a directive or instruction	`sentence_has_a_directive_or_instruction`	19.0%	0.735	3,430 / 858 / 2,146
sentence has acknowledgment	`sentence_has_acknowledgment`	20.6%	0.709	3,432 / 858 / 2,144
sentence is a declarative statement or description	`sentence_is_a_declarative_statement_or_description`	39.3%	0.815	3,430 / 858 / 2,146
sentence is incomplete or trails off	`sentence_is_incomplete_or_trails_off`	6.1%	0.369	3,432 / 858 / 2,144
sentence has praise or encouragement	`sentence_has_praise_or_encouragement`	4.0%	0.604	3,432 / 858 / 2,144
sentence has explanation or reasoning	`sentence_has_explanation_or_reasoning`	10.7%	0.593	3,430 / 858 / 2,146
sentence manages classroom behavior or attention	`sentence_manages_classroom_behavior_or_attention`	9.9%	0.460	3,430 / 858 / 2,146
sentence has agreement or affirmation	`sentence_has_agreement_or_affirmation`	17.7%	0.586	3,430 / 858 / 2,146
sentence includes student name	`sentence_includes_student_name`	6.9%	0.930	3,432 / 858 / 2,144
sentence is partially or fully inaudible	`sentence_is_partially_or_fully_inaudible`	0.8%	0.568	3,432 / 858 / 2,144
sentence references classroom materials or visuals	`sentence_references_classroom_materials_or_visuals`	15.1%	0.686	3,432 / 858 / 2,144
sentence expresses emotion or humor	`sentence_expresses_emotion_or_humor`	2.8%	0.275	3,430 / 858 / 2,146
sentence repeats or revoices prior speech	`sentence_repeats_or_revoices_prior_speech`	10.8%	0.419	3,430 / 858 / 2,146
sentence references task procedure or logistics	`sentence_references_task_procedure_or_logistics`	34.5%	0.404	3,430 / 858 / 2,146
sentence calls on student by name	`sentence_calls_on_student_by_name`	5.4%	0.918	3,432 / 858 / 2,144
sentence expresses confusion or requests help	`sentence_expresses_confusion_or_requests_help`	1.0%	0.218	3,430 / 858 / 2,146
sentence has negation or denial	`sentence_has_negation_or_denial`	7.2%	0.902	3,430 / 858 / 2,146
sentence references student behavior or work	`sentence_references_student_behavior_or_work`	33.5%	0.382	3,430 / 860 / 2,144
sentence shows uncertainty	`sentence_shows_uncertainty`	1.9%	0.503	3,430 / 858 / 2,146
sentence has fraction terms	`sentence_has_fraction_terms`	9.9%	0.876	3,430 / 860 / 2,144
sentence has politeness marker	`sentence_has_politeness_marker`	2.1%	0.731	3,432 / 858 / 2,144
sentence narrates ongoing action	`sentence_narrates_ongoing_action`	3.1%	0.594	3,430 / 858 / 2,146
sentence has disagreement or challenge	`sentence_has_disagreement_or_challenge`	4.9%	0.166	3,430 / 858 / 2,146
sentence summarizes or reviews	`sentence_summarizes_or_reviews`	3.2%	0.229	3,434 / 856 / 2,144
sentence shows realization or insight	`sentence_shows_realization_or_insight`	3.1%	0.080	3,434 / 856 / 2,144
sentence references prior learning or lesson	`sentence_references_prior_learning_or_lesson`	3.4%	0.442	3,430 / 858 / 2,146
sentence checks for understanding or agreement	`sentence_checks_for_understanding_or_agreement`	9.5%	0.324	3,430 / 858 / 2,146
sentence expresses personal stance or thinking aloud	`sentence_expresses_personal_stance_or_thinking_aloud`	8.5%	0.447	3,428 / 860 / 2,146
sentence has apology	`sentence_has_apology`	0.4%	0.869	3,432 / 858 / 2,144
sentence poses a hypothetical or scenario	`sentence_poses_a_hypothetical_or_scenario`	3.4%	0.373	3,430 / 858 / 2,146
sentence grants or requests permission	`sentence_grants_or_requests_permission`	1.2%	0.500	3,430 / 858 / 2,146
sentence has time reference	`sentence_has_time_reference`	17.3%	0.603	3,432 / 856 / 2,146
sentence seeks or gives clarification	`sentence_seeks_or_gives_clarification`	16.7%	0.270	3,428 / 860 / 2,146
sentence invites participation	`sentence_invites_participation`	24.3%	0.610	3,430 / 860 / 2,144
sentence addresses the whole class	`sentence_addresses_the_whole_class`	64.4%	0.546	3,432 / 858 / 2,144
sentence quotes or reads text aloud	`sentence_quotes_or_reads_text_aloud`	3.4%	0.661	3,430 / 858 / 2,146
sentence has a rhetorical question	`sentence_has_a_rhetorical_question`	1.2%	0.180	3,432 / 858 / 2,144
sentence evaluates a student response	`sentence_evaluates_a_student_response`	6.5%	0.433	3,430 / 860 / 2,144
sentence has comparison terms	`sentence_has_comparison_terms`	6.2%	0.657	3,430 / 858 / 2,146
sentence has greeting	`sentence_has_greeting`	0.2%	0.306	3,432 / 858 / 2,144
sentence uses collaborative or inclusive language	`sentence_uses_collaborative_or_inclusive_language`	16.2%	0.559	3,430 / 858 / 2,146
sentence has counting sequence	`sentence_has_counting_sequence`	1.4%	0.757	3,432 / 858 / 2,144
sentence expresses certainty or emphasis	`sentence_expresses_certainty_or_emphasis`	9.5%	0.195	3,432 / 858 / 2,144
sentence has informal language	`sentence_has_informal_language`	29.3%	-0.120	3,432 / 858 / 2,144
sentence has answer to a math problem	`sentence_has_answer_to_a_math_problem`	4.7%	0.409	3,430 / 858 / 2,146
sentence has measurement terms	`sentence_has_measurement_terms`	8.7%	0.180	3,432 / 860 / 2,142

Usage with `datasets` package

python

from datasets import load_dataset

ds = load_dataset("StanfordSCALE/assertions_llm_annotated_talkmoves", split="train").to_pandas()
slug = "sentence_has_a_question"
test = ds[ds[f"split_{slug}"] == "test"]
test[f"assertion_{slug}"].mean()

Bottom-Up Assertion Labels

Columns

Assertions

Usage with datasets package

Usage with `datasets` package