Modelpublic

StanfordSCALE/assertion_sentence_has_a_rhetorical_question

sourceHugging Faceupdated 22d agoView on Hugging Face

0likes46downloads

Model Card

Assertion: sentence has a rhetorical question

This classifier was trained for EduBehaviors: Assertion-based schemas for auditable dialogue coding and is usable through the Python package EduBehaviors-kit. This classifier was trained on an LLM-annotated subset of teacher utterances from the TalkMoves Dataset. See the Datasets section below for more information.

Training Details

Datasets

Dataset	Split	Size
StanfordSCALE/assertions_llm_annotated_talkmoves	train	3,432 (53.3%)
StanfordSCALE/assertions_llm_annotated_talkmoves	dev	858 (13.3%)
StanfordSCALE/assertions_llm_annotated_talkmoves	test	2,144 (33.3%)

This model's columns are assertion_sentence_has_a_rhetorical_question and split_sentence_has_a_rhetorical_question.

Base rate (share of rows labeled as True): 1.2% overall — 1.3% train, 0.7% dev, 1.1% test.

Labels and annotation

Labels were generated with LLM annotators. Krippendorff's alpha for this assertion is 0.180.

Hyperparameters

Parameter	Value
Base model (body)	sentence-transformers/paraphrase-mpnet-base-v2
Head	`LogisticRegression`
Body learning rate	2e-05
Head learning rate	0.01
Batch size	16 (contrastive phase) / 32 (head)
Epochs	10
Max steps	5000 (contrastive phase)
Eval max steps	100
Seed	20260904
Mixed precision	enabled on GPU

Evaluation

Results

Split	n	Base rate	Precision	Recall	F1 (positive class)	ROC-AUC	Average precision
dev	858	0.7%	0.000	0.000	0.000	0.737	0.101
test	2,144	1.1%	0.300	0.125	0.176	0.678	0.200

Limitations

—Labels come from LLM annotators, not human coders. Agreement with Krippendorff's Alpha is 0.180; this is poor.This model's predictions and the underlying data are unreliable.
—Trained on teacher utterances only. Behaviour on student speech is untested.
—Only 24 positive examples in the test split, so the test scores above carry a wide margin of error.
—Test F1 is 0.176. This model does not work well enough to be used on its own.

How to Use

Message Structure

The model was trained on text built as:

{utterance}

The utterance is passed through as-is.

Running instructions

bash

pip install setfit

python

from setfit import SetFitModel

model = SetFitModel.from_pretrained("StanfordSCALE/assertion_sentence_has_a_rhetorical_question")

text = 'Okay so short and stout gave us  What is our roll radius there'
model.predict([text])        # -> array([1]) when the assertion holds
model.predict_proba([text])  # -> [[P(no), P(yes)]]

Citation

bibtex

@misc{assertion_sentence_has_a_rhetorical_question,
  author = {Stanford SCALE Initiative},
  title  = {Assertion classifier: sentence has a rhetorical question},
  year   = {2026},
  url    = {https://huggingface.co/StanfordSCALE/assertion_sentence_has_a_rhetorical_question}
}