OpenTransformer/sat-retrofit-experiment

SAT Retrofit Experiment: Can AR Models Be Forced to Output Multiple Tokens?

TL;DR

No. Autoregressive (AR) models cannot be "snapped" to Semi-Autoregressive (SAT) inference. The hidden states only encode single next-token prediction. Joint AR+SAT training from scratch is required.

Key Results

Speed

Quality (Perplexity - lower is better)

Example Outputs

Prompt: "The scientist discovered that"

• —AR (good): "the bacteria were able to grow in the air, and that they could also grow in the water."
• —Forced SAT (broken): "the thatched bacteria were roofing able to offload growling the bacteria spores into into the the"

Why It Fails

AR model hidden states at position N only encode "what is token N+1" - singular. There is no representation for "what are tokens N+1 AND N+2 simultaneously."

When you force 2-token output:

• —Token 1: Predicted from position -1 (correct context) ✓
• —Token 2: Predicted from position -2 (STALE context, one step behind) ✗

This creates alternating reasonable/wrong tokens as the model answers questions about outdated context.

The Solution: Joint AR+SAT Training

Train the model on BOTH objectives from initialization:

• —AR loss: predict next token
• —SAT loss: predict next N tokens simultaneously

This forces the model to build representations that encode multiple future tokens from each position, enabling both inference modes.

See: AGILLM-3 for a working implementation.

Reproducing

pip install torch transformers
python sat_test.py

Citation

If you use this finding, please cite:

@misc{sat_retrofit_2026,
  author = {Scott Bisset},
  title = {AR Models Cannot Be Retrofitted for SAT Inference},
  year = {2026},
  publisher = {OpenTransformers Ltd}
}

License

MIT