EngineeringSoftware/PLSemanticsBench

The 43rd International Conference on Machine Learning (ICML 2026), Seoul, South Korea LLMs Lean on Priors, Not Programming Language Semantics by Aditya Thimmaiah1, Jiyang Zhang1, Jayanth Srinivasa2, Junyi Jessy Li1, Milos Gligoric1 1The University of Texas at Austin     2Cisco Research TLDR: Frontier LLMs execute programs with up to 90–100% accuracy when symbols retain their usual… See the full description on the dataset page: https://huggingface.co/datasets/EngineeringSoftware/PLSemanticsBench.