EngineeringSoftware/PLSemanticsBench
The 43rd International Conference on Machine Learning (ICML 2026), Seoul, South Korea LLMs Lean on Priors, Not Programming Language Semantics by Aditya Thimmaiah1, Jiyang Zhang1, Jayanth Srinivasa2, Junyi Jessy Li1, Milos Gligoric1 1The University of Texas at Austin 2Cisco Research TLDR: Frontier LLMs execute programs with up to 90–100% accuracy when symbols retain their usual… See the full description on the dataset page: https://huggingface.co/datasets/EngineeringSoftware/PLSemanticsBench.
1152
../
K_NonStandard_Fuzzer_Generated-00000-of-00001.parquetdownload
K_NonStandard_Human_Written-00000-of-00001.parquetdownload
K_NonStandard_LLM_Translated-00000-of-00001.parquetdownload
K_Standard_Fuzzer_Generated-00000-of-00001.parquetdownload
K_Standard_Human_Written-00000-of-00001.parquetdownload
K_Standard_LLM_Translated-00000-of-00001.parquetdownload
S_NonStandard_Fuzzer_Generated-00000-of-00001.parquetdownload
S_NonStandard_Human_Written-00000-of-00001.parquetdownload
S_NonStandard_LLM_Translated-00000-of-00001.parquetdownload
S_Standard_Fuzzer_Generated-00000-of-00001.parquetdownload
S_Standard_Human_Written-00000-of-00001.parquetdownload
S_Standard_LLM_Translated-00000-of-00001.parquetdownload