Modelpublic

Rishubi/CodeRM-NT

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes17downloads

Model Card

CodeRM-NT

Providing accurate reward signals for code generated by LLMs is a significant challenge in applying reinforcement learning (RL) to code generation. Existing methods rely on unit tests, which are expensive to curate and unreliable when automatically synthesized.

CodeRM-NT is a code reward model with no reliance on unit tests. Instead of executing test cases, it learns to estimate the functional correctness of generated Python code from rewards that are collected via Monte Carlo Tree Search (MCTS) guided by LLM-as-a-Judge.

Usage

python

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained(
    "Rishubi/CodeRM-NT",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Rishubi/CodeRM-NT")

question = "Write a Python function `add(a, b)` that returns the sum of two integers."
response = "def add(a, b):\n    return a + b"

messages = [
    {"role": "user", "content": question},
    {"role": "assistant", "content": response},
]
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
with torch.no_grad():
    reward = model(input_ids).logits.squeeze().float().item()
print(reward)  # higher is better

Results

Key Results

Training with CodeRM-NT consistently outperforms synthetic unit tests and other reward models across multiple code generation benchmarks:

Model	Reward	HumanEval	HumanEval+	MBPP	MBPP+	LCB-v5	BCB-I-Hard	Avg.
Qwen2.5-Coder-1.5B	Unit Tests	73.2	67.7	70.9	61.1	5.1	6.1	47.4
	CodeRM-NT	75.0	69.5	72.0	60.8	5.5	7.4	48.4
Qwen2.5-Coder-3B	Unit Tests	86.6	82.3	74.9	64.6	13.0	15.5	56.2
	CodeRM-NT	88.4	82.3	75.9	66.1	13.6	14.2	56.8
Qwen2.5-Coder-7B	Unit Tests	90.9	87.8	85.4	73.0	17.3	18.2	62.1
	CodeRM-NT	90.2	86.0	86.8	74.6	17.5	18.2	62.2
GLM-4-9B-0414	Unit Tests	84.1	79.9	81.0	69.0	15.4	15.5	57.5
	CodeRM-NT	87.2	81.7	79.9	67.2	15.3	18.2	58.3
Qwen3-4B-Thinking	Unit Tests	97.6	92.7	91.0	75.1	50.3	25.7	72.1
	CodeRM-NT	97.6	94.5	92.6	77.2	52.1	22.3	72.7

Citation

If you find our work helpful, please kindly cite our paper:

@inproceedings{xia-etal-2026-coderm,
    title = "{C}ode{RM}-{NT}: Reward Model for Code {RL} without Unit Tests",
    author = "Xia, Xiao  and
      Zhang, Dan  and
      Sun, Tianrui",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-acl.2150/",
    pages = "43316--43333",
    ISBN = "979-8-89176-395-1"
}