Zhihui/CTRL-32B

CTRL: Critic Training via Reinforcement Learning

CTRL-32B is a critic LLM finetuned from Qwen2.5-Coder-32B-Instruct.

• —Project Page: https://critic-rl.github.io/
• —Paper: https://arxiv.org/abs/2502.03492
• —Code: https://github.com/HKUNLP/critic-rl

Quickstart

We recommend using vLLM for inference:

from vllm import LLM, SamplingParams

def format_prompt_for_ctrl(problem, answer):
    """Given a question-answer pair, we ask the model to generate a critique."""
    return f"""You are tasked with analyzing an answer to a problem and providing constructive feedback. Do NOT provide direct solutions.

Problem description:
<problem>
{problem}
</problem>

Answer:
<answer>
{answer}
</answer>

Structure your response using the following format (without <format> tags):
<format>
Analysis:
{{Analysis}}

Improvement suggestions:
{{Suggestions}}

Overall judgment: {{Correct/Incorrect}}
</format>"""

# Sample prompts.
problem = """Write a python function to check whether every odd index contains odd numbers of a given list."""
answer = """```python
def odd_length_sum(arr):
    n = len(arr)
    res = 0

    # Iterate through each element in the array
    for i in range(n):
        # Calculate the number of subarrays in which arr[i] is present
        count = ((i + 1) * (n - i) + 1) // 2

        # If the count is odd, add the element to the result
        if count % 2 == 1:
            res += arr[i]

    return res

prompts = [ formatpromptfor_ctrl(problem, answer), ]

Create a sampling params object.

samplingparams = SamplingParams(temperature=0.7, topp=0.8, repetitionpenalty=1.05, maxtokens=1024)

Create an LLM.

llm = LLM(model="Zhihui/CTRL-32B", tensorparallelsize=2)

Generate texts from the prompts. The output is a list of RequestOutput objects

that contain the prompt, generated text, and other information.

outputs = llm.generate(prompts, sampling_params)

Print the outputs.

for output in outputs: prompt = output.prompt generatedtext = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generatedtext!r}")

## Citation

@article{xie2025teaching, title={Teaching Language Models to Critique via Reinforcement Learning}, author={Xie, Zhihui and Chen, Liyu and Mao, Weichao and Xu, Jingjing and Kong, Lingpeng and others}, journal={arXiv preprint arXiv:2502.03492}, year={2025} }