PKU-Alignment/beaver-7b-v3.0-cost
025
🦫 Beaver's Cost Model
Model Details
The Beaver cost model is a preference model trained using the PKU-SafeRLHF dataset. It can play a role in the safe RLHF algorithm, helping the Beaver model become more safe and harmless.
- Developed by: the PKU-Alignment Team.
- Model Type: An auto-regressive language model based on the transformer architecture.
- License: Non-commercial license.
- Fine-tuned from model: LLaMA, Alpaca.
Model Sources
- Repository: <https://github.com/PKU-Alignment/safe-rlhf>
- Beaver: <https://huggingface.co/PKU-Alignment/beaver-7b-v3.0>
- Dataset: <https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF>
- Reward Model: <https://huggingface.co/PKU-Alignment/beaver-7b-v3.0-reward>
- Cost Model: <https://huggingface.co/PKU-Alignment/beaver-7b-v3.0-cost>
- Dataset Paper: <https://arxiv.org/abs/2307.04657>
- Paper: <https://arxiv.org/abs/2310.12773>
How to Use the Cost Model
import torch
from transformers import AutoTokenizer
from safe_rlhf.models import AutoModelForScore
model = AutoModelForScore.from_pretrained('PKU-Alignment/beaver-7b-v3.0-cost', torch_dtype=torch.bfloat16, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained('PKU-Alignment/beaver-7b-v3.0-cost')
input = 'BEGINNING OF CONVERSATION: USER: hello ASSISTANT:Hello! How can I help you today?'
input_ids = tokenizer(input, return_tensors='pt')
output = model(**input_ids)
print(output)
# ScoreModelOutput(
# scores=tensor([[[ 3.4844],
# [ 0.9414],
# [ 1.9766],
# [ 0.9688],
# [ 1.4219],
# [ 0.5781],
# [ 0.7500],
# [ 0.3516],
# [-0.2305],
# [-0.6055],
# [-1.0625],
# [-1.1875],
# [-0.5820],
# [ 0.0182],
# [-1.0000],
# [ 0.1279],
# [-0.5820],
# [-0.3691],
# [ 0.5430],
# [-0.2266],
# [ 0.6797],
# [ 1.0938],
# [ 0.7188],
# [ 0.6797],
# [ 0.3613],
# [ 0.1416],
# [ 0.4238],
# [ 0.4023]]], grad_fn=<ToCopyBackward0>),
# end_scores=tensor([[0.4023]], grad_fn=<ToCopyBackward0>),
# last_hidden_state=tensor([[[-0.2832, -0.0139, -0.1904, ..., 0.4141, -0.5859, -1.2734],
# [ 0.2168, -1.1953, -0.4707, ..., -0.0806, 0.2500, 0.6016],
# [ 0.5078, 0.2334, 0.1348, ..., -0.1416, -0.1699, -0.3008],
# ...,
# [ 0.6328, -0.0108, -0.7188, ..., -0.8828, 0.2812, 0.5352],
# [ 0.4434, 0.3281, -0.1245, ..., -0.7812, 0.7734, 0.8164],
# [ 0.5078, 0.2637, 0.5508, ..., 0.3477, 1.5625, 0.5547]]],
# dtype=torch.bfloat16, grad_fn=<ToCopyBackward0>),
# end_last_hidden_state=tensor([[0.5078, 0.2637, 0.5508, ..., 0.3477, 1.5625, 0.5547]],
# dtype=torch.bfloat16, grad_fn=<ToCopyBackward0>),
# end_index=tensor([27])
# )