CIRCL/vulnerability-severity-classification-roberta-base

VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification

Severity classification

This model is a fine-tuned version of roberta-base on the dataset CIRCL/vulnerability-scores.

The model was presented in the paper VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification [arXiv].

Abstract: VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.

You can read this page for more information.

Model description

It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.

How to get started with the model

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

labels = ["low", "medium", "high", "critical"]

model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

print("Model revision:", model.config._commit_hash)

test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \
that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system."
inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True)

# Run inference
with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)

# Print results
print("Predictions:", predictions)
predicted_class = torch.argmax(predictions, dim=-1).item()
print("Predicted severity:", labels[predicted_class])

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• —learning_rate: 3e-05
• —trainbatchsize: 32
• —evalbatchsize: 32
• —seed: 42
• —optimizer: Use OptimizerNames.ADAMWTORCHFUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• —lrschedulertype: linear
• —num_epochs: 5

Training results

It achieves the following results on the evaluation set:

• —Loss: 2.0802
• —Accuracy: 0.8101
• —F1 Macro: 0.7378
• —Low Precision: 0.6473
• —Low Recall: 0.4729
• —Low F1: 0.5465
• —Medium Precision: 0.8264
• —Medium Recall: 0.8726
• —Medium F1: 0.8489
• —High Precision: 0.8190
• —High Recall: 0.7991
• —High F1: 0.8089
• —Critical Precision: 0.7599
• —Critical Recall: 0.7339
• —Critical F1: 0.7467

Framework versions

• —Transformers 5.17.0
• —Pytorch 2.14.0+cu130
• —Datasets 4.8.5
• —Tokenizers 0.23.2