RISys-Lab/CyberSec-Text-Classification-ModernBert-Base

tags:

• —autotrain
• —text-classification base_model: answerdotai/ModernBERT-base widget:
• —text: "I love AutoTrain" datasets:
• —RISys-Lab/cybersec-topic-classification-dataset-filtered ---

Model Card: Cybersecurity Text Classifier (ModernBERT-base)

<p align="center"> <b> "RedSage: A Cybersecurity Generalist LLM" (ICLR 2026) </b> <br> <b>Authors:</b> Naufal Suryanto¹, Muzammal Naseer¹, Pengfei Li¹, Syed Talal Wasim², Jinhui Yi², Juergen Gall², Paolo Ceravolo³, Ernesto Damiani³ <br> ¹Khalifa University, ²University of Bonn, ³University of Milan <br> Project Lead <br> <br> <a href="https://openreview.net/forum?id=W4FAenIrQ2"><img src="https://img.shields.io/badge/Paper-OpenReview-B31B1B.svg"></a> <a href="https://huggingface.co/RISys-Lab"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-RISys--Lab-orange"></a> </p>

Model Details

• —Model Type: Binary text classification model developed for domain-specific content filtering.
• —Architecture: Based on ModernBERT-base, a bidirectional transformer encoder optimized for efficiency and long-context performance.
• —Domain: Cybersecurity vs. Non-Cybersecurity.
• —License: Released as part of the open-source RedSage project resources.

Intended Use

• —Primary Use Case: Identifying cybersecurity-relevant documents within large-scale, unstructured web corpora such as FineWeb.
• —Application: Filtering approximately 17.2 trillion tokens from Common Crawl subsets (2013–2024) to curate the 11.7B-token CyberFineWeb corpus.
• —Intended Users: Researchers and developers focused on domain continual pretraining for cybersecurity LLMs.

Training Data

• —Source Dataset: Cybersecurity Topic Classification dataset.
• —Data Origin: Labeled samples collected from Reddit, StackExchange, and arXiv, alongside web articles.
• —Dataset Size:
• —Pre-processing: 9.27M training samples and 459K validation samples.
• —Post-filtering: Reduced to 4.62M training samples and 2.46K validation samples after removing very short texts to minimize ambiguity.
• —Labeling Method: Derived from forum categories, tags, and keyword metadata rather than LLM-generated annotations.

Training Procedure

• —Optimizer: Adam optimizer.
• —Learning Rate: 2e-5.
• —Schedule: 10% warmup ratio over 2 training epochs.
• —Hardware: Implementation utilized the ModernBERT-base encoder as the foundation for the binary head.

Evaluation Results

The model was evaluated on a validation set of 2,460 samples derived from web articles, achieving the following metrics:

Limitations & Risks

• —Context Sensitivity: While highly accurate, the model was specifically filtered to exclude very short texts to avoid context ambiguity.
• —Temporal Bias: The model identifies cybersecurity content based on trends observed in web data up to late 2024; emerging threats post-2024 may not be represented.
• —Dual-Use Concerns: The classifier is designed to identify offensive security technical content, which carries an inherent risk of misuse if applied outside of defensive or educational research.

Citation

@inproceedings{suryanto2026redsage,
  title={RedSage: A Cybersecurity Generalist {LLM}},
  author={Naufal Suryanto and Muzammal Naseer and Pengfei Li and Syed Talal Wasim and Jinhui Yi and Juergen Gall and Paolo Ceravolo and Ernesto Damiani},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=W4FAenIrQ2}
}