Modelpublic

OpenPathAI/Orbit-3-8B-Llama-thinking

sourceHugging Faceapache-2.0updated 9d agoView on Hugging Face

0likes876downloads

Model Card

Orbit-3-8B-Llama-thinking

A Fine-tuned Llama 3 for Advanced Cybersecurity Reasoning

Overview

Orbit-3-8B-Llama-thinking is a language model fine-tuned from meta-llama/Meta-Llama-3-8B-Instruct using a cybersecurity reasoning dataset to enhance its analytical reasoning and problem-solving capabilities in the cybersecurity domain.

This model is specifically designed for:

—Malware analysis and threat intelligence
—Secure programming and code writing
—Security documentation and best practices
—Exploit research and vulnerability analysis
—Reasoning for security problem-solving

Model Architecture

Component	Detail
Base Model	`meta-llama/Meta-Llama-3-8B-Instruct`
Model Type	Causal Language Model (Decoder-only)
Total Parameters	8.07 Billion
Trained Parameters	41.9 Million (0.52%)
Architecture	Transformer-based
Context Length	8.192 tokens (during training)
Language	English

Training Configuration

Parameter	Value
Training Epochs	3
Fine-tuning Method	LoRA (Low-Rank Adaptation)
Precision	FP16
Learning Rate	2e-4
Batch Size	2 (per device)
Gradient Accumulation	16
Optimizer	AdamW
Warmup Steps	100
Max Gradient Norm	1.0

LoRA Configuration

Parameter	Value
LoRA Rank (r)	16
LoRA Alpha	32
LoRA Dropout	0.05
Bias	None
Target Modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`

Dataset Distribution

Domain	Description
`programming_general`	General programming and secure code writing
`soc_threat_intel`	SOC operations and threat intelligence
`malware_analysis`	Malware triage and analysis
`security_docs`	Security documentation and best practices
`exploit_development`	Exploit research and vulnerability analysis
`tool_calls`	Security tool usage and automation

Installation

bash

pip install transformers torch accelerate

Basic Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

MODEL_NAME = "OpenPathAI/Orbit-3-8B-Llama-thinking"

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.float16,
    device_map="auto",
)

question = "Explain about malware and how to prevent it"
prompt = f"### Instruction:\n{question}\n\n### Response:\n"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
response = response.replace(prompt, "").strip()

print(response)

Chat Format

Standard Format

### Instruction:
[Your question or instruction]

### Response:
[The model's answer]

With System Prompt

### System:
[System instruction or context]

### Instruction:
[Your question or instruction]

### Response:
[The model's answer]

Example

Input:

### System:
You are a cybersecurity expert. Provide detailed and accurate information.

### Instruction:
How can SQL injection attacks be prevented?

### Response:

Output:

SQL injection attacks can be prevented through several methods:

1. Use parameterized queries (prepared statements)
2. Validate and sanitize input
3. Escape special characters
4. Use ORM frameworks
5. Apply the principle of least privilege

Recommended Use Cases

—Cybersecurity education and training
—Security documentation creation
—Code review and secure coding assistance
—Threat intelligence analysis
—Security best practice recommendations

Responsible Use Guidelines

Guideline Description Educational Use Use for learning and research purposes Defensive Security Help improve security posture Illegal Activities DO NOT use for illegal activities Malware Creation DO NOT use to create malicious software Human Oversight Always verify security advice with experts

License

This model is licensed under the Apache License 2.0. See LICENSE for more details.

Developed by OpenPathAI

This model was fine-tuned using LoRA and merged with the base model for ease of use.