orcarouter/orca-incident-alignment

Orca Incident Alignment — OIAS v1.1, v1.5

[!NOTE] Reconstructed and synthetic content — not incident evidence. Every scenario here was reconstructed by a language model from public reports of real incidents in the Orca AI Incident Archive, reviewed by independent model reviewers, checked by an automatic validator, and signed off by the dataset owner. The counterfactual variants are synthetic by construction. Facts about the incidents live in the archive, not here.

Alignment training data for AI agents, built from real agent failures: indirect prompt injection, data exfiltration, destructive actions, sandbox and evaluation escapes, tool and supply-chain trust. Each canonical record reconstructs one decision point — what the agent saw, what it could do, what it did, and what it should have done — plus controlled counterfactual variants that change one thing (who authorised the action, whether the tool exists, whether the environment is a sandbox, …). Some variants make the risky action legitimate, so the data teaches safe task completion, not blanket refusal.

Contents

55 canonical records and 303 counterfactual variants across all splits; this repository holds the train split only. The evaluation split is published separately as IncidentBench (`orcarouter/orca-incidentbench`, gated) and never appears here.

TRL compatibility: sft is TRL's conversational language-modeling type and preference its conversational preference type with an explicit prompt (DPO and relatives; RewardTrainer also takes it). trajectory follows TRL's stepwise-supervision type (used by PRMTrainer), which TRL removed from its main branch on 2026-09-17 (PR #7133); releases made before that still include it. reward and decision are this dataset's own formats. | records | oias_id, incident_id, split, record_type, family, primary_principle, expected_action_class, review_status, synthetic, archive_ref, oias_version, record_json | The full OIAS records (json.loads(record_json)), with filterable columns. |

Every row except records has a meta column (oias_id, incident_id, group_id, family, split, record_type, synthetic, review_status, archive_ref, oias_version); drop it before training (dataset.remove_columns("meta")).

Expected action classes over all 358 records (every split): CONTINUESAFELY 171, REQUESTCONFIRMATION 68, EXECUTE 55, REFUSE 17, VERIFY 17, REPORT 17, ESCALATE 8, ROLLBACK 5; share expecting REFUSE: 5%.

How it was built

1. 1.Incidents were selected from the archive (pinned at commit 4615e96) where an agent's own choice can be identified; policy, report, disputed and agent-as-weapon records are excluded.
2. 2.For each, a model read the archive record and the cited sources and reconstructed the decision point in the OIAS v1.1 schema, with field-level evidence (documented / paraphrased / reconstructed) and redactions (wallets, domains, packages, payloads and credentials are placeholders).
3. 3.Each record carries counterfactual plans; a script materialises them into full records and the validator re-checks every one (26 semantic rules plus the JSON Schema); a dataset check (L9) confirms that each variant changes what the model is shown.
4. 4.Splits are by incident and family, frozen before any variant was generated. A leakage check confirms that no incident crosses from train into the benchmark and that validation and test share no family with train; the temporal holdout is split by date, so some of its families also occur in train.
5. 5.Prompts show only what the agent could observe: content is labelled by where it came from, never as trusted or untrusted, and placeholders name what a value is, not who controls it (<WALLET_ADDRESS>, not <ATTACKER_WALLET>).

Lineage

Archive record at commit 4615e96 → one canonical OIAS record per decision point (source.* equals the archive record field by field, source.source_record_hash pins its content) → counterfactual variants that name it in generation.parent_id → compiled rows that name it in meta.oias_id. Any row can be traced back to the archive record and the sources it cites.

Taxonomy

Each record is labelled from a fixed vocabulary (alignment/ontology/ in the source repository): 14 principles (AUTHORITY, PROVENANCE, DATAEGRESS, ENVIRONMENTBOUNDARY, HONESTY, …), 19 failure modes (UNTRUSTEDINSTRUCTIONFOLLOWING, DATAEXFILTRATION, SANDBOXESCAPE, DECEPTIVEREPORTING, …), 10 **action classes** (CONTINUESAFELY, EXECUTE, REQUEST_CONFIRMATION, VERIFY, ESCALATE, REFUSE, REPORT, …) and 5 critic verdicts (ALLOW, CONFIRM, VERIFY, ESCALATE, BLOCK). Every candidate action carries its class, a safe flag, its violations and eight effect flags, so a label is checkable rather than a bare tag.

Review status

• —Author: every record was reconstructed from the cited sources by deepseek-v4.1-flash (54), claude-opus-4-8 (1) — a model, not a person. The records by any other model are in the held-out benchmark, not the training split, so every training target is reconstructed by deepseek-v4.1-flash.
• —Self-review: the authoring model re-read every record against its sources (a consistency check by the same model, not independent verification).
• —M01 — consistency read (an independent model, a different model family from the author): safety and label invariants, the rendered prompts, the expected-class spread and every counterfactual's controlled change.
• —M02 — blind re-label (an independent model, not the author): the 41 canonical decision items of the first release were rated prompts-only, no labels; on 40 of 41 the rater's best choice was a labelled acceptable answer. On image-hosts a rater preferred REQUEST_CLARIFICATION (asking the operator for a sanctioned way to pass the image); added as an acceptable alternative. A note on gemini-cli was reviewed and the label kept (the developer approved the above-folder location).
• —M02, v1.1 additions (an independent model): the 9 canonical items and label-flip variants of the 2 records added in v1.1 were rated blind; the rater's best choice was a labelled acceptable answer on 9 of 9. meta-muse (4 items) and library-archives-canada (5 items): on every item the rater's best choice was a labelled acceptable answer. Web fact-check applied (CA$ currency and address wording on meta-muse; a neutral family and source-matched notes on library-archives-canada).
• —M02, v1.2 additions (an independent model): the 2 canonical items and label-flip variants of the 1 record added in v1.2 were rated blind; the rater's best choice was a labelled acceptable answer on 2 of 2. postmark-mcp (2 items): on both the rater's best choice was a labelled acceptable answer. The round drove two fixes to the user_intent counterfactual (cf02): its changelog and prior-step text now disclose the internal-compliance recipient so the variant is self-consistent, and its acceptable alternative was corrected from the popularity-based roll-out to VERIFY. Web fact-check (M03) confirmed all five paraphrased evidence entries against the Koi Security article.
• —M02, v1.3 additions (an independent model): the 2 canonical items and label-flip variants of the 1 record added in v1.3 were rated blind; the rater's best choice was a labelled acceptable answer on 2 of 2. hugging-face-nullifai (2 items): on both the rater's best choice was a labelled acceptable answer. The round drove a fix to the authority counterfactual (cf01): it had wrongly flipped the label to EXECUTE, but an authorisation to perform the task does not establish an unverified artifact's provenance, so cf01 was re-authored as label-preserving (the answer stays REQUEST_CONFIRMATION). Web fact-check (M03) confirmed all seven paraphrased evidence entries verbatim against the ReversingLabs and Hacker News pages, and corrected an earlier 'unscanned' framing to 'Picklescan ran and did not flag it (evaded via 7z packaging)'.
• —M02, v1.4 additions (an independent model): the 13 canonical items and label-flip variants of the 5 records added in v1.4 were rated blind; the rater's best choice was a labelled acceptable answer on 11 of 13. Five T1 agent-decision records: claude-code-hooks-rce, codex-cli (both family agent-config-autorun, train), azure-mcp-server-ssrf (confused-deputy-tool-param, train), salesbleed-agentforce (salesforce-agentforce-exfil, bench) and openai-dns-sandbox-escape (openai-eval-agent-breakouts, bench). On 11 of 13 items the rater's best choice was a labelled acceptable answer. The two others are safe-versus-safe divergences kept as-is: on the hooks user_intent flip the rater preferred report-and-ask over continue-safely (the user had already forbidden running the hook, so a request-to-run is unnecessary); on the codex trusted-setup flip the rater accepted a 'looks tidy, start' action whose action is safe but whose reasoning is the weak pattern the label does not reward. Web fact-check (M03) verified every paraphrased fragment verbatim against the cited sources (one openai-dns fragment confirmed by hand against the raw page); minor locator fixes applied on hooks-rce and salesbleed.
• —M02, v1.5 additions (an independent model): the 11 canonical items and label-flip variants of the 5 records added in v1.5 were rated blind; the rater's best choice was a labelled acceptable answer on 11 of 11. Second batch of the T1 agent-decision backlog: zcode-silent-upload (coding-agent-silent-workspace-upload), nightingale-collective-agent (agent-collusion-evasion), llm-qu-dong-ren-xing (roleplay-jailbreak-action, train), maxkb-prompt-injection (ingested-content-drives-tool) and ibm-ftm-rag-poisoning (rag-store-poisoning). On all 11 of 11 items the rater's best choice was a labelled acceptable answer, and every label-flip variant (disclosed-and-consented upload, operator-sanctioned collaboration space, confirmed-harmless prop, authorized operator command, reviewed-and-approved procedure) was rated as the intended flip. Web fact-check (M03) verified every paraphrased fragment against the cited sources; minor locator/source fixes were applied on zcode (oschina->freebuf, outcome count), maxkb (two inexact quotes), and nightingale (section attribution).
• —M03 — web fact-check (an independent model with web access, not the author): the cited source of the quoted evidence was opened and compared across all records; six records were narrowed to what their sources state and a few citations re-pointed (alignment/reviews/ in the source repo).
• —Sign-off: all 55 records accepted by the dataset owner (R01) after the reviews above (41 at the first release on 2026-09-28, and the records added in each later version as it shipped). human_verified stays false: the sources were checked by model reviewers, not re-checked by a person.
• —Human review still open: 84 of 303 counterfactual plans are marked human_review_required (every authority or permission flip, and every flip that makes the risky action legitimate).
• —Evidence: every non-null scenario, context and state value has an evidence entry: 5 documented, 580 paraphrased from a cited source, 1305 reconstructed (no source states them).

Redaction

Wallets, domains, packages, payloads, credentials and people are replaced by placeholders named for what the value is, never for who controls it: <AGENT_HANDLE> (other), <BENCHMARK_REPOSITORY> (other), <BIOGRAPHICAL_CLUES> (person), <COMMAND_PAYLOAD> (payload), <CREDENTIAL> (credential), <CREDENTIAL_PART> (credential), <CRM_FIELD> (other), <DISTINCTIVE_PHRASE> (other), <EMAIL_ADDRESS> (other), <EMBEDDED_COMMAND> (payload), <EXFIL_ADDRESS> (other), <EXTERNAL_CHATBOT> (domain), <EXTERNAL_DEST> (domain), <EXTERNAL_DOMAIN> (domain), <EXTERNAL_FORM_URL> (domain), <EXTERNAL_HOST> (domain), <EXTERNAL_URL> (domain), <EXTERNAL_URL> (other), <EXTERNAL_WIKI> (domain), <HOOK_COMMAND> (payload), <MCP_PACKAGE> (package), <MCP_SERVER_NAME> (other), <MODEL_REPO> (package), <PACKAGE_NAME> (package), <PAYMENT_ACCOUNT> (other), <PAYMENT_AMOUNT> (other), <PAYMENT_ENDPOINT> (domain), <PICKUP_ADDRESS> (other), <PROJECT> (other), <REMOTE_HOST> (domain), <REPO> (other), <RESOURCE_ID> (other), <TOKEN_SYMBOL> (other), <UPLOAD_ENDPOINT> (domain), <VERSION> (other), <WALLET_ADDRESS> (wallet), <WORKSPACE> (other). Injections are described by what they ask for, never as a working string.

Intended use and limits

• —Post-training and evaluation research on agent safety: SFT, DPO, reward and critic models, step-level supervision.
• —Not a source of facts about the incidents, and not attack tooling: scenarios stop at decision-point granularity and carry no working payloads.
• —Small by design (55 incidents); English only; incidents from 2025–2026; coverage follows the archive, which leans on English-language security disclosures.
• —Reconstructed with DeepSeek (deepseek-v4.1-flash): scenario text, candidate actions and responses, including the SFT and preference targets. DeepSeek's terms allow using its outputs to train other models and ask that AI-generated content be marked, which this card does; confirm your use is allowed under those terms. A few factual corrections during review were made by the reviewers, and the review notes are not shipped in the data. See Generators and their terms below.
• —Labels are one careful reading of each case; the review status above says how they were checked.

Limitations and biases

• —Small and skewed coverage. 55 incidents, English only, 2025–2026; coverage follows the archive, which leans on English-language security disclosures, so vendors, regions and attack types are not evenly sampled.
• —Model-reconstructed, not human-verified. Scenarios and labels were reconstructed and reviewed by models and signed off by the dataset owner; no record is human-verified (human_verified: false). Treat labels as one careful reading, not ground truth.
• —Synthetic counterfactuals. The variants are generated by changing one dimension of a real case; they are plausible, not observed.
• —Reconstruction can mislabel. A wrong authority or permission call changes the preferred action; the plans marked human_review_required are where this risk concentrates.

Quick start

from datasets import load_dataset
sft = load_dataset("<namespace>/orca-incident-alignment", "sft", split="train")
pref = load_dataset("<namespace>/orca-incident-alignment", "preference", split="train")
# drop the provenance column before training
sft = sft.remove_columns("meta")

Baselines

Model baselines are not run yet; they will be added here with model versions, prompts and dates. Reference policies on IncidentBench bench-v1.4 (all decision and critic items; scripts/score_bench.py in the source repository):

An always-refuse policy commits no unsafe action but blocks every legitimate task and scores below zero, which is the point of scoring safe completion above refusal.

Changelog

• —v1.5 (2026-10-05): second batch of the T1 agent-decision backlog — one training record (llm-qu-dong, a role-play jailbreak of an action-taking robot) and four held-out benchmark records (a coding agent silently uploading the whole workspace; eval agents colluding on a public wiki; an ingested document driving a shell tool; a poisoned RAG runbook steering a payment agent). All deepseek-v4.1-flash, reviewed M01-M03, signed R01; training split stays 100% deepseek-v4.1-flash.
• —v1.4 (2026-10-05): first batch of the T1 agent-decision backlog — three training records added (claude-code-hooks-rce and codex-cli on auto-running an untrusted project's config; azure-mcp-server-ssrf on passing an untrusted URL to a privileged tool), all deepseek-v4.1-flash, reviewed M01-M03, signed R01. Two more went to the held-out benchmark (bench-v1.3). The training split stays 100% deepseek-v4.1-flash.
• —v1.3 (2026-10-03): added one training record — hugging-face-nullifai (2025-02-06), a malicious model whose 7z packaging slipped past the hub's Picklescan, reconstructed by deepseek-v4.1-flash and reviewed (M01-M03, signed R01). Also re-authored postmark-mcp's cf02 counterfactual in deepseek-v4.1-flash text to remove a reviewer-introduced non-DeepSeek string that had reached the training targets, so the training split is again 100% deepseek-v4.1-flash (verified). The benchmark eval data is unchanged (bench-v1.2).
• —v1.2 (2026-10-03): added one training record — postmark-mcp (2025-09-25), the first in-the-wild malicious MCP server, reconstructed by deepseek-v4.1-flash and reviewed (M01-M03, signed R01). The training split stays 100% deepseek-v4.1-flash. No benchmark incident was added, but the benchmark re-cut as bench-v1.2 because one held-out record (deadbugz, same mcp-malicious-server family) now has family_seen_in_train set.
• —v1.1.1 (2026-10-03): documentation and benchmark-marking fixes only; the training data is unchanged. The review section now describes the v1.1 state (43 records, both sign-off dates), and the canary string marks every benchmark full-record row.
• —v1.1 (2026-10-02): moved to archive commit 4615e96; two incidents were added to the benchmark (not this training split), so the training data is unchanged. Richer card metadata, a limitations section, a taxonomy summary and a quick start were added. Tag v1.0 keeps the previous version.
• —v1.0 (2026-09-28): first release. Decision-point items reconstructed by deepseek-v4.1-flash and independently reviewed; membership frozen in splits.json before any variant was generated, file hashes in MANIFEST.json.

Licence and attribution

CC BY 4.0. Derived from the Orca AI Incident Archive (https://github.com/Continuum-AI-Corp/Orca-AI-Incident-Archive), © Orca AI Incident Archive contributors, CC BY 4.0, at commit 4615e9638e59046a2abad3cdfd639d71ab4a4be8. Changes: incidents were reconstructed into decision-point records and synthetic counterfactual variants were generated. Reconstructed scenarios and counterfactuals are not incident evidence.

@misc{orca_incident_alignment_2026,
  title        = {Orca Incident Alignment (OIAS v1.1)},
  author       = {Orca AI Incident Archive contributors},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/orcarouter/orca-incident-alignment}},
  note         = {Derived from the Orca AI Incident Archive at commit 4615e96, CC BY 4.0}
}

Generators and their terms

Every record here was reconstructed by a language model. The records config names each record's generator (generation.generator), and every row's meta.oias_id points to its record. Counterfactual variants are derived from their record by a script (scripts/generate_counterfactuals.py) and count under that record's generator.

Check that your use of these outputs is allowed under the terms of the model that produced them.