datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
JSONSchemaBench
JSONSchemaBench
JSONSchemaBench is a benchmark of real-world JSON schemas designed to evaluate structured output generation for Large Language Models (LLMs). It contains approximately 10,000 JSON schemas, capturing diverse constraints and complexities.
import datasets
from datasets import load_dataset
def main():
# Inspect the available subsets of the datasetall_subsets = datasets.get_dataset_config_names("epfl-dlab/JSONSchemaBench")
print("Available subsets:"… See the full description on the dataset page: https://huggingface.co/datasets/epfl-dlab/JSONSchemaBench.json-schema-storeThis contains a set of schemas obtained via the JSON Schema Store catalog.
json-schema
JSON Schema Dataset
This dataset consists of a collection of JSON Schema documents collected from GitHub by searching using the Sourcegraph API.
Step 1: Find a list of JSON Schema paths
The Sourcegraph code search API is used to find files with a .json extension and containing {\n "$schema": "https://json-schema.org/".
This is somewhat restrictive, but still manages to find a large number of schemas.
pipenv run python slurp.py --outfile repos.csv
Step 2:… See the full description on the dataset page: https://huggingface.co/datasets/dataunitylab/json-schema.json-schema-instances-training-pool
JSON schema and instance training pool
Real JSON Schemas from the public collections named below, read at the pinned revisions given there,
each paired where possible with documents that satisfy it, laid out twice. Train on either layer or
on both.
pool.jsonl
Every source rewritten into one shape, 20004 rows, one JSON object per line, with these fields.
Field
What it holds
id
a row identifier unique within this file
prompt
the request a model would… See the full description on the dataset page: https://huggingface.co/datasets/Emulated-Inc/json-schema-instances-training-pool.schema-to-jsoncybersec-jsonschemabench-cloudtrail-v6
CybersecJSONSchemaBench CloudTrail Attack Reconstruction v6
A 100-problem long-context cybersecurity reasoning benchmark over real flAWS CloudTrail logs with synthetically injected MITRE ATT&CK attack chains.
Each task gives the model 600 real CloudTrail records (280-380K tokens of JSON) containing a single hidden multi-step attack chain. The model must produce a structured answer identifying the attacking principal, the MITRE ATT&CK technique, the per-phase records that constitute… See the full description on the dataset page: https://huggingface.co/datasets/achinta3/cybersec-jsonschemabench-cloudtrail-v6.jsonschemabench-maskbenchjson-schema-compliance-benchmark
JSON Schema Compliance Benchmark
A 500-example benchmark for evaluating whether language models can generate valid JSON conforming to provided schemas. Designed with strict contamination prevention to test generalization, not memorization.
Purpose
This is the primary Tier 1 evaluation metric for the Trellis SFT project. It measures a model's ability to produce structured output that passes jsonschema.validate() against novel, niche-domain schemas the model has never seen… See the full description on the dataset page: https://huggingface.co/datasets/mdonigian/json-schema-compliance-benchmark.cybersec-jsonschemabench-cloudtrail-v6
CybersecJSONSchemaBench CloudTrail Attack Reconstruction v6
A 100-problem long-context cybersecurity reasoning benchmark over real flAWS CloudTrail logs with synthetically injected MITRE ATT&CK attack chains.
Each task gives the model 600 real CloudTrail records (280-380K tokens of JSON) containing a single hidden multi-step attack chain. The model must produce a structured answer identifying the attacking principal, the MITRE ATT&CK technique, the per-phase records that… See the full description on the dataset page: https://huggingface.co/datasets/siddartha382/cybersec-jsonschemabench-cloudtrail-v6.fineweb-edu-json-schema-deduction
FineWeb-Edu json-schema-deduction
Source: fineweb-edu dataset.
Task: JSON schema deduction.
5,000 entries from fineweb-edu dataset
btw, every single key in the schema is unique. The model reasoning was high. The ontology went too deep haha.
It generated over 51,000 unique keys across 5,000 documents. it basically baked raw text directly into the structural keys.
however! json is 100% valid and correct so theres that
Columns are raw_text and schema
json-schema-store-reasoningjson-schema-store-rlcybersec-jsonschemabench-cloudtrail-objective-hard-v3
CybersecJSONSchemaBench CloudTrail Objective Hard v3
This is a 100-problem objective long-context cybersecurity reasoning subset built from the full flAWS CloudTrail corpus.
Each row contains an objective query prompt, a large CloudTrail JSONL context, and the JSON schema the answer must match. Gold answers are deterministic query results over the serialized slice and are not included in this public export.
Families
apigateway_restapi_event_profile: 10… See the full description on the dataset page: https://huggingface.co/datasets/achinta3/cybersec-jsonschemabench-cloudtrail-objective-hard-v3.cybersec-jsonschemabench
CybersecJSONSchemaBench Hard
This hard split is a JSONSchemaBench-style cybersecurity benchmark built from
normalized CloudTrail and Suricata EVE records. It replaces anchored lookup
questions with unanchored, deterministic multi-hop reasoning programs over
large nested JSONL slices.
Each row includes:
unique_id
json_schema
prompt
input_jsonl
ground_truth_json
reasoning_family
candidate_count
distractor_count
Current Version
benchmark version: 1.0.0-hard
total… See the full description on the dataset page: https://huggingface.co/datasets/achinta3/cybersec-jsonschemabench.cybersec-jsonschemabench-cloudtrail-objective-natural-hard-v5
CybersecJSONSchemaBench CloudTrail Objective Natural Hard v5
This is a 100-problem natural-prompt long-context cybersecurity reasoning subset built from the full flAWS CloudTrail corpus.
Each row contains a natural analyst-style question, a large CloudTrail JSONL context, and the JSON schema the answer must match. Gold answers are deterministic hidden-oracle results over the serialized slice and are not included in this public export.
Families
actor_recon_to_change: 22… See the full description on the dataset page: https://huggingface.co/datasets/achinta3/cybersec-jsonschemabench-cloudtrail-objective-natural-hard-v5.json-ld-schema-meta-tag-extractor-sample-data
JSON-LD Schema & Meta Tag Extractor
Extract JSON-LD/Schema.org structured data, Meta tags, OpenGraph and Twitter Cards from any URL. Get page title + meta description with a clean JSON output for SEO audits, validation, competitor research and AI datasets. Proxy-ready for large crawls.
What the actor scrapes
🧩 JSON-LD Schema & Meta Tag Extractor — Scrape Schema.org, OpenGraph & Meta Tags Extract structured data and SEO metadata from any webpage in seconds. This… See the full description on the dataset page: https://huggingface.co/datasets/logiover/json-ld-schema-meta-tag-extractor-sample-data.JSONSchemaBench__subsampledjsonschema-sftcybersec-jsonschemabench-cloudtrail-hard-v2-400
CybersecJSONSchemaBench CloudTrail Hard v2 400
This is a 100-problem synthetic long-context cybersecurity reasoning subset built from the full flAWS CloudTrail corpus.
The benchmark asks models to return JSON matching the provided answer schema. Each row contains a short analyst request, a large CloudTrail JSONL context, and hidden deterministic evaluation metadata.
This variant uses shorter 400-record contexts than the full CloudTrail Hard v2 export so direct API evaluation is less… See the full description on the dataset page: https://huggingface.co/datasets/achinta3/cybersec-jsonschemabench-cloudtrail-hard-v2-400.jsonschema-goldcybersec-jsonschemabench-cloudtrail-natural-hard-v4
CybersecJSONSchemaBench CloudTrail Natural Hard v4
This is a 100-problem natural-prompt long-context cybersecurity reasoning subset built from the full flAWS CloudTrail corpus.
Each row contains a natural analyst-style question, a large CloudTrail JSONL context, and the JSON schema the answer must match. Gold answers are deterministic hidden-oracle results over the serialized slice and are not included in this public export.
Families
actor_recon_to_change: 20… See the full description on the dataset page: https://huggingface.co/datasets/achinta3/cybersec-jsonschemabench-cloudtrail-natural-hard-v4.json-schema-definitionstext_to_schema.org_json-ldjson-schema-descriptionsjson-schema-keywordsagentic_dpo_json_schema_v1poc-llama-cpp-json-schema-stack-overflow
PoC: Stack Overflow in llama.cpp JSON Schema to Grammar Converter — Remote DoS
Target: ggml-org/llama.cpp
File: common/json-schema-to-grammar.cpp line 419
Type: CWE-674 — Uncontrolled Recursion
CVSS: 7.5 (AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H)
Bounty program: huntr.com OSV
What This Repo Contains
File
Purpose
poc_regex_crash.py
Crashes llama-server via nested regex in json_schema — primary PoC
poc_anyof_crash.py
Crashes llama-server via nested anyOf schema —… See the full description on the dataset page: https://huggingface.co/datasets/NOTTIBOI1337/poc-llama-cpp-json-schema-stack-overflow.