anonymous-md/EDGAR_FILINGS_DATASET

SFD: SEC Filings Dataset (v1)

SFD-v1 is an open, layout-faithful reconstruction of U.S. Securities and Exchange Commission (SEC) EDGAR filings into token-efficient MultiMarkdown (MMD), targeted at long-context language modeling, financial reasoning, document understanding, and evaluation.

This release covers filings from January 2022 through June 2025 (~3.4M filings), produced by the SFD parser described in:

The SEC Filings Dataset: Reconstructing U.S. Corporate and Financial Disclosures into Layout-Faithful and Token-Efficient Pretraining Data. Anonymous Authors (NeurIPS 2026 Evaluations & Datasets Track, under review).

The full SFD corpus is estimated at ~500B tokens across ~18.4M filings (1994–present); this release is a public snapshot focused on the most recent four-and-a-half years.

Key facts

• —Format: MultiMarkdown — preserves merged-cell tables (|| colspan, ^^ rowspan), indentation, and visual hierarchy that standard text extraction destroys.
• —Source formats handled: HTML (\~62%), XML (\~18%), plaintext (\~18%), SGML (\~2%), PDF-via-OCR (\~1%) — see paper §3.
• —Filings: ~3.4M parsed .md documents across 350+ filing types (10-K, 10-Q, 8-K, Form 4, N-PORT, 13F, 485BPOS, ABS-EE, …).
• —License: CC-BY-NC-4.0 (parsed). Underlying SEC filings are U.S. Government public domain.
• —Storage: Parquet shards with zstd-15 compression, one shard per (year, month). Token counts use the Qwen3-1.7B tokenizer. The headline ~152B figure counts the full released parsed_md field, including the prepended SEC/SGML filing metadata header. The regulatory body content alone is ~111B tokens; the difference is metadata such as CIK, accession, form type, filing date, period of report, and related SEC header fields.

Schema

Each row is one parsed filing.

Loading

from datasets import load_dataset

ds = load_dataset("anonymous-md/EDGAR_FILINGS_DATASET", split="train", streaming=True)
for row in ds.take(1):
    print(row["accession"], row["year"], row["month"], len(row["parsed_md"]))

For a full local materialization (~50 GB on disk):

ds = load_dataset("anonymous-md/EDGAR_FILINGS_DATASET", split="train")
ds = ds.filter(lambda r: r["year"] == 2024)  # or by month, source_format, etc.

To recover the canonical SEC EDGAR URL for any row:

acc = row["accession"]                              # e.g. "0000320193-24-000123"
acc_clean = acc.replace("-", "")
url = f"https://www.sec.gov/Archives/edgar/data/{int(acc.split('-')[0])}/{acc_clean}/"

Sample subset (for reviewer inspection)

Because the full corpus exceeds 4 GB, a small sample is provided at sample/sfd-v1-sample.parquet to allow quick inspection of data quality without materializing the full dataset.

• —Path: sample/sfd-v1-sample.parquet
• —Sampling methodology: stratified by year — first/last representative month of each year in the release (2022-01, 2023-01, 2024-01, 2025-01, 2025-06), with the first 25 rows from each, sorted by accession. This surfaces a mix of source formats (HTML / XML / SGML / PDF-OCR) and form types (Form 4, 10-Q, 8-K, NPORT-P, etc.) representative of the full corpus.
• —Loading: pq.read_table("sample/sfd-v1-sample.parquet") with the same schema as the full dataset.

Methodology summary

SFD treats filings as 2-D rendered grids rather than DOM trees:

• —HTML: Reconstructs the visual coordinate system; collapses the "Three-Column Hack" (\$ / value / closing-paren split into separate cells); coalesces fragmented multi-row headers using border-* and margin-* cues; preserves indentation hierarchy by binning CSS units into discrete   levels.
• —XML: Routes 33 specialized schemas (Forms 3/4/5, 13F, N-PORT, N-CEN, etc.) through schema-aware emitters that reconstruct human-readable disclosures from field hierarchies.
• —Plaintext / SGML: Wraps fixed-width legacy filings in code fences to preserve column alignment; collapses 3+ blank lines to 2.
• —PDFs: Run through Mistral OCR 3 in 10-page batches; HTML table fragments converted to MMD; near-blank pages skipped via pixel-variance filter.

Every row is prepended with <SEC-HEADER>-derived metadata (CIK, SIC, filing type, period of report, etc.) so each filing is self-contained.

See paper §3 for full details.

Companion benchmarks

Two evaluation benchmarks are derived from SFD and reported alongside this dataset:

• —EDGAR-OCR — 300 hand-selected SEC tables, synthetically perturbed for contamination resistance, scored by adjusted recall over (content × placement × inline formatting).
• —EDGAR-Forecast — 50 companies × 5 numeric targets each (250 total) drawn from 2026 10-Q filings, evaluated agentically with prior 5-year filing history visible.

Released separately when scoring is finalized.

Citation

@inproceedings{anonymous2026sfd,
  title={The SEC Filings Dataset: Reconstructing U.S. Corporate and Financial Disclosures into Layout-Faithful and Token-Efficient Pretraining Data},
  author={Anonymous Authors},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS), Evaluations \& Datasets Track},
  year={2026}
}

License & terms

• —Parsed corpus (this release): CC BY-NC 4.0 — non-commercial use only with attribution. Derivative works must keep this notice.
• —Underlying raw filings: U.S. Government public domain, available canonically from the SEC EDGAR system at <https://www.sec.gov/edgar>. Redistribution of raw filings under CC BY 4.0 attribution-style metadata is permitted.
• —Mistral OCR outputs (the small PDF subset, has_ocr == True) are subject to the Mistral AI Terms of Service at the time of generation; downstream redistribution within this CC BY-NC 4.0 corpus is permitted.

Ethics, privacy, limitations

• —All filings are public regulatory disclosures with no expectation of privacy.
• —The dataset preserves filer-supplied content verbatim; SFD does not correct factual or accounting errors in the source filings.
• —The MMD reconstruction is high-fidelity but not perfect; estimated ~99% structural/semantic accuracy. A small minority of filings (notably highly visual exhibits with low OCR-recoverable content) may have degraded representation.
• —Token counts reflect the Qwen3-1.7B tokenizer; other tokenizers will differ.

Provenance & versions

• —v1 (this release): 2022-01 → 2025-06, parsed by SFD pipeline rev sec_parser.
• —Future releases will extend coverage to 1994–2021 and incrementally to 2025-07+.

Contact

• —Anonymous Authors