Team Ai
Datasetpublic

justicedao/open-us-law-sparse-graphrag

State laws snapshot sparse GraphRAG CID-keyed retrieval release in the same thin-client layout as Publicus/skillcenter-ir: Zstandard Parquet shards of at most 4,096 rows entry_cid as the canonical content identity; document_index is a compact pointer compact BM25 term-range, vector-centroid, corpus, and adjacency routing indexes thenlper/gte-small 384-d L2 vectors, cosine-sorted inside centroid shards queries fetch the manifest, routing indexes, and only the routed shards This… See the full description on the dataset page: https://huggingface.co/datasets/justicedao/open-us-law-sparse-graphrag.

sourceHugging Faceotherupdated 14d agoView on Hugging Face
0likes71downloads
Dataset Card

State laws snapshot sparse GraphRAG

CID-keyed retrieval release in the same thin-client layout as `Publicus/skillcenter-ir`:

  • —Zstandard Parquet shards of at most 4,096 rows
  • —entry_cid as the canonical content identity; document_index is a compact pointer
  • —compact BM25 term-range, vector-centroid, corpus, and adjacency routing indexes
  • —thenlper/gte-small 384-d L2 vectors, cosine-sorted inside centroid shards
  • —queries fetch the manifest, routing indexes, and only the routed shards

This is a snapshot/research retrieval aid. It is not a current-bundle, does not close LCR-084, and does not replace justicedao/ipfs_state_laws. Official state publications remain the authority. Remaining jurisdictions NY, WI, TN, AR, NH, MS, and GA include Vaquill Open US Law snapshot rows because this host cannot complete official live fetches through Cloudflare/WAF without residential proxies.

Remote retrieval without a full download

bash
python scripts/query_open_us_law_hf.py \
  --local-root . --fixture-mode --json --trace \
  bm25 "public records" --top-k 10

Vector retrieval probes compact centroids and exact-scores only selected shards.

NY, WI, TN, AR, NH, MS, and GA come from local /tmp/tmp_laws dumps. Other jurisdictions use the best local LCR JSON-LD available. Partial JSON-LD coverage is recorded; it does not become an official source receipt.

Open US Law Sparse GraphRAG

Dataset repository: justicedao/open-us-law-sparse-graphrag Source bucket: justicedao/open-us-law-bucket

Release profile

  • —Profile: open-us-law-sparse-graphrag/v1
  • —Pinned source revision: 7c1e4a90b2d65f83e0a91c4d6b7e8f0123456789abcdef0123456789abcdef01
  • —Embedding model: thenlper/gte-small @ 17e1f347d17fe144873b1201da91788898c639cd
  • —Vector space: gte-small@17e1f347d17fe144873b1201da91788898c639cd:d384:pool=mean:norm=l2
  • —Primary key (default config): entry_cid
  • —Default configuration: state_statutes_exact_51 (Viewer-safe exact-51)
  • —Required jurisdictions: 51 (50 states plus DC)

Dataset configurations

The default configuration is Viewer-safe exact-51 state and DC statutes only. Recovery JSON, quarantine JSON, federal US Code, Puerto Rico, constitutions, and historical rows are advertised as separate named configs and never contaminate the default Dataset Viewer schema or the exact-51 gate.

  • —state_statutes_exact_51 (default exact-51) — primary key entry_cid
  • —Viewer-safe default exact-51 configuration.
  • —Current official statutes for exactly the 50 states plus DC.
  • —Excludes recovery, quarantine, federal US Code, Puerto Rico, constitutions, and historical rows.
  • —federal_uscode (named non-default) — primary key entry_cid
  • —Federal United States Code rows. Useful corpus; never default exact-51.
  • —Must not be the default Dataset Viewer config.
  • —puerto_rico (named non-default) — primary key entry_cid
  • —Puerto Rico statutory rows. Explicit non-default configuration.
  • —Must not be the default Dataset Viewer config.
  • —constitutions (named non-default) — primary key entry_cid
  • —Federal, state, DC, and territorial constitution rows.
  • —Must not be the default Dataset Viewer config.
  • —historical (named non-default) — primary key entry_cid
  • —Superseded, repealed, or otherwise non-current statute versions.
  • —Must not be the default Dataset Viewer config.
  • —recovery (recovery) — primary key recovery_id
  • —Recovery records that cannot enter canonical default counts.
  • —Never included in the default config or exact-51 gate.
  • —quarantine (quarantine) — primary key recovery_id
  • —Quarantined or rejected rows excluded from the Viewer default split.
  • —Never included in the default config or exact-51 gate.

Artifact layout

text
README.md
manifest.json
release_metadata.json
dataset_configs.json
dataset_infos.json
data/corpus/part-*.parquet
data/bm25/documents/part-*.parquet
data/bm25/postings/part-*.parquet
data/vectors/centroid-*-part-*.parquet
data/vectors/centroids/part-*.parquet
data/vectors/locator/part-*.parquet
data/graph/nodes/part-*.parquet
data/graph/edges/part-*.parquet
data/graph/adjacency/out/part-*.parquet
data/graph/adjacency/in/part-*.parquet
indexes/*
receipts/source/*
reports/admission.json
reports/quality.json
reports/reproducibility.json
reports/lineage.json   # verbose lineage (not control plane)
recovery/...          # recovery config only
quarantine/...        # quarantine config only
configs/federal_uscode/...
configs/puerto_rico/...
configs/constitutions/...
configs/historical/...

Control plane vs verbose lineage

The control plane consists of manifest.json, release_metadata.json, routing indexes, source-receipt descriptors, and compact admission/quality/reproducibility reports. Verbose per-row source lineage lives only in reports/lineage.json and is not mixed into Dataset Viewer configs or the release control plane.

Route bounds

  • —At most 4096 rows per physical shard.
  • —At most 4096 BM25 posting pointers per cell.
  • —At most 4096 adjacency pointers per page.
  • —At most 8192 vectors and 2 shards per centroid.
  • —Model token ceiling is 512; it is not a shard bound.

Currentness disclaimer

This release is a research retrieval aid. Observation times, edition pins, and acquisition receipts are not legal-currentness claims. Official state and District of Columbia publications remain the authority.

Limitations

  • —Retrieval output is a research aid and is not a substitute for official state or District of Columbia publications.
  • —Acquisition and publication timestamps are not legal-currentness claims.
  • —Recovery and quarantine rows are excluded from the default exact-51 config and from corpus/BM25/vector/graph counts until admitted.
  • —Federal US Code, Puerto Rico, constitutions, and historical rows are explicit non-default configurations and cannot satisfy the exact-51 gate.

Integrity

Every artifact is descriptor-bound in manifest.json with relative path, media type, row count, byte count, SHA-256, schema identifier, and optional key range. The manifest also binds corpus, BM25, vectors, centroids, the vector locator, the legal graph, two-way adjacency, recovery, source receipts, the pinned model revision, and route bounds.

Producer: open_us_law_hf_release.py (OUL-032 / OUL-G040).

Formalization prep

formalization_prep is an additive statute-text preparation for a later formal-logic pass. It does not replace data/corpus, the BM25 postings, the vectors, or the graph. entry_cid still points at the published section. Cleaned text drops replacement characters and joins a hyphenated line break only when the next letter is lowercase. No statutory wording is added. Rows marked formalization_ready are operative sections. Reserved numbers, source-link stubs, catchlines, contents lines, source notes, and archive wrappers are labeled and left out of that set. Long sections are sliced on markers already in the text. This config is not the default and does not satisfy the exact-51 gate.