vishwr/claim_drafter

Claim Drafter — datasets

Training and evaluation data for **`vishwr/claim_drafter`**, a LoRA adapter that drafts US patent claims from a plain-English invention disclosure.

Two datasets are included:

• —`sft` — 9,662 train / 1,314 validation examples in conversational (messages) format. Targets are as-GRANTED claims fetched per patent number (not the as-filed claims that ship with HUPD), and prompts are stripped of the patent summary so the model must draft rather than reformat.
• —`dpo` — 5,614 examiner-labelled preference pairs. Each row holds a comparison (a prompt plus two candidate claim sets) and a label of the preferred side. The preference is real: the allowed (as-granted) claims are chosen over the refused (as-filed) claims for the same application — no LLM judge and no manual annotation.

Layout

sft/train.jsonl              9,662 rows  — conversational messages
sft/val.jsonl                1,314 rows  — full held-out evaluation pool
sft/validation/*.jsonl       11 slices   — disjoint subsets of the pool (per-domain, hard-low-leakage, ...)
sft/manifest.jsonl                       — provenance + exact token counts per example
dpo/comparisons.jsonl        5,614 rows  — examiner-labelled preference pairs
dpo/manifest.jsonl                       — provenance

Provenance for every example (patent number, IPC class, domain, split, token counts) lives in the manifest.jsonl files, making the corpus reproducible.

Patent text is US government work and not subject to copyright. Applications come from the Harvard USPTO Patent Dataset; granted and as-filed claims were fetched from Google Patents by the pipeline in the model repository. Code is MIT licensed.

Original dataset notes

Claim Drafter Dataset v3 — final

Train on this. 9,662 training examples and a 1,314-example evaluation pool sliced 11 ways, spanning all six target industry domains.

Contents

Domain mix (train)

No single domain exceeds 30% (MAX_DOMAIN_SHARE), so electronics does not swamp the mix. Domain is assigned from the IPC code by domains.py, which covers 99.96% of the corpus.

Validation slices

All 11 are disjoint from train.jsonl (verified: zero overlap) and are subsets of the same held-out pool, so they can be compared against each other.

Caveat on the cohort slices: the 2014 material was pulled specifically to boost the thin domains, so filing year and domain are correlated. A gap between the two cohorts is not clean evidence of temporal drift.

Format

Verified against the Tinker cookbook: FromConversationFileBuilder requires exactly this — one JSON object per line with a "messages" key, and it raises DataFormatError on anything else. Do not add trainable keys (they are only valid with TrainOnWhat.CUSTOMIZED and otherwise trip an assertion).

{"messages": [
  {"role": "system", "content": "You are an expert US patent attorney. Draft a set of independent and dependent claims based on the provided invention description. Use correct USPTO formatting."},
  {"role": "user", "content": "Title: ...\n\nTechnical Field and Background:\n...\n\nInvention Disclosure:\n..."},
  {"role": "assistant", "content": "1. A system comprising:\na processor ...;\nand a memory ...\n\n2. The system of claim 1, wherein ..."}
]}

Verified properties

role sequence errors:        0
claim numbering errors:      0
invalid forward references:  0
claim-1 8-gram leakage:      median 3.7%   (v1 was 46.2%)
claims per example:          min 3  median 16  max 30
real max tokens (Qwen3.5):   7,824  -- 0 examples exceed the 8,192 cap
train/val patent overlap:    0

The token cap matters: the cookbook silently right-truncates over-length sequences, which would eat the end of the claim set with no warning. Every example is pre-checked against the real Qwen tokenizer so this cannot happen.

Model selection

Filtering Tinker's 24 models by what this task actually needs:

Type — use Hybrid, with thinking disabled.

• —Reasoning (gpt-oss-*) is wrong here: those models emit reasoning traces by design, and every target in this dataset is pure claim text with none. You would fight the architecture in training and waste compute on unwanted thinking tokens at inference. They are also outside the Qwen/Llama families that hyperparam_utils.get_lr() supports, so the LR helper raises NotImplementedError.
• —Base (Qwen3.5-9B-Base, Qwen3.5-35B-A3B-Base) is tempting — no instruction-tuned prior to overwrite — but base models have no chat template, so this system/user/assistant JSONL does not map cleanly onto the renderer. More decisively, real users type messy, varied disclosures; 9,662 examples can teach an output format but cannot rebuild instruction-following. Base wins only when inputs are perfectly uniform.
• —Vision adds no overhead here and is unused. Worth remembering that patent drawings are genuinely informative for claim drafting, so a later version could feed figures to a vision model without changing platform.

Architecture — a real trade-off, not a clear win.

• —Dense keeps every parameter active on each token, which favours the long-range structural consistency claims demand: antecedent basis ("a processor" before "the processor") has to hold across a 16–30 claim set.
• —MoE packs far more total knowledge per active parameter — relevant because this dataset spans six technical domains. Qwen3.6-35B-A3B is newer than Qwen3.5-9B, with 35B total parameters, but only ~3B are active per token.

Default below is dense. If you want evidence rather than a guess, train both for 2 epochs and compare on the per-domain and hard_low_leakage slices — that is what the multi-slice validation pool is for.

Training config

Token totals: 18,344,477 per epoch (6,524,712 prompt + 11,819,765 completion).

Known limits

• —No design or plant patents. HUPD contains utility applications only. Design claims are a single ornamental sentence ("The ornamental design for X, as shown and described") and plant patents likewise — a different task needing a different source, and arguably not worth model capacity.
• —Chemistry is partly unlearnable from text. 22% of pharma and 59% of biotech claim sets were dropped because they reference drawn structures ("a compound of formula (I)") or sequence listings ("SEQ ID NO: 1") that exist only as images or external files. Expect the model to be weakest on chemical composition claims; it is trained on the method and device claims that survive.
• —Still not a deployment test. Every prompt here is patent-office prose. Real users write rough disclosures. Hand-write a dozen and check those separately.