D2I-CUHK-Shenzhen/FormStruct-Bench

FormStruct-Bench

Dataset Description

FormStruct-Bench is a multilingual benchmark for extracting the semantic and spatial structure of forms from document images. The repository combines a 7,000-page main benchmark, a controlled visual-degradation set, and template-level layout annotations. It supports evaluation of vision-language models and document AI systems on hierarchical key-value extraction, document structure recovery, region localization, table and line-item understanding, and selection-widget interpretation.

The main task takes a single form image as input and predicts its complete hierarchical answer tree. Template annotations provide complementary pixel-space regions and layout metadata for structure-aware evaluation.

Repository Contents

The repository currently retains directories for 80 templates, but the main benchmark contains 100 filled instances for each of 70 canonical templates (7,000 pages). The 10 additional directories correspond to the redundant templates listed below and are excluded from the official benchmark and all splits. The augmentation data contains degraded variants of selected source pages. The template annotations contain annotations for all 70 main-benchmark templates plus 10 redundant templates. The 70 templates in datasets/ define the canonical dataset scope. The extra annotation files are retained only as redundant data and are not part of the formal benchmark.

Main Benchmark

Statistics

Each main sample contains the same answer in three representations:

• —answer.json: machine-readable hierarchical key-value data;
• —answer.md: a human-readable nested list; and
• —answer.html: a browser-renderable hierarchical view.

Language Distribution

The filename prefix zn is retained from the original data and denotes Chinese; it is not an ISO 639 language code. zn_en denotes bilingual Chinese-English templates.

Directory Structure

datasets/
  {template_name}/
    {instance_id}/
      {template_name}-{instance_id}.png
      answer.json
      answer.md
      answer.html

Example:

datasets/en_1/01/
  en_1-01.png
  answer.json
  answer.md
  answer.html

Answer Format

answer.json stores the semantic answer as a nested JSON object. Internal objects represent sections or semantic groups, while leaf values contain the text associated with individual form fields.

{
  "PRODUCT SPECIFICATION": {
    "Brand": "Marlboro",
    "Company": "Philip Morris International",
    "Country": "United States"
  },
  "Prepared by": "Laura Bennett",
  "Date": "28/02/2024"
}

The schema varies by template and can also vary across instances of the same template. Systems should therefore predict the full hierarchy instead of assuming one fixed global field schema.

Template Annotations

template_annotation/ contains 80 standalone JSON files with one reviewed template annotation per file. Exactly 70 files correspond to the canonical templates in the main dataset. The following 10 files are redundant data and must be excluded from official dataset statistics, splits, training scope, and evaluation:

de_3.json
de_4.json
es_4.json
ja_23.json
ja_24.json
ja_25.json
ja_26.json
ja_27.json
ja_28.json
zn_12.json

The annotation directory as a whole covers Arabic, German, English, Spanish, Japanese, Portuguese, Chinese, and bilingual Chinese-English forms. Unless a separate exploratory use explicitly requires the redundant files, consumers should join annotations against the 70 template names present in datasets/.

Annotation Statistics

The structural statistics below use only the 70 canonical benchmark annotations and exclude the 10 redundant files.

Annotation Schema

Each file contains:

• —id: template identifier;
• —img: source-image reference from the annotation environment;
• —original_width, original_height: page dimensions in pixels;
• —fields: recursive field annotations;
• —semantic_key: normalized semantic field name;
• —original_label: label in the source document language;
• —bbox: pixel-space box in [x_min, y_min, x_max, y_max] format;
• —data_type: types such as text, number, checkbox, and checkbox_multi;
• —value or values: one or more associated value or option regions;
• —keys: nested child fields; and
• —metadata: structural, visual, domain, language, difficulty, section, region, table-region, and line-item-group metadata.

Shortened example:

{
  "id": 182,
  "img": "/data/upload/2/49a188a5-en_1.jpg",
  "original_width": 864,
  "original_height": 1232,
  "fields": [
    {
      "semantic_key": "Brand",
      "original_label": "Brand",
      "bbox": [84, 192, 129, 208],
      "data_type": "text",
      "value": {
        "bbox": [142, 190, 335, 207],
        "data_type": "text"
      }
    }
  ],
  "metadata": {
    "language": "English",
    "domain": "business",
    "layout_structure": {
      "page_bbox": [0, 0, 864, 1232]
    }
  }
}

The img entries are internal annotation-system paths, not downloadable URLs. Use the JSON filenames to associate template annotations with matching template names in the main benchmark.

Visual-Degradation Data

dataset-augment/ supports controlled robustness evaluation. It contains 76 source pages with:

• —1,140 factorial variants from five degradation families at three severity levels; and
• —76 additional combined degradation images.

The five factorial degradation families are:

Factorial variants use low, medium, and high severity levels. Their directory structure is:

dataset-augment/
  {template_name}/
    {source_instance_id}/
      {variant}/
        {level}/
          {template_name}-{source_instance_id}.png
          answer.json
          augment_meta.json

Each augment_meta.json records the deterministic seed, transformation parameters, source and output sizes, before/after image metrics, and pixel difference statistics. The 76 top-level combined images have image and augmentation metadata but do not include an answer.json sidecar. Evaluation code should pair only samples that have the required clean source and answer.

Loading the Data

The repository uses a task-specific directory structure rather than a single tabular file. A minimal Python loader for the main benchmark is:

import json
from pathlib import Path

root = Path("datasets")
samples = []

for image_path in sorted(root.glob("*/*/*.png")):
    answer_path = image_path.parent / "answer.json"
    if not answer_path.is_file():
        continue
    samples.append(
        {
            "template": image_path.parent.parent.name,
            "instance_id": image_path.parent.name,
            "image_path": str(image_path),
            "answer": json.loads(answer_path.read_text(encoding="utf-8")),
        }
    )

print(len(samples))  # 7000

Template annotations can be loaded independently:

annotation_root = Path("template_annotation")
annotations = {
    path.stem: json.loads(path.read_text(encoding="utf-8"))
    for path in sorted(annotation_root.glob("*.json"))
}

Tasks and Evaluation

The repository is suitable for:

• —image-to-hierarchical-JSON extraction;
• —form key-value extraction with full semantic paths;
• —document schema and hierarchy recovery;
• —region and line-item-group localization;
• —table, widget, and key-value relation analysis;
• —multilingual and right-to-left form understanding; and
• —robustness evaluation under controlled visual degradation.

Relevant evaluation families include whole-page exact match, normalized schema tree-edit similarity, normalized value edit similarity, path-sensitive field accuracy, region F1 at an IoU threshold, line-item-group F1, and widget answer accuracy. Evaluation code and exact metric definitions are maintained in the associated FormStruct-Bench project repository.

Splits

FormStruct-Bench defines one official, fixed, template-disjoint split generated with seed 42. The split used in the paper is:

The authoritative release files are:

• —splits/template_stratified_seed42/train_index.jsonl;
• —splits/template_stratified_seed42/val_index.jsonl;
• —splits/template_stratified_seed42/test_index.jsonl.

The JSONL paths use the public repository's normalized template directory names and resolve from the repository root. The 10 redundant templates are excluded from every split. Do not randomly split pages: instances from the same template share substantial visual and semantic structure and would leak across partitions. Results should report the dataset revision and use these manifests.

Privacy and Responsible Use

The forms include identity-like names and values as well as fields associated with potentially sensitive domains. The current release records do not establish that every value is synthetic or anonymized. This is tracked per template in provenance/template_rights.csv; an unverified row is not privacy-cleared. Report suspected personal or sensitive information through the repository's Community tab and identify the template and instance so maintainers can remove or quarantine it.

FormStruct-Bench is intended for document AI research and system evaluation. It is not intended for identity verification, eligibility decisions, surveillance, or automated decisions that affect individuals.

License and Rights

This dataset has no blanket Apache-2.0 license. Hugging Face metadata uses license: other because rights are mixed and source-specific:

• —Apache-2.0 covers only code in the associated software repository.
• —CC BY-NC-SA 4.0 applies only to files whose per-template audit row cites evidence for that license; its attribution, non-commercial, and share-alike conditions remain in force.
• —Base images and document designs under other terms remain subject to those source terms.
• —Answers and annotations may be derivative of the underlying form, and augmented images inherit restrictions from their clean source image.

DATA_LICENSE.md defines the component-level policy and provenance/template_rights.csv is the controlling per-template record. Access to repository files does not itself grant copyright, privacy, publicity, trademark, database, or other rights.