1fanj/Chartographer

Chartographer

Chartographer is a chart reasoning dataset for evaluating whether vision-language models answer chart questions through visual reasoning rather than shortcuts or prior familiarity with a chart. Each chart-question family contains an upstream original chart, a reconstructed chart, and ten seed-controlled counterfactual variants with the same Chartographer chart_id and question_id.

More details on the construction pipeline and evaluation protocol are available in the paper.

<img src="assets/pipeline_overview.png" alt="Chartographer pipeline overview" width="100%">

Fields

Rows are stored in data/*.jsonl.

• —chart_id: chart identifier, shared by the original chart, reconstruction, and seed variants of the same chart.
• —question_id: question identifier, shared by all variants for the same question.
• —variant: chart variant, one of original, reconstruction, or seed_0 through seed_9.
• —image: relative path to the released chart image; empty for original rows because upstream original images are not included.
• —question: question asked about the chart image.
• —answer: reference answer for the row's chart variant.
• —chart_data: relative path to the released chart data JSON; empty for original rows.
• —source_dataset_split: source dataset split identifier for this Chartographer row, combining the upstream dataset and split used in the release.
• —source_row_index: zero-based row index in the upstream dataset split identified by source_dataset_split.

Intended Use

Chartographer is intended for chart visual question answering evaluation, especially comparisons between original charts, reconstructed charts, and controlled counterfactual chart variants.

License Notes

Chartographer is released under CC BY-SA 4.0 where permitted by the source terms. Upstream original images are not included and may be governed by separate licenses. To recover upstream original images and source records, use source_row_index with the upstream Hugging Face dataset and split listed in metadata/dataset_info.json.

Chartographer keeps its own evaluated question and answer fields, which may differ from the exact QA text stored in the upstream Hugging Face row.

Citation

@inproceedings{jiang2026chartographer,
  title = {Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models},
  author = {Jiang, Yifan and Hwang, Dae Yon and Cresswell, Jesse C. and Shi, Freda},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year = {2026},
  publisher = {Association for Computational Linguistics},
  url = {https://arxiv.org/abs/2605.27311}
}

Loading

from datasets import load_dataset

ds = load_dataset("json", data_files={
    "chartmuseum_dev": "data/chartmuseum_dev.jsonl",
    "chartqa_val": "data/chartqa_val.jsonl",
    "charxiv_validation": "data/charxiv_validation.jsonl",
})