llm-jp/HakushoBench

<div align="center" style="line-height: 1;"> <h1>HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Papers</h1>

| <a href="https://huggingface.co/datasets/llm-jp/HakushoBench/" target="blank">🤗 Dataset</a> &nbsp;| <a href="https://arxiv.org/abs/2606.01132" target="blank">📄 Paper</a> &nbsp;| <a href="https://github.com/llm-jp/simple-evals-mm" target="_blank">🧑‍💻 Code</a> &nbsp;|

<br/>

<img src="./showcasebytype.png" width="80%"/> </div>

Overview

HakushoBench is a challenging Japanese chart and table VQA benchmark built from 33 governmental white papers. HakushoBench contains 2,053 images spanning over 10 image types, with manually annotated QA pairs, designed to assess deep and holistic understanding of charts and tables, rather than local visual cues alone.

Construction Pipeline of HakushoBench

HakushoBench is constructed through the following three stages:

1. 1.Chart & Table Image Collection: Chart and table images are collected from 33 Japanese governmental white papers and filtered to 5,903 candidate images.
2. 2.QA Annotation: Annotators create one challenging QA pair for each image.
3. 3.QA Verification: All annotations undergo independent verification, resulting in a final benchmark of 2,053 manually validated VQA pairs.

<figure> <img src="./HakushoBench_pipeline.svg" alt="Construction pipeline of HakushoBench" style="width: 80%"> <figcaption>Construction pipeline of HakushoBench.</figcaption> </figure>

Domain Coverage

HakushoBench is constructed from Japanese governmental white papers spanning diverse domains, including security, economics, society. This broad coverage enables comprehensive evaluation of chart and table understanding across a wide range of real-world topics.

<figure> <img src="./topic_distribution.svg" alt="Topic distribution" style="width: 40%"> <figcaption>Distribution of QA pairs in HakushoBench.</figcaption> </figure>

Evaluation

You can evaluate models on HakushoBench using simple-evals-mm

The figure below presents evaluation results on HakushoBench. Despite recent advances in multimodal models, a substantial performance gap remains between the strongest proprietary model, Gemini 3 Pro, and the best open-weight model, Qwen3-VL-8B, suggesting considerable room for improvement in open-weight models on chart and table understanding. <figure> <img src="./hakushobench_results.svg" alt="evaluation results" style="width: 80%"> <figcaption>Performance of each model on HakushoBench.</figcaption> </figure>

How to Use

Due to copyright restrictions, the images in HakushoBench are not distributed through Hugging Face and are instead hosted on a domestic server in Japan.

To obtain the images required for evaluation, clone the GitLab repository:

$ git clone https://gitlab.llm-jp.nii.ac.jp/datasets/hakushobench

After downloading the dataset locally, you can load it using the 🤗 Datasets library:

from datasets import load_dataset
ds = load_dataset("hakushobench/data", split="test")
print(ds)
Dataset({
    features: ['original_id', 'image', 'category', 'question', 'answer', 'question_type', 'image_url', 'image_path', 'image_type'],
    num_rows: 2053
})

License

HakushoBench (excluding images) is licensed under the Apache License 2.0.

Images are provided solely for use within the scope permitted by Article 30-4 of the Japanese Copyright Act. Copyright for each image remains with its respective copyright holder.

Citation

If you find HakushoBench useful, please consider citing our work:

@misc{sugiura2026hakushobenchjapanesecharttable,
      title={HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Papers}, 
      author={Issa Sugiura and Shuhei Kurita and Yusuke Oda and Naoaki Okazaki},
      year={2026},
      eprint={2606.01132},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2606.01132}, 
}