Datasetpublic

Chrisyichuan/madqa-training

Chrisyichuan/madqa-training MADQA document QA contrastive training data with hard negatives. Contents madqa_converted.jsonl — query-image pairs with hard negatives images/ — all referenced images Each metadata row: { "query": "...", "chunk_path": "images/...", "neg_chunk_paths": ["images/...", "images/..."], "source_positive_rank": 0, "source_positive_score": 0.0, "source_dataset": "moca" } Summary rows: 1840 unique images: 3598 avg… See the full description on the dataset page: https://huggingface.co/datasets/Chrisyichuan/madqa-training.

sourceHugging Facemitupdated 6mo agoView on Hugging Face

0likes61downloads

Dataset Card

Chrisyichuan/madqa-training

MADQA document QA contrastive training data with hard negatives.

—madqa_converted.jsonl — query-image pairs with hard negatives
—images/ — all referenced images

Each metadata row:

json

{
  "query": "...",
  "chunk_path": "images/...",
  "neg_chunk_paths": ["images/...", "images/..."],
  "source_positive_rank": 0,
  "source_positive_score": 0.0,
  "source_dataset": "moca"
}

Summary

—rows: 1840
—unique images: 3598
—avg negatives/row: 2.00

Download

python

from huggingface_hub import snapshot_download
snapshot_download(repo_id="Chrisyichuan/madqa-training", repo_type="dataset", local_dir="data/madqa-training")

Image Storage

Images are stored as 4 tar shards under image_shards/ for fast download.

After cloning/downloading, extract images:

bash

python extract_hf_image_shards.py --dataset-dir .

This creates images/ with all referenced image files.