BreakpointAI/breakpoint-grounding-55m

Breakpoint Grounding 55M

Quick start

from datasets import load_dataset
ds = load_dataset("BreakpointAI/breakpoint-grounding-55m", split="train")
ds[0]  # {'image': <PIL.Image>, 'image_caption': ..., 'object_captions': [...], 'normalized_boxes': [...], 'img_size_wh': [...]}

Dataset summary

Breakpoint Grounding 55M is, to our knowledge, the largest instance-grounded image–text dataset released publicly. Every image comes with an image-level caption, a set of object bounding boxes, and a caption for every box. It was built by Breakpoint AI to train a 10B-parameter joint image + bounding-box diffusion model for synthetic object-detection data generation.

[image]

• —54,909,477 images, each with one image-level caption
• —511,652,664 bounding boxes (9.32 per image on average), each with its own caption

For comparison, ROVI (ICCV 2025), the most recent comparable instance-grounded dataset, labels 1M web images with a similar VLM + open-vocabulary-detector pipeline — this dataset is more than 50× larger.

Why this dataset exists

At Breakpoint, we trained diffusion models to generate synthetic training data for object detection models. Our ultimate model was a 10-billion-parameter joint diffusion model that generated images and bounding boxes simultaneously. Given 10 labeled images, it could adapt to a new scene and start producing labeled training images.

In academia, there has been considerable research on conditional generation (bounding boxes to image) (e.g. GLIGEN [Li et al., 2023], InstanceDiffusion [Wang et al., 2024]), and the associated datasets (e.g. ROVI [Peng et al., 2025]). While helpful, this work did not touch joint generation, and unfortunately was not at the scale necessary to be actually useful.

In order to get a better performing model, we decided to build this dataset, which is over 50 times larger than comparable datasets (e.g. ROVI, ICCV 2025, at 1M images). Having this much data is what allowed us to train a large and high-performing model.

Image sources

Images are drawn from four public web-image collections:

How it was built

• —Images: collected from the four sources above, resized to ~1MP, light integrity filtering (corrupt / unreadable files removed).
• —Annotations (image_caption, object_captions, normalized_boxes): generated by Breakpoint AI's automated annotation pipeline (object detection + captioning models). They are not human-labeled. See Appendix: Annotation pipeline for details.

Dataset structure

Fields

Which of the four source datasets each row came from is published separately, as metadata/source_dataset_index.parquet — see Which rows come from which source.

Bounding box format

Each box is [x_min, y_min, x_max, y_max] (xyxy), with coordinates normalized to [0, 1] relative to image width/height. normalized_boxes[i] corresponds to object_captions[i].

Splits

Example

from datasets import load_dataset

ds = load_dataset("BreakpointAI/breakpoint-grounding-55m", split="train", streaming=True)
row = next(iter(ds))

row["image"]              # PIL.Image
row["image_caption"]      # "a photograph of ..."
row["object_captions"]    # ["a dog", "a red ball", ...]
row["normalized_boxes"]   # [[0.12, 0.34, 0.56, 0.78], ...]
row["img_size_wh"]        # [1820, 1024]

Licensing and responsible use

This dataset aggregates images from multiple public sources, each under its own terms. They fall into three groups, not one blanket restriction:

• —No research-only restriction — usable for any purpose, including commercial, per the source's own license (with attribution where required):
• —The bounding boxes and captions contributed by Breakpoint AI itself — CC BY 4.0.
• —Images from Open Images — CC BY 2.0 (images), CC BY 4.0 (Google's annotations).
• —Images from Wikimedia Commons — CC BY-SA, CC BY, CC0, or public domain, per file; check the specific file's license for attribution/share-alike requirements.
• —Restricted to non-commercial research use:
• —Images from RedCaps — under the RedCaps terms, non-commercial research only.
• —Rights not cleared by Breakpoint — neither confirmed open nor research-restricted, genuinely case-by-case:
• —Images from LAION — the URLs/metadata are CC BY 4.0, but each image remains under its original, third-party owner's rights, which Breakpoint has not individually verified. Downstream users are responsible for determining the applicable rights before using these images beyond research.

Which rows come from which source

The rows themselves don't carry a source column, but `metadata/source_dataset_index.parquet` in this repo maps every row to its source. It has one row per dataset row, in the same order as the train split, with columns index (int32) and source_dataset (one of laion, wikimedia, redcaps, openimages).

from datasets import load_dataset
from huggingface_hub import hf_hub_download
import pyarrow.parquet as pq

ds = load_dataset("BreakpointAI/breakpoint-grounding-55m", split="train")

path = hf_hub_download(
    repo_id="BreakpointAI/breakpoint-grounding-55m",
    repo_type="dataset",
    filename="metadata/source_dataset_index.parquet",
)
sources = pq.read_table(path).column("source_dataset").to_pylist()

ds = ds.add_column("source_dataset", sources)          # now filterable
redcaps_only = ds.filter(lambda r: r["source_dataset"] == "redcaps")

The join is positional, so it only holds if you haven't shuffled or filtered the dataset before attaching the column.

Use of the images is governed by the applicable source's terms above; downstream users are responsible for complying with them.

Personal data. The images depict real, sometimes identifiable, people and places. Material being publicly posted does not remove it from the scope of data-protection laws such as the GDPR, UK GDPR, and CCPA. Do not use this dataset to identify, profile, track, surveil, or contact individuals, or to train biometric identification systems.

No affiliation. This dataset is not affiliated with, endorsed by, or connected to Google / Open Images, the Wikimedia Foundation, Reddit, or LAION e.V.

Removal requests. If you are the rights holder for an image, or are depicted in one, and want it removed, open a discussion on this repository identifying the affected row indices (or attaching the image) and the affected rows will be removed.

Limitations and biases

• —Annotations are model-generated and contain errors: missed objects, spurious boxes, imprecise coordinates, and caption hallucinations.
• —Inherits the content distribution and biases of the four source collections, including web-scale skews in geography, language, subject matter, and the demographics of who posts images to Reddit and Wikimedia.
• —Images vary in aesthetic and resolution quality; only a light integrity filter was applied.

Acknowledgements

• —Hosted on the Hugging Face Hub with a public dataset storage grant.
• —Thanks to Daniel van Strien and the Hugging Face datasets team.
• —Images from Open Images, Wikimedia Commons, RedCaps, and LAION.

Appendix: Annotation pipeline

• —Object detection: bounding boxes and class labels were produced by an open-vocabulary detector (Grounding DINO) as an upstream step.
• —Per-object captions: InternVL3-2B, served via LMDeploy (TurboMind engine, internvl2_5 chat template). Each detected box was cropped (no padding) and captioned with the prompt:

<image> Describe this <category name> in 20 words or less and focus on its color, appearance, size, and style, etc.

where <category name> is the detector's class label for that box. Generated with max_new_tokens=64.

• —Image-level captions: InternVL3-8B (via transformers), prompted with:

<image> Write a single, natural-language description of this image. Do not use bullet points or lists. Keep it under 100 words.

Generated with max_new_tokens=128, sampling enabled.

Citation

@misc{breakpoint_grounding_55m,
  title  = {Breakpoint Grounding 55M},
  author = {Wang, Franklin and Zhong, Desmond and Murdoch, Jamie},
  year   = {2026},
  url    = {https://huggingface.co/datasets/BreakpointAI/breakpoint-grounding-55m}
}

Related work

@inproceedings{peng2025rovi,
  title     = {ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary
               Instance-Grounded Text-to-Image Generation},
  author    = {Peng, Cihang and Hou, Qiming and Ren, Zhong and Zhou, Kun},
  booktitle = {ICCV},
  year      = {2025}
}

@inproceedings{li2023gligen,
  title     = {GLIGEN: Open-Set Grounded Text-to-Image Generation},
  author    = {Li, Yuheng and Liu, Haotian and Wu, Qingyang and Mu, Fangzhou and Yang, Jianwei
               and Gao, Jianfeng and Li, Chunyuan and Lee, Yong Jae},
  booktitle = {CVPR},
  year      = {2023}
}

@inproceedings{wang2024instancediffusion,
  title     = {InstanceDiffusion: Instance-level Control for Image Generation},
  author    = {Wang, Xudong and Darrell, Trevor and Rambhatla, Sai Saketh and Girdhar, Rohit
               and Misra, Ishan},
  booktitle = {CVPR},
  year      = {2024}
}

<!-- TODO: verify these two entries (author order / exact venue) before publishing — drafted from memory, not looked up. -->