Team Ai
Datasetpublic

dali-does/clevr-math

CLEVR-Math is a dataset for compositional language, visual and mathematical reasoning. CLEVR-Math poses questions about mathematical operations on visual scenes using subtraction and addition, such as "Remove all large red cylinders. How many objects are left?". There are also adversarial (e.g. "Remove all blue cubes. How many cylinders are left?") and multihop questions (e.g. "Remove all blue cubes. Remove all small purple spheres. How many objects are left?").

sourceHugging Facecc-by-4.0updated 4y agoView on Hugging Face
26likes1.7kdownloads
Dataset Card

Dataset Card for CLEVR-Math

Table of Contents

Dataset Description

  • —Homepage:
  • —*Repository:https://github.com/dali-does/clevr-math*
  • —*Paper:https://arxiv.org/abs/2208.05358*
  • —Leaderboard:
  • —*Point of Contact:dali@cs.umu.se*

Dataset Summary

Dataset for compositional multimodal mathematical reasoning based on CLEVR.

Loading the data, preprocessing text with CLIP
from transformers import CLIPPreprocessor
from datasets import load_dataset, DownloadConfig


dl_config = DownloadConfig(resume_download=True,
        num_proc=8,
        force_download=True)

# Load 'general' instance of dataset
dataset = load_dataset('dali-does/clevr-math', download_config=dl_config)

# Load version with only multihop in test data
dataset_multihop = load_dataset('dali-does/clevr-math', 'multihop',
                                download_config=dl_config)


model_path = "openai/clip-vit-base-patch32"
extractor = CLIPProcessor.from_pretrained(model_path)
def transform_tokenize(e):
  e['image'] = [image.convert('RGB') for image in e['image']]
  return extractor(text=e['question'],
                           images=e['image'],
                           padding=True)

dataset = dataset.map(transform_tokenize,
                      batched=True,
                      num_proc=8,
                      padding='max_length')

dataset_subtraction = dataset.filter(lambda e:
        e['template'].startswith('subtraction'), num_proc=4)

Supported Tasks and Leaderboards

Leaderboard will be announced at a later date.

Languages

The dataset is currently only available in English. To extend the dataset to other languages, the CLEVR templates must be rewritten in the target language.

Dataset Structure

Data Instances

  • —general containing the default version with multihop questions in train and test
  • —multihop containing multihop questions only in test data to test generalisation of reasoning

Data Fields

features = datasets.Features(
  {
    "template": datasets.Value("string"),
    "id": datasets.Value("string"),
    "question": datasets.Value("string"),
    "image": datasets.Image(),
    "label": datasets.Value("int64")
  }
)

Data Splits

train/val/test

Dataset Creation

Data is generated using code provided with the CLEVR-dataset, using blender and templates constructed by the dataset curators.

Considerations for Using the Data

Other Known Limitations

[More Information Needed]

Additional Information

Dataset Curators

Adam Dahlgren Lindström - dali@cs.umu.se

Licensing Information

Licensed under Creative Commons Attribution Share Alike 4.0 International (CC-by 4.0).

Citation Information

[More Information Needed]

@misc{https://doi.org/10.48550/arxiv.2208.05358,
  doi = {10.48550/ARXIV.2208.05358},
  url = {https://arxiv.org/abs/2208.05358},
  author = {Lindström, Adam Dahlgren and Abraham, Savitha Sam},
  keywords = {Machine Learning (cs.LG), Computation and Language (cs.CL), Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences, I.2.7; I.2.10; I.2.6; I.4.8; I.1.4},
  title = {CLEVR-Math: A Dataset for Compositional Language, Visual, and Mathematical Reasoning},
  publisher = {arXiv},
  year = {2022},
  copyright = {Creative Commons Attribution Share Alike 4.0 International}
}

Contributions

Thanks to @dali-does for adding this dataset.