Datasetpublic

GEM/references

GEM References What is it? This repository contains all the reference datasets that are used for running evaluation on the GEM benchmark. Some of these datasets were originally hosted as a GitHub release on the GEM-metrics repository, but have been migrated to the Hugging Face Hub. Converting datasets to JSON We provide a convert_dataset_to_json.py conversion script that converts the datasets in the GEM organisation to the JSON format expected by… See the full description on the dataset page: https://huggingface.co/datasets/GEM/references.

sourceHugging Faceupdated 4y agoView on Hugging Face

0likes1kdownloads

Dataset Card

GEM References

What is it?

This repository contains all the reference datasets that are used for running evaluation on the GEM benchmark. Some of these datasets were originally hosted as a GitHub release on the `GEM-metrics` repository, but have been migrated to the Hugging Face Hub.

Converting datasets to JSON

We provide a convert_dataset_to_json.py conversion script that converts the datasets in the GEM organisation to the JSON format expected by the GEM-metrics library. To run the script, first install `jq` and then install the script's Python dependencies:

python -m pip install -r requirements.txt

You can then run the script as follows:

python

python generate_evaluation_datasets.py

This script will:

—Download and convert the datasets under the GEM organisation to JSON format
—Validate that the each dataset has the expected columns of gem_id, target, and references