Datasetpublic

yayoimizuha/Glint360k

Dataset Card for Glint360K Citiation by InsightFace Repository We clean, merge, and release the largest and cleanest face recognition dataset Glint360K, which contains 17091657 images of 360232 individuals. By employing the Patial FC training strategy, baseline models trained on Glint360K can easily achieve state-of-the-art performance. Detailed evaluation results on the large-scale test set (e.g. IFRT, IJB-C and Megaface) are as follows: Dataset Details… See the full description on the dataset page: https://huggingface.co/datasets/yayoimizuha/Glint360k.

sourceHugging Faceupdated 1y agoView on Hugging Face

5likes1.6kdownloads

Dataset Card

Dataset Card for Glint360K

Citiation by InsightFace Repository

We clean, merge, and release the largest and cleanest face recognition dataset Glint360K, which contains 17091657 images of 360232 individuals. By employing the Patial FC training strategy, baseline models trained on Glint360K can easily achieve state-of-the-art performance. Detailed evaluation results on the large-scale test set (e.g. IFRT, IJB-C and Megaface) are as follows:

Dataset Details

Dataset Description

—Curated by: InsightFace
—Shared by: Academic Torrents
—License: The license isn't clearly stated, but InsightFace says it's for "available for non-commercial research purposes only."

Dataset Sources

—Repository: deepinsight/insightface
—Paper: https://arxiv.org/abs/2010.05222

Uses

It is used for training face recognition models such as RetinaFace, FaceNet, etc.

Dataset Structure

It adopts the WebDataset format, with images and metadata (class: cls) stored in tar files split every 16GB.

Dataset Creation

Source Data

Get Data from torrent and concatenate divided tar files. Next, extract the tar file. Finally, the directory structure is as follows:

.\glint360k\
├── agedb_30.bin
├── calfw.bin
├── cfp_ff.bin
├── cfp_fp.bin
├── cplfw.bin
├── lfw.bin
├── train.idx
├── train.rec
└── vgg2_fp.bin

Data Collection and Processing

use train.rec and train.idx. Save the following script and run it with uv run script.py.

# /// script
# dependencies = [
#     "mxnet",
#     "numpy=<1.24",
#     "Pillow",
#     "tqdm",
# ]
# requires-python = "==3.10.*"
# ///
import mxnet
import os
from PIL import Image
from tqdm import tqdm

glint360k_root = "/path/to/glint360k"
idx_path = os.path.join(glint360k_root, "train.idx")
rec_path = os.path.join(glint360k_root, "train.rec")
export_path = "/path/to/glint360k_export"

imgrec = mxnet.recordio.MXIndexedRecordIO(idx_path, rec_path, 'r')

print(f"Total records to process: {imgrec.keys.__len__()}")

for i in tqdm(imgrec.keys):
    header, content = mxnet.recordio.unpack(imgrec.read_idx(i))
    label = int(header.label if isinstance(header.label, (int, float)) else header.label[0])
    label_dir = os.path.join(export_path, str(label))
    if not os.path.exists(label_dir):
        os.makedirs(label_dir, exist_ok=True)
    
    img = mxnet.image.imdecode(content).asnumpy()
    img = Image.fromarray(img.astype('uint8'))
    img_save_path = os.path.join(label_dir, f'{i}.jpg')
    img.save(img_save_path, quality=93)

print("Export complete.")

This converts Glint360K to PyTorch ImageFolder format. To convert it to WebDataset format, follow the steps below.

# /// script
# dependencies = [
#   "webdataset",
#   "torchvision",
#   "tqdm",
#   "torch",
# ]
# requires-python = ">=3.9"
# ///
import os
import webdataset
from torchvision import datasets
from tqdm import tqdm

imagefolder_path = "/path/to/glint360k_export"

output_prefix = "/path/to/glint360k_WebDataset/glint360k_train"

dataset = datasets.ImageFolder(
    root=imagefolder_path,
    transform=None
)

with webdataset.ShardWriter(f"{output_prefix}-%02d.tar", maxsize=1.1e+10, maxcount=float('inf')) as writer:
    for image_path, label in tqdm(dataset.imgs, desc="Converting to WebDataset"):
        with open(image_path, "rb") as image_file:
            image_bytes = image_file.read()

        basename = os.path.splitext(os.path.basename(image_path))[0]

        sample = {
            "__key__": basename,
            "jpg": image_bytes,
            "cls": str(label)
        }

        writer.write(sample)

print("Conversion to WebDataset format complete.")

Personal and Sensitive Information

This dataset collects human faces and contains personally identifiable information. Please handle it with care.

Bias, Risks, and Limitations

The dataset may not consider the diversity of race, gender, age distribution, and shooting environments in the included facial images. This may lead to biases against specific groups.