davanstrien/viewer-test-25-large-images-fail
Test Dataset: viewer-test-25-large-images-fail Purpose: Demonstrate dataset viewer behavior with large images and row group sizing. Dataset Details Number of images: 25 Total size: ~318MB Average per image: ~12.7MB Expected Behavior 25 large images (~14MB each, ~350MB total). Even with fewer images, the row group still exceeds 300MB. Prediction: FAIL - Scan size limit exceeded Technical Context The dataset viewer converts… See the full description on the dataset page: https://huggingface.co/datasets/davanstrien/viewer-test-25-large-images-fail.
Test Dataset: viewer-test-25-large-images-fail
Purpose: Demonstrate dataset viewer behavior with large images and row group sizing.
Dataset Details
- Number of images: 25
- Total size: ~318MB
- Average per image: ~12.7MB
Expected Behavior
25 large images (~14MB each, ~350MB total). Even with fewer images, the row group still exceeds 300MB.
Prediction: FAIL - Scan size limit exceeded
Technical Context
The dataset viewer converts imagefolder datasets to parquet with:
- Row group size: 100 rows (for image datasets)
- Scan size limit: 300MB
If all images end up in one row group and exceed 300MB, the viewer fails with:
Scan size limit exceeded: attempted to read X bytes, limit is 300000000 bytesRelated Issue
This dataset was created to help debug row group sizing for large images. See: https://github.com/huggingface/dataset-viewer for context.