uv-scripts/data-processing

1---2viewer: false3tags:4  - uv-script5  - data-processing6  - parquet7  - buckets8  - webhooks9---10 11# Data processing12 13> Part of [uv-scripts](https://huggingface.co/uv-scripts) — self-contained UV scripts you run on Hugging Face Jobs in one command.14 15General data processing recipes: convert, clean and prepare data files.16 17| Script | What it does |18|---|---|19| [`optimize-parquet.py`](#optimize-parquetpy-optimized-parquet-from-bucket-uploads) | Converts CSV, JSON and Parquet files uploaded to a bucket into optimized Parquet, triggered by a bucket webhook |20 21## optimize-parquet.py: optimized Parquet from bucket uploads22 23Upload a CSV, JSON or Parquet file to a [Storage Bucket](https://huggingface.co/docs/hub/storage-buckets) and get an optimized Parquet version in a second bucket, automatically. A bucket [webhook](https://huggingface.co/docs/hub/webhooks) starts a [Job](https://huggingface.co/docs/hub/jobs) for each upload, and the Job converts only the files that changed.24 25```26input bucket  ──upload──▶  webhook  ──▶  Job (optimize-parquet.py)  ──▶  output bucket27data.csv                                                                 data.csv/data/train-00000-of-00001.parquet28```29 30The output is written by [`datasets`](https://huggingface.co/docs/datasets), so it gets the same [optimizations](https://huggingface.co/docs/hub/datasets-libraries#optimized-parquet-files) as `push_to_hub`: content-defined chunking for Xet deduplication, a page index for fast filtering and random access, and row groups of at most 100 MB.31 32### Setup33 34You need two buckets: one you upload to, and one for the output. The Job writes to a different bucket so that its own output does not trigger it again.35 36```bash37hf buckets create my-raw-files --private38hf buckets create my-parquet --private39```40 41**1. Create a base Job for the webhook to re-run.** With no webhook payload, this first run exits straight away:42 43```bash44hf jobs run --flavor cpu-upgrade --timeout 2h -e OUTPUT_BUCKET=<user>/my-parquet \45    ghcr.io/astral-sh/uv:python3.12-bookworm \46    uv run https://huggingface.co/datasets/uv-scripts/data-processing/raw/main/optimize-parquet.py47```48 49Use `hf jobs run ... uv run <url>` here, not `hf jobs uv run <url>`. `hf jobs uv run` uploads the script as a volume, and webhook runs don't keep volumes.50 51**2. Create a webhook on the input bucket that re-runs this Job:**52 53```bash54hf webhooks create --job-id <job id from step 1> \55    --watch bucket:<user>/my-raw-files --domain repo --secrets HF_TOKEN56```57 58`--secrets HF_TOKEN` stores a token with the webhook, encrypted, and every triggered run receives it as a secret to read and write the buckets. The value comes from `HF_TOKEN` in your environment, or from the token you logged in with; pass `--secrets HF_TOKEN=hf_…` to use a different one. Use a fine-grained token, not your main one. The base Job needs no `--secrets`, because a webhook run does not inherit the Job's own secrets.59 60**3. Upload a file:**61 62```bash63hf buckets cp data.csv hf://buckets/<user>/my-raw-files/data.csv64```65 66After about a minute, the output is in `<user>/my-parquet/data.csv/`: the Parquet file(s) under `data/`, plus a README written by `datasets`.67 68### Options69 70| Environment variable | Default | Meaning |71|---|---|---|72| `OUTPUT_BUCKET` | required | Bucket to write the Parquet files to. Must differ from the watched bucket. |73| `STREAM_ABOVE_BYTES` | 1/3 of free disk | Files larger than this are streamed instead of loaded to disk. |74 75Files that fit on the Job's disk are loaded in full. Larger files are streamed, so they don't need to fit on the disk (50 GB on `cpu-upgrade`); raise `--timeout` for very large files. Supported inputs: `.csv`, `.json`, `.jsonl`, `.parquet`. Other files are skipped, and deleted files are ignored.76 77### Notes78 79- **Cost:** a small file takes about 20 seconds on `cpu-upgrade` ($0.03/hour). In testing, one `hf buckets sync` of several files sent one webhook event, so it started one Job.80- **Pin the script:** each webhook run downloads the script again. To stop changes to this recipe from reaching your webhook, replace `main` in the URL with a commit hash.81- **Limits:** a webhook can trigger at most 1,000 times per 24 hours. Above 10,000 changed files in one event, the payload list is truncated; those files are not converted.82