SparseWake/sparsewake

1---2license: cc-by-4.03pretty_name: SparseWake4tags:5- benchmark6- synthetic-data7- hydrodynamics8- time-series9- regression10- physical-sensing11- hdf512- fish-schooling13---14 15# SparseWake16 17SparseWake is a synthetic benchmark for sparse temporal hydrodynamic sensing.18 19## ICLR 2027 release20 21The expanded release adds controlled multi-source mixtures and common-prior nearest-source tasks, with complete core data banks, reference checkpoints, a small review supplement, and reproduction code with a frozen wake-library input.22 23**[Download release iclr2027-v1.0rc2](versions/iclr2027-v1.0rc2/README.md)**24 25The version page lists the three archives, exact sizes, checksums, extraction instructions, generation scope, and component licenses. Data and result tables are CC BY 4.0; new original code is MIT; historical and third-party components retain their supplied notices. The generation code supports sensing scenes based on the cited reduced-order model, not a complete free-swimming schooling simulator or DNS reproduction.26 27For reproducibility, record the release identifier and final immutable repository commit. The historical root-level manifest and metadata describe the May 2026 files; the new release has its own archive checksums and package manifests.28 29## Historical May 2026 release30 31The files and instructions below describe the original May 2026 release only. They are retained for compatibility; their inclusion and license statements do not override the expanded release documentation linked above.32 33SparseWake is an anonymous benchmark artifact for sparse temporal hydrodynamic sensing. It contains processed HDF5 datasets, metadata, summary CSV files, checksums, and lightweight Python scripts for inspecting the data and reproducing benchmark-level evaluation outputs.34### Dataset Summary35 36SparseWake evaluates whether sparse body-fixed temporal flow measurements can recover the relative state of a neighboring wake-producing fish in a controlled single-leader setting. The benchmark data are synthetic and were generated from a DNS-parameterized fish-schooling agent-based model. The artifact starts from processed HDF5 files and does not require the upstream simulator or high-fidelity solver.37 38The released data support held-out-pose evaluation, temporal-history sweeps, sensor-count ablations, component-separated flow inputs, common raw-noise stress tests, sample-size convergence checks, architecture screening summaries, and paired self-signal controls.39 40### What Is Included41 42- Full processed HDF5 benchmark datasets in `data/processed/`.43- A small sample HDF5 file in `data/sample/` for inspection and smoke tests.44- Dataset metadata, schema documentation, split protocol, provenance notes, and limitations in `docs/`.45- Summary CSV files for reported tables and figure-level results in `data/results/`.46- Reproduced table CSVs in `tables/`.47- Lightweight loading, verification, plotting, and smoke-test training scripts in `scripts/`.48- Minimal reusable Python utilities in `src/sparsewake/`.49- Checksums in `data/checksums.sha256` and a full file manifest in `MANIFEST.json`.50- Croissant-style metadata in `croissant_metadata.json`.51 52### Main Files53 54| Path | Purpose |55|---|---|56| `data/processed/abm_sensing_dataset_v04_close_orient30_potential_160k.h5` | Main external component benchmark with wake-only, potential-only, and wake-plus-potential signals. |57| `data/processed/abm_sensing_dataset_v05_self_external_paired_160k_compressed.h5` | Paired aligned-orientation self-signal control with external-only, self-only, and total-like external-plus-self arrays. |58| `data/processed/abm_sensing_dataset_v05_self_external_orient30_160k.h5` | Randomized-orientation self-signal control with external-only, self-only, and total-like arrays. |59| `data/sample/sparsewake_sample.h5` | Small sample file for quick inspection and smoke tests. |60| `docs/data_fields.md` | HDF5 fields, shapes, and label definitions. |61| `docs/benchmark_protocol.md` | Evaluation split, feature construction, standardization, and metrics. |62| `docs/provenance.md` | Data-generation provenance and non-redistributed upstream components. |63| `dataset_card.md` | Dataset-card details beyond this landing page. |64| `evaluation_card.md` | Benchmark task, metrics, ablations, and expected summary numbers. |65| `scripts/verify_dataset.py` | HDF5 structure and checksum verifier. |66| `scripts/train_temporal_mlp.py` | Small temporal-MLP smoke test and rerun utility. |67| `scripts/reproduce_tables.py` | Copies stored summary CSVs into `tables/`. |68| `scripts/make_main_figures.py`, `scripts/make_supp_figures.py` | Recreate lightweight figures from stored CSV summaries. |69 70### What Is Not Included71 72The upstream fish-schooling ABM source code, DNS solver files, MATLAB simulator-query code, and simulator internals are not redistributed. This release is intended to reproduce benchmark evaluations from processed HDF5 files, not to regenerate the DNS-parameterized wake library from first principles.73 74### Intended Use75 76SparseWake is intended for evaluating sparse temporal physical sensing, hydrodynamic neighbor-state estimation, robustness under controlled flow-component and noise protocols, and reproducibility of benchmark-level evaluation summaries.77 78### Out-of-Scope Use79 80SparseWake should not be treated as a complete biological lateral-line model, a full natural fish-schooling simulator, a pressure/shear sensing dataset, a direct CFD replacement, or a validation dataset for multi-neighbor closed-loop schooling behavior.81 82### Data Structure83 84HDF5 arrays are stored in MATLAB-style or sample-first layouts depending on source export; `src/sparsewake/data.py` loads them sample-first. Core fields include:85 86- `X_raw`: wake-plus-potential body-frame induced-velocity features.87- `X_wake_raw`: wake-only component where available.88- `X_potential_raw`: potential-only component where available.89- `X_external_raw`, `X_self_raw`, `X_total_raw`: paired self-signal controls where available.90- `y`: labels with columns `delta_x`, `delta_y`, `theta_rel`, `sin_phi`, `cos_phi`, `phi`.91- `groups`: wake phase index.92- `region_id`: sampling region code.93- `sensor_world_positions`: sensor coordinates for each sample.94 95See `docs/data_fields.md` for the full schema and `docs/benchmark_protocol.md` for the split and feature protocol.96 97### Quick Start98 99```bash100python -m venv .venv101.venv/Scripts/pip install -r requirements.txt102python scripts/verify_dataset.py --data data/sample/sparsewake_sample.h5 --checksums data/checksums.sha256103python scripts/train_temporal_mlp.py --config configs/main_v04.yaml --data data/sample/sparsewake_sample.h5 --quick104python scripts/reproduce_tables.py --results data/results --out tables105```106 107On Unix-like systems, use `.venv/bin/pip` and `.venv/bin/python`.108 109### Reproduce Stored Tables and Diagnostic Plots110 111```bash112python scripts/reproduce_tables.py --results data/results --out tables113python scripts/make_main_figures.py --results data/results --out figures114python scripts/make_supp_figures.py --results data/results --out figures115```116 117These commands use stored summary CSV files and are intended for fast numerical verification and diagnostic plotting. Final manuscript figures may include manual layout adjustments and are not part of the dataset artifact.118### License119 120The SparseWake processed datasets, metadata, result summaries, documentation, and release utilities are made available under the Creative Commons Attribution 4.0 International license (CC BY 4.0). Upstream simulator source code and solver assets are not redistributed and are not covered by this dataset release.121 122### Citation123 124During anonymous review, cite this dataset as:125 126```bibtex127@dataset{sparsewake_anonymous,128  title = {SparseWake: A Temporal Hydrodynamic Sensing Benchmark},129  author = {{Anonymous Authors}},130  year = {2026}131}132```133 134Author-identifying contact information is intentionally omitted during anonymous review.135