juancobos/MultiTaskMouseBehaviour

1---2pretty_name: Multi-Task Mouse Behaviour Dataset3license: cc-by-nc-4.04viewer: false5task_categories:6  - object-detection7  - image-segmentation8  - keypoint-detection9tags:10  - animal-pose11  - behavioral-neuroscience12  - mouse13  - instance-segmentation14  - multi-object-tracking15  - video16  - sam317  - deeplabcut18size_categories:19  - 10K<n<100K20---21 22<table>23<tr>24<td align="center"><img src="https://huggingface.co/datasets/juancobos/MultiTaskMouseBehaviour/resolve/main/assets/gifs/barnes_maze.gif" width="260"><br><code>barnes_maze</code></td>25<td align="center"><img src="https://huggingface.co/datasets/juancobos/MultiTaskMouseBehaviour/resolve/main/assets/gifs/direct_interaction.gif" width="260"><br><code>direct_interaction</code></td>26<td align="center"><img src="https://huggingface.co/datasets/juancobos/MultiTaskMouseBehaviour/resolve/main/assets/gifs/marble.gif" width="260"><br><code>marble</code></td>27</tr>28<tr>29<td align="center"><img src="https://huggingface.co/datasets/juancobos/MultiTaskMouseBehaviour/resolve/main/assets/gifs/nort.gif" width="260"><br><code>nort</code></td>30<td align="center"><img src="https://huggingface.co/datasets/juancobos/MultiTaskMouseBehaviour/resolve/main/assets/gifs/nort2.gif" width="260"><br><code>nort2</code></td>31<td align="center"><img src="https://huggingface.co/datasets/juancobos/MultiTaskMouseBehaviour/resolve/main/assets/gifs/open_field.gif" width="260"><br><code>open_field</code></td>32</tr>33<tr>34<td align="center"><img src="https://huggingface.co/datasets/juancobos/MultiTaskMouseBehaviour/resolve/main/assets/gifs/social_interaction.gif" width="260"><br><code>social_interaction</code></td>35<td align="center"><img src="https://huggingface.co/datasets/juancobos/MultiTaskMouseBehaviour/resolve/main/assets/gifs/t_maze.gif" width="260"><br><code>t_maze</code></td>36<td align="center"><img src="https://huggingface.co/datasets/juancobos/MultiTaskMouseBehaviour/resolve/main/assets/gifs/three_chamber.gif" width="260"><br><code>three_chamber</code></td>37</tr>38</table>39 40# Multi-Task Mouse Behaviour Dataset41 42Bounding boxes, instance masks, 27-point poses and persistent track ids for43laboratory mice, **on the same frames**, across nine standard behavioural44assays. 14,300 frames, 27,716 annotated instances, 1280×720.45 46Most animal-behaviour datasets give you one annotation type. Training or47evaluating a model that does detection *and* segmentation *and* pose *and*48tracking usually means stitching together sources that disagree on species,49viewpoint, and label conventions. This dataset provides all four on every frame,50under one category schema, from real recordings of running experiments.51 52## Where the annotations come from53 54Every label is **machine-generated** by the [DeepLabSAM](https://github.com/juan-cobos/DeepLabSAM) pipeline.55The pipeline is a two-stage composition:56 571. **Detection, segmentation and tracking — SAM 3 (video).** A text prompt58   (`"mouse"`) drives promptable detection and tracking, producing a pixel-accurate59   mask and a persistent `track_id` per animal per frame.602. **Pose — DeepLabCut SuperAnimal (`superanimal_topviewmouse`), mask-gated.**61   Each instance is cropped to its own box and the crop's background is zeroed62   **using that instance's mask** before it reaches the pose head. Masking is63   applied after normalisation, where the network's neutral value is 0, so the64   background drops to that baseline instead of becoming an out-of-distribution65   black box.66 67Mask-gating is what makes stage 2 usable as an annotator. A top-down pose model fed68a raw crop of two touching animals has no way to know which one it should fit;69gated by the mask, the intended animal is the only signal present. That is why the70poses survive the crowded assays (`open_field`, `social_interaction`) where a plain71detect-then-crop cascade puts keypoints on the neighbour.72 73## Tasks74 75Each task is one continuous recording of a standard behavioural assay, filmed from76above and decoded to consecutive frames at 30 fps. Frames are contiguous, so77**neighbours are near-duplicates** — the ~14k frames are roughly 7.9 minutes of78footage, not 14k independent samples.79 80| Task | Frames | Instances | Animals | ≈ Duration | Median box (px²) |81|---|---:|---:|---:|---:|---:|82| `barnes_maze` | 1,847 | 1,847 | 1 | 62 s | 8,241 |83| `direct_interaction` | 1,430 | 2,860 | 2 | 48 s | 92,987 |84| `marble` | 1,797 | 1,797 | 1 | 60 s | 102,820 |85| `nort` | 1,719 | 1,708 | 1 | 57 s | 2,880 |86| `nort2` | 1,053 | 1,053 | 1 | 35 s | 22,518 |87| `open_field` | 1,939 | 7,690 | 4 | 65 s | 1,377 |88| `social_interaction` | 1,472 | 5,888 | 4 | 49 s | 2,352 |89| `t_maze` | 2,128 | 2,128 | 1 | 71 s | 1,409 |90| `three_chamber` | 915 | 2,745 | 3 | 30 s | 2,816 |91| **Total** | **14,300** | **27,716** | | **≈ 7.9 min** | |92 93**`barnes_maze`** — Spatial-learning assay. One mouse on a circular platform ringed94with escape holes.95 96**`direct_interaction`** — Two mice in a small bedding-filled arena with no divider97between them. The only task with two coat colours.98 99**`marble`** — Marble-burying assay for repetitive and anxiety-like behaviour.100 101**`nort` / `nort2`** — Novel Object Recognition: one mouse investigating objects in102an open arena, recorded twice in different arenas. The only within-assay pair.103 104**`open_field`** — Locomotion and anxiety assay. Four mice in a quadrant-divided105arena, one per compartment.106 107**`social_interaction`** — Four mice in a divided arena, approaching and contacting108one another across dividers.109 110**`t_maze`** — Spatial working-memory assay. One mouse running the arms of a111T-shaped maze.112 113**`three_chamber`** — Sociability assay. Three mice in a chamber divided by114transparent partitions.115 116Four of nine tasks are multi-animal.117 118## Layout119 120```121<task>/122  images/frame_00000.jpg ...        # consecutive decoded frames, 1280x720123  annotations.json                  # COCO: bbox, segmentation, track_id, keypoints124assets/gifs/<task>.gif              # gallery previews for this card125assets/examples/<task>.mp4          # 10 s unannotated clips, e.g. as demo inputs126```127 128One COCO file per task. Splits and training exports (e.g. RF-DETR layout) are129built from these with the [`mtmb`](https://github.com/juan-cobos/mtmb) package.130 131### Annotation schema132 133Standard COCO detection + keypoints, one category (`mouse`, id 1), with a few134additions:135 136| Field | Meaning |137|---|---|138| `track_id` | Persistent identity across frames within a task. |139| `score` | SAM 3's detection confidence. |140| `keypoints` | Flat `[x, y, v] × 27`, SuperAnimal `topviewmouse` order. |141| `keypoint_scores` | Per-keypoint model confidence — use this to re-threshold. |142| `num_keypoints` | How many cleared the 0.3 threshold. |143| `images[*].frame_index` | Position in the source recording (temporal order). |144 145The 27 keypoints follow SuperAnimal `topviewmouse` naming exactly, so DeepLabCut146models are directly comparable:147 148`nose`, `left_ear`, `right_ear`, `left_ear_tip`, `right_ear_tip`, `left_eye`,149`right_eye`, `neck`, `mid_back`, `mouse_center`, `mid_backend`, `mid_backend2`,150`mid_backend3`, `tail_base`, `tail1`, `tail2`, `tail3`, `tail4`, `tail5`,151`left_shoulder`, `left_midside`, `left_hip`, `right_shoulder`, `right_midside`,152`right_hip`, `tail_end`, `head_midpoint`153 154## License155 156CC BY-NC 4.0. The pose annotations are the output of DeepLabCut's157`superanimal_topviewmouse` checkpoint, whose weights (distinct from the LGPL-3.0158DeepLabCut codebase) are licensed for academic, non-commercial use only. That159restriction is what this dataset inherits and passes on; it is not a choice made160independently of the pipeline that produced the labels.161 162## Acknowledgments163 164Annotations were produced with:165 166- **SAM 3** (Meta AI / FAIR) — detection, segmentation and tracking.167- **DeepLabCut SuperAnimal, `superanimal_topviewmouse`** (Mathis Lab, EPFL) — pose168  estimation. See [Ye et al., 2024](https://www.nature.com/articles/s41467-024-48792-2).169 170Recordings were collected at [IGF, CNRS](https://www.igf.cnrs.fr/) and [UPV/EHU]().171 172## Citation173 174A paper describing this dataset is in preparation. Citation details (BibTeX) will175be added here on publication.176 177```bibtex178@dataset{mtmb,179  title   = {Multi-Task Mouse Behaviour Dataset},180  author  = {TBD},181  year    = {TBD},182  note    = {TBD},183}184```185