Juartaurus/poi_Engineering

1# Deep SORT2 3## Introduction4 5This repository contains code for *Simple Online and Realtime Tracking with a Deep Association Metric* (Deep SORT).6We extend the original [SORT](https://github.com/abewley/sort) algorithm to7integrate appearance information based on a deep appearance descriptor.8See the [arXiv preprint](https://arxiv.org/abs/1703.07402) for more information.9 10## Dependencies11 12The code is compatible with Python 2.7 and 3. The following dependencies are13needed to run the tracker:14 15* NumPy16* sklearn17* OpenCV18 19Additionally, feature generation requires TensorFlow (>= 1.0).20 21## Installation22 23First, clone the repository:24```25git clone https://github.com/nwojke/deep_sort.git26```27Then, download pre-generated detections and the CNN checkpoint file from28[here](https://drive.google.com/open?id=18fKzfqnqhqW3s9zwsCbnVJ5XF2JFeqMp).29 30*NOTE:* The candidate object locations of our pre-generated detections are31taken from the following paper:32```33F. Yu, W. Li, Q. Li, Y. Liu, X. Shi, J. Yan. POI: Multiple Object Tracking with34High Performance Detection and Appearance Feature. In BMTT, SenseTime Group35Limited, 2016.36```37We have replaced the appearance descriptor with a custom deep convolutional38neural network (see below).39 40## Running the tracker41 42The following example starts the tracker on one of the43[MOT16 benchmark](https://motchallenge.net/data/MOT16/)44sequences.45We assume resources have been extracted to the repository root directory and46the MOT16 benchmark data is in `./MOT16`:47```48python deep_sort_app.py \49    --sequence_dir=./MOT16/test/MOT16-06 \50    --detection_file=./resources/detections/MOT16_POI_test/MOT16-06.npy \51    --min_confidence=0.3 \52    --nn_budget=100 \53    --display=True54```55Check `python deep_sort_app.py -h` for an overview of available options.56There are also scripts in the repository to visualize results, generate videos,57and evaluate the MOT challenge benchmark.58 59## Generating detections60 61Beside the main tracking application, this repository contains a script to62generate features for person re-identification, suitable to compare the visual63appearance of pedestrian bounding boxes using cosine similarity.64The following example generates these features from standard MOT challenge65detections. Again, we assume resources have been extracted to the repository66root directory and MOT16 data is in `./MOT16`:67```68python tools/generate_detections.py \69    --model=resources/networks/mars-small128.pb \70    --mot_dir=./MOT16/train \71    --output_dir=./resources/detections/MOT16_train72```73The model has been generated with TensorFlow 1.5. If you run into74incompatibility, re-export the frozen inference graph to obtain a new75`mars-small128.pb` that is compatible with your version:76```77python tools/freeze_model.py78```79The ``generate_detections.py`` stores for each sequence of the MOT16 dataset80a separate binary file in NumPy native format. Each file contains an array of81shape `Nx138`, where N is the number of detections in the corresponding MOT82sequence. The first 10 columns of this array contain the raw MOT detection83copied over from the input file. The remaining 128 columns store the appearance84descriptor. The files generated by this command can be used as input for the85`deep_sort_app.py`.86 87**NOTE**: If ``python tools/generate_detections.py`` raises a TensorFlow error,88try passing an absolute path to the ``--model`` argument. This might help in89some cases.90 91## Training the model92 93To train the deep association metric model we used a novel [cosine metric learning](https://github.com/nwojke/cosine_metric_learning) approach which is provided as a separate repository.94 95## Highlevel overview of source files96 97In the top-level directory are executable scripts to execute, evaluate, and98visualize the tracker. The main entry point is in `deep_sort_app.py`.99This file runs the tracker on a MOTChallenge sequence.100 101In package `deep_sort` is the main tracking code:102 103* `detection.py`: Detection base class.104* `kalman_filter.py`: A Kalman filter implementation and concrete105   parametrization for image space filtering.106* `linear_assignment.py`: This module contains code for min cost matching and107   the matching cascade.108* `iou_matching.py`: This module contains the IOU matching metric.109* `nn_matching.py`: A module for a nearest neighbor matching metric.110* `track.py`: The track class contains single-target track data such as Kalman111  state, number of hits, misses, hit streak, associated feature vectors, etc.112* `tracker.py`: This is the multi-target tracker class.113 114The `deep_sort_app.py` expects detections in a custom format, stored in .npy115files. These can be computed from MOTChallenge detections using116`generate_detections.py`. We also provide117[pre-generated detections](https://drive.google.com/open?id=1VVqtL0klSUvLnmBKS89il1EKC3IxUBVK).118 119## Citing DeepSORT120 121If you find this repo useful in your research, please consider citing the following papers:122 123    @inproceedings{Wojke2017simple,124      title={Simple Online and Realtime Tracking with a Deep Association Metric},125      author={Wojke, Nicolai and Bewley, Alex and Paulus, Dietrich},126      booktitle={2017 IEEE International Conference on Image Processing (ICIP)},127      year={2017},128      pages={3645--3649},129      organization={IEEE},130      doi={10.1109/ICIP.2017.8296962}131    }132 133    @inproceedings{Wojke2018deep,134      title={Deep Cosine Metric Learning for Person Re-identification},135      author={Wojke, Nicolai and Bewley, Alex},136      booktitle={2018 IEEE Winter Conference on Applications of Computer Vision (WACV)},137      year={2018},138      pages={748--756},139      organization={IEEE},140      doi={10.1109/WACV.2018.00087}141    }