apple/DepthPro

1---2license: apple-amlr3pipeline_tag: depth-estimation4library_name: depth-pro5---6 7# Depth Pro: Sharp Monocular Metric Depth in Less Than a Second8 9![Depth Pro Demo Image](https://github.com/apple/ml-depth-pro/raw/main/data/depth-pro-teaser.jpg)10 11We present a foundation model for zero-shot metric monocular depth estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction, a training protocol that combines real and synthetic datasets to achieve high metric accuracy alongside fine boundary tracing, dedicated evaluation metrics for boundary accuracy in estimated depth maps, and state-of-the-art focal length estimation from a single image.12 13Depth Pro was introduced in **[Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://arxiv.org/abs/2410.02073)**, by *Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, and Vladlen Koltun*.14 15The checkpoint in this repository is a reference implementation, which has been re-trained. Its performance is close to the model reported in the paper but does not match it exactly.16 17## How to Use18 19Please, follow the steps in the [code repository](https://github.com/apple/ml-depth-pro) to set up your environment. Then you can download the checkpoint from the _Files and versions_ tab above, or use the `huggingface-hub` CLI:20 21```bash22pip install huggingface-hub23huggingface-cli download --local-dir checkpoints apple/DepthPro24```25 26### Running from commandline27 28The code repo provides a helper script to run the model on a single image:29 30```bash31# Run prediction on a single image:32depth-pro-run -i ./data/example.jpg33# Run `depth-pro-run -h` for available options.34```35 36### Running from Python37 38```python39from PIL import Image40import depth_pro41 42# Load model and preprocessing transform43model, transform = depth_pro.create_model_and_transforms()44model.eval()45 46# Load and preprocess an image.47image, _, f_px = depth_pro.load_rgb(image_path)48image = transform(image)49 50# Run inference.51prediction = model.infer(image, f_px=f_px)52depth = prediction["depth"]  # Depth in [m].53focallength_px = prediction["focallength_px"]  # Focal length in pixels.54```55 56### Evaluation (boundary metrics) 57 58Boundary metrics are implemented in `eval/boundary_metrics.py` and can be used as follows:59 60```python61# for a depth-based dataset62boundary_f1 = SI_boundary_F1(predicted_depth, target_depth)63 64# for a mask-based dataset (image matting / segmentation) 65boundary_recall = SI_boundary_Recall(predicted_depth, target_mask)66```67 68 69## Citation70 71If you find our work useful, please cite the following paper:72 73```bibtex74@article{Bochkovskii2024:arxiv,75  author     = {Aleksei Bochkovskii and Ama\"{e}l Delaunoy and Hugo Germain and Marcel Santos and76               Yichao Zhou and Stephan R. Richter and Vladlen Koltun}77  title      = {Depth Pro: Sharp Monocular Metric Depth in Less Than a Second},78  journal    = {arXiv},79  year       = {2024},80}81```82 83## Acknowledgements84 85Our codebase is built using multiple opensource contributions, please see [Acknowledgements](https://github.com/apple/ml-depth-pro/blob/main/ACKNOWLEDGEMENTS.md) for more details.86 87Please check the paper for a complete list of references and datasets used in this work.88