Arulkumar03/Fox_Sheep_Detector_Computer_Vision_model
0
1# ViTDet: Exploring Plain Vision Transformer Backbones for Object Detection2 3Yanghao Li, Hanzi Mao, Ross Girshick†, Kaiming He†4 5[[`arXiv`](https://arxiv.org/abs/2203.16527)] [[`BibTeX`](#CitingViTDet)]6 7In this repository, we provide configs and models in Detectron2 for ViTDet as well as MViTv2 and Swin backbones with our implementation and settings as described in [ViTDet](https://arxiv.org/abs/2203.16527) paper.8 9 10## Pretrained Models11 12### COCO13 14#### Mask R-CNN15 16<table><tbody>17<!-- START TABLE -->18<!-- TABLE HEADER -->19<th valign="bottom">Name</th>20<th valign="bottom">pre-train</th>21<th valign="bottom">train<br/>time<br/>(s/im)</th>22<th valign="bottom">inference<br/>time<br/>(s/im)</th>23<th valign="bottom">train<br/>mem<br/>(GB)</th>24<th valign="bottom">box<br/>AP</th>25<th valign="bottom">mask<br/>AP</th>26<th valign="bottom">model id</th>27<th valign="bottom">download</th>28<!-- TABLE BODY -->29<!-- ROW: mask_rcnn_vitdet_b_100ep -->30 <tr><td align="left"><a href="configs/COCO/mask_rcnn_vitdet_b_100ep.py">ViTDet, ViT-B</a></td>31<td align="center">IN1K, MAE</td>32<td align="center">0.314</td>33<td align="center">0.079</td>34<td align="center">10.9</td>35<td align="center">51.6</td>36<td align="center">45.9</td>37<td align="center">325346929</td>38<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/mask_rcnn_vitdet_b/f325346929/model_final_61ccd1.pkl">model</a></td>39</tr>40<!-- ROW: mask_rcnn_vitdet_l_100ep -->41 <tr><td align="left"><a href="configs/COCO/mask_rcnn_vitdet_l_100ep.py">ViTDet, ViT-L</a></td>42<td align="center">IN1K, MAE</td>43<td align="center">0.603</td>44<td align="center">0.125</td>45<td align="center">20.9</td>46<td align="center">55.5</td>47<td align="center">49.2</td>48<td align="center">325599698</td>49<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/mask_rcnn_vitdet_l/f325599698/model_final_6146ed.pkl">model</a></td>50</tr>51<!-- ROW: mask_rcnn_vitdet_b_75ep -->52 <tr><td align="left"><a href="configs/COCO/mask_rcnn_vitdet_h_75ep.py">ViTDet, ViT-H</a></td>53<td align="center">IN1K, MAE</td>54<td align="center">1.098</td>55<td align="center">0.178</td>56<td align="center">31.5</td>57<td align="center">56.7</td>58<td align="center">50.2</td>59<td align="center">329145471</td>60<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/mask_rcnn_vitdet_h/f329145471/model_final_7224f1.pkl">model</a></td>61</tr>62</tbody></table>63 64#### Cascade Mask R-CNN65 66<table><tbody>67<!-- START TABLE -->68<!-- TABLE HEADER -->69<th valign="bottom">Name</th>70<th valign="bottom">pre-train</th>71<th valign="bottom">train<br/>time<br/>(s/im)</th>72<th valign="bottom">inference<br/>time<br/>(s/im)</th>73<th valign="bottom">train<br/>mem<br/>(GB)</th>74<th valign="bottom">box<br/>AP</th>75<th valign="bottom">mask<br/>AP</th>76<th valign="bottom">model id</th>77<th valign="bottom">download</th>78<!-- TABLE BODY -->79<!-- ROW: cascade_mask_rcnn_swin_b_in21k_50ep -->80 <tr><td align="left"><a href="configs/COCO/cascade_mask_rcnn_swin_b_in21k_50ep.py">Swin-B</a></td>81<td align="center">IN21K, sup</td>82<td align="center">0.389</td>83<td align="center">0.077</td>84<td align="center">8.7</td>85<td align="center">53.9</td>86<td align="center">46.2</td>87<td align="center">342979038</td>88<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_swin_b_in21k/f342979038/model_final_246a82.pkl">model</a></td>89</tr>90<!-- ROW: cascade_mask_rcnn_swin_l_in21k_50ep -->91 <tr><td align="left"><a href="configs/COCO/cascade_mask_rcnn_swin_l_in21k_50ep.py">Swin-L</a></td>92<td align="center">IN21K, sup</td>93<td align="center">0.508</td>94<td align="center">0.097</td>95<td align="center">12.6</td>96<td align="center">55.0</td>97<td align="center">47.2</td>98<td align="center">342979186</td>99<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_swin_l_in21k/f342979186/model_final_7c897e.pkl">model</a></td>100</tr>101<!-- ROW: cascade_mask_rcnn_mvitv2_b_in21k_100ep -->102 <tr><td align="left"><a href="configs/COCO/cascade_mask_rcnn_mvitv2_b_in21k_100ep.py">MViTv2-B</a></td>103<td align="center">IN21K, sup</td>104<td align="center">0.475</td>105<td align="center">0.090</td>106<td align="center">8.9</td>107<td align="center">55.6</td>108<td align="center">48.1</td>109<td align="center">325820315</td>110<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_mvitv2_b_in21k/f325820315/model_final_8c3da3.pkl">model</a></td>111</tr>112</tr>113<!-- ROW: cascade_mask_rcnn_mvitv2_l_in21k_50ep -->114 <tr><td align="left"><a href="configs/COCO/cascade_mask_rcnn_mvitv2_l_in21k_50ep.py">MViTv2-L</a></td>115<td align="center">IN21K, sup</td>116<td align="center">0.844</td>117<td align="center">0.157</td>118<td align="center">19.7</td>119<td align="center">55.7</td>120<td align="center">48.3</td>121<td align="center">325607715</td>122<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_mvitv2_l_in21k/f325607715/model_final_2141b0.pkl">model</a></td>123</tr>124</tr>125<!-- ROW: cascade_mask_rcnn_mvitv2_h_in21k_36ep -->126 <tr><td align="left"><a href="configs/COCO/cascade_mask_rcnn_mvitv2_h_in21k_36ep.py">MViTv2-H</a></td>127<td align="center">IN21K, sup</td>128<td align="center">1.655</td>129<td align="center">0.285</td>130<td align="center">18.4*</td>131<td align="center">55.9</td>132<td align="center">48.3</td>133<td align="center">326187358</td>134<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_mvitv2_h_in21k/f326187358/model_final_2234d7.pkl">model</a></td>135</tr>136<!-- ROW: cascade_mask_rcnn_vitdet_b_100ep -->137 <tr><td align="left"><a href="configs/COCO/cascade_mask_rcnn_vitdet_b_100ep.py">ViTDet, ViT-B</a></td>138<td align="center">IN1K, MAE</td>139<td align="center">0.362</td>140<td align="center">0.089</td>141<td align="center">12.3</td>142<td align="center">54.0</td>143<td align="center">46.7</td>144<td align="center">325358525</td>145<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_vitdet_b/f325358525/model_final_435fa9.pkl">model</a></td>146</tr>147<!-- ROW: cascade_mask_rcnn_vitdet_l_100ep -->148 <tr><td align="left"><a href="configs/COCO/cascade_mask_rcnn_vitdet_l_100ep.py">ViTDet, ViT-L</a></td>149<td align="center">IN1K, MAE</td>150<td align="center">0.643</td>151<td align="center">0.142</td>152<td align="center">22.3</td>153<td align="center">57.6</td>154<td align="center">50.0</td>155<td align="center">328021305</td>156<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_vitdet_l/f328021305/model_final_1a9f28.pkl">model</a></td>157</tr>158<!-- ROW: cascade_mask_rcnn_vitdet_h_75ep -->159 <tr><td align="left"><a href="configs/COCO/cascade_mask_rcnn_vitdet_h_75ep.py">ViTDet, ViT-H</a></td>160<td align="center">IN1K, MAE</td>161<td align="center">1.137</td>162<td align="center">0.196</td>163<td align="center">32.9</td>164<td align="center">58.7</td>165<td align="center">51.0</td>166<td align="center">328730692</td>167<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/COCO/cascade_mask_rcnn_vitdet_h/f328730692/model_final_f05665.pkl">model</a></td>168</tr>169</tbody></table>170 171 172### LVIS173 174#### Mask R-CNN175 176<table><tbody>177<!-- START TABLE -->178<!-- TABLE HEADER -->179<th valign="bottom">Name</th>180<th valign="bottom">pre-train</th>181<th valign="bottom">train<br/>time<br/>(s/im)</th>182<th valign="bottom">inference<br/>time<br/>(s/im)</th>183<th valign="bottom">train<br/>mem<br/>(GB)</th>184<th valign="bottom">box<br/>AP</th>185<th valign="bottom">mask<br/>AP</th>186<th valign="bottom">model id</th>187<th valign="bottom">download</th>188<!-- TABLE BODY -->189<!-- ROW: mask_rcnn_vitdet_b_100ep -->190 <tr><td align="left"><a href="configs/LVIS/mask_rcnn_vitdet_b_100ep.py">ViTDet, ViT-B</a></td>191<td align="center">IN1K, MAE</td>192<td align="center">0.317</td>193<td align="center">0.085</td>194<td align="center">14.4</td>195<td align="center">40.2</td>196<td align="center">38.2</td>197<td align="center">329225748</td>198<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/mask_rcnn_vitdet_b/329225748/model_final_5251c5.pkl">model</a></td>199</tr>200<!-- ROW: mask_rcnn_vitdet_l_100ep -->201 <tr><td align="left"><a href="configs/LVIS/mask_rcnn_vitdet_l_100ep.py">ViTDet, ViT-L</a></td>202<td align="center">IN1K, MAE</td>203<td align="center">0.576</td>204<td align="center">0.137</td>205<td align="center">24.7</td>206<td align="center">46.1</td>207<td align="center">43.6</td>208<td align="center">329211570</td>209<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/mask_rcnn_vitdet_l/329211570/model_final_021b3a.pkl">model</a></td>210</tr>211<!-- ROW: mask_rcnn_vitdet_b_75ep -->212 <tr><td align="left"><a href="configs/LVIS/mask_rcnn_vitdet_h_100ep.py">ViTDet, ViT-H</a></td>213<td align="center">IN1K, MAE</td>214<td align="center">1.059</td>215<td align="center">0.186</td>216<td align="center">35.3</td>217<td align="center">49.1</td>218<td align="center">46.0</td>219<td align="center">332434656</td>220<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/mask_rcnn_vitdet_h/332434656/model_final_866730.pkl">model</a></td>221</tr>222</tbody></table>223 224#### Cascade Mask R-CNN225 226<table><tbody>227<!-- START TABLE -->228<!-- TABLE HEADER -->229<th valign="bottom">Name</th>230<th valign="bottom">pre-train</th>231<th valign="bottom">train<br/>time<br/>(s/im)</th>232<th valign="bottom">inference<br/>time<br/>(s/im)</th>233<th valign="bottom">train<br/>mem<br/>(GB)</th>234<th valign="bottom">box<br/>AP</th>235<th valign="bottom">mask<br/>AP</th>236<th valign="bottom">model id</th>237<th valign="bottom">download</th>238<!-- TABLE BODY -->239<!-- ROW: cascade_mask_rcnn_swin_b_in21k_50ep -->240 <tr><td align="left"><a href="configs/LVIS/cascade_mask_rcnn_swin_b_in21k_50ep.py">Swin-B</a></td>241<td align="center">IN21K, sup</td>242<td align="center">0.368</td>243<td align="center">0.090</td>244<td align="center">11.5</td>245<td align="center">44.0</td>246<td align="center">39.6</td>247<td align="center">329222304</td>248<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/cascade_mask_rcnn_swin_b_in21k/329222304/model_final_a3a348.pkl">model</a></td>249</tr>250<!-- ROW: cascade_mask_rcnn_swin_l_in21k_50ep -->251 <tr><td align="left"><a href="configs/LVIS/cascade_mask_rcnn_swin_l_in21k_50ep.py">Swin-L</a></td>252<td align="center">IN21K, sup</td>253<td align="center">0.486</td>254<td align="center">0.105</td>255<td align="center">13.8</td>256<td align="center">46.0</td>257<td align="center">41.4</td>258<td align="center">329222724</td>259<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/cascade_mask_rcnn_swin_l_in21k/329222724/model_final_2b94db.pkl">model</a></td>260</tr>261<!-- ROW: cascade_mask_rcnn_mvitv2_b_in21k_100ep -->262 <tr><td align="left"><a href="configs/LVIS/cascade_mask_rcnn_mvitv2_b_in21k_100ep.py">MViTv2-B</a></td>263<td align="center">IN21K, sup</td>264<td align="center">0.475</td>265<td align="center">0.100</td>266<td align="center">11.8</td>267<td align="center">46.3</td>268<td align="center">42.0</td>269<td align="center">329477206</td>270<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/cascade_mask_rcnn_mvitv2_b_in21k/329477206/model_final_a00567.pkl">model</a></td>271</tr>272</tr>273<!-- ROW: cascade_mask_rcnn_mvitv2_l_in21k_50ep -->274 <tr><td align="left"><a href="configs/LVIS/cascade_mask_rcnn_mvitv2_l_in21k_50ep.py">MViTv2-L</a></td>275<td align="center">IN21K, sup</td>276<td align="center">0.844</td>277<td align="center">0.172</td>278<td align="center">21.0</td>279<td align="center">49.4</td>280<td align="center">44.2</td>281<td align="center">329661552</td>282<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/cascade_mask_rcnn_mvitv2_l_in21k/329661552/model_final_7838a5.pkl">model</a></td>283</tr>284</tr>285<!-- ROW: cascade_mask_rcnn_mvitv2_h_in21k_36ep -->286 <tr><td align="left"><a href="configs/LVIS/cascade_mask_rcnn_mvitv2_h_in21k_50ep.py">MViTv2-H</a></td>287<td align="center">IN21K, sup</td>288<td align="center">1.661</td>289<td align="center">0.290</td>290<td align="center">21.3*</td>291<td align="center">49.5</td>292<td align="center">44.1</td>293<td align="center">330445165</td>294<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/cascade_mask_rcnn_mvitv2_h_in21k/330445165/model_final_ad4220.pkl">model</a></td>295</tr>296<!-- ROW: cascade_mask_rcnn_vitdet_b_100ep -->297 <tr><td align="left"><a href="configs/LVIS/cascade_mask_rcnn_vitdet_b_100ep.py">ViTDet, ViT-B</a></td>298<td align="center">IN1K, MAE</td>299<td align="center">0.356</td>300<td align="center">0.099</td>301<td align="center">15.2</td>302<td align="center">43.0</td>303<td align="center">38.9</td>304<td align="center">329226874</td>305<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/cascade_mask_rcnn_vitdet_b/329226874/model_final_df306f.pkl">model</a></td>306</tr>307<!-- ROW: cascade_mask_rcnn_vitdet_l_100ep -->308 <tr><td align="left"><a href="configs/LVIS/cascade_mask_rcnn_vitdet_l_100ep.py">ViTDet, ViT-L</a></td>309<td align="center">IN1K, MAE</td>310<td align="center">0.629</td>311<td align="center">0.150</td>312<td align="center">24.9</td>313<td align="center">49.2</td>314<td align="center">44.5</td>315<td align="center">329042206</td>316<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/cascade_mask_rcnn_vitdet_l/329042206/model_final_3e81c2.pkl">model</a></td>317</tr>318<!-- ROW: cascade_mask_rcnn_vitdet_h_75ep -->319 <tr><td align="left"><a href="configs/LVIS/cascade_mask_rcnn_vitdet_h_100ep.py">ViTDet, ViT-H</a></td>320<td align="center">IN1K, MAE</td>321<td align="center">1.100</td>322<td align="center">0.204</td>323<td align="center">35.5</td>324<td align="center">51.5</td>325<td align="center">46.6</td>326<td align="center">332552778</td>327<td align="center"><a href="https://dl.fbaipublicfiles.com/detectron2/ViTDet/LVIS/cascade_mask_rcnn_vitdet_h/332552778/model_final_11bbb7.pkl">model</a></td>328</tr>329</tbody></table>330 331Note: Unlike the system-level comparisons in the paper, these models use a lower resolution (1024 instead of 1280) and standard NMS (instead of soft NMS). As a result, they have slightly lower box and mask AP.332 333We observed higher variance on LVIS evalution results compared to COCO. For example, the standard deviations of box AP and mask AP were 0.30% (compared to 0.10% on COCO) when we trained ViTDet, ViT-B five times with varying random seeds.334 335The above models were trained and measured on 8-node with 64 NVIDIA A100 GPUs in total. *: Activation checkpointing is used.336 337 338## Training339All configs can be trained with:340 341```342../../tools/lazyconfig_train_net.py --config-file configs/path/to/config.py343```344By default, we use 64 GPUs with batch size as 64 for training.345 346## Evaluation347Model evaluation can be done similarly:348```349../../tools/lazyconfig_train_net.py --config-file configs/path/to/config.py --eval-only train.init_checkpoint=/path/to/model_checkpoint350```351 352 353## <a name="CitingViTDet"></a>Citing ViTDet354 355If you use ViTDet, please use the following BibTeX entry.356 357```BibTeX358@article{li2022exploring,359 title={Exploring plain vision transformer backbones for object detection},360 author={Li, Yanghao and Mao, Hanzi and Girshick, Ross and He, Kaiming},361 journal={arXiv preprint arXiv:2203.16527},362 year={2022}363}364```365