README.md

Deformable Convolutional Networks

Abstract

Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in its building modules. In this work, we introduce two new modules to enhance the transformation modeling capacity of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the effectiveness of our approach on sophisticated vision tasks of object detection and semantic segmentation.

Citation

@inproceedings{dai2017deformable,
  title={Deformable Convolutional Networks},
  author={Dai, Jifeng and Qi, Haozhi and Xiong, Yuwen and Li, Yi and Zhang, Guodong and Hu, Han and Wei, Yichen},
  booktitle={Proceedings of the IEEE international conference on computer vision},
  year={2017}
}

@article{zhu2018deformable,
  title={Deformable ConvNets v2: More Deformable, Better Results},
  author={Zhu, Xizhou and Hu, Han and Lin, Stephen and Dai, Jifeng},
  journal={arXiv preprint arXiv:1811.11168},
  year={2018}
}

Results and Models

Backbone	Model	Style	Conv	Pool	Lr schd	Mem (GB)	Inf time (fps)	box AP	mask AP	Config	Download
R-50-FPN	Faster	pytorch	dconv(c3-c5)	-	1x	4.0	17.8	41.3		config	model | log
R-50-FPN	Faster	pytorch	mdconv(c3-c5)	-	1x	4.1	17.6	41.4		config	model | log
*R-50-FPN (dg=4)	Faster	pytorch	mdconv(c3-c5)	-	1x	4.2	17.4	41.5		config	model | log
R-50-FPN	Faster	pytorch	-	dpool	1x	5.0	17.2	38.9		config	model | log
R-50-FPN	Faster	pytorch	-	mdpool	1x	5.8	16.6	38.7		config	model | log
R-101-FPN	Faster	pytorch	dconv(c3-c5)	-	1x	6.0	12.5	42.7		config	model | log
X-101-32x4d-FPN	Faster	pytorch	dconv(c3-c5)	-	1x	7.3	10.0	44.5		config	model | log
R-50-FPN	Mask	pytorch	dconv(c3-c5)	-	1x	4.5	15.4	41.8	37.4	config	model | log
R-50-FPN	Mask	pytorch	mdconv(c3-c5)	-	1x	4.5	15.1	41.5	37.1	config	model | log
R-101-FPN	Mask	pytorch	dconv(c3-c5)	-	1x	6.5	11.7	43.5	38.9	config	model | log
R-50-FPN	Cascade	pytorch	dconv(c3-c5)	-	1x	4.5	14.6	43.8		config	model | log
R-101-FPN	Cascade	pytorch	dconv(c3-c5)	-	1x	6.4	11.0	45.0		config	model | log
R-50-FPN	Cascade Mask	pytorch	dconv(c3-c5)	-	1x	6.0	10.0	44.4	38.6	config	model | log
R-101-FPN	Cascade Mask	pytorch	dconv(c3-c5)	-	1x	8.0	8.6	45.8	39.7	config	model | log
X-101-32x4d-FPN	Cascade Mask	pytorch	dconv(c3-c5)	-	1x	9.2		47.3	41.1	config	model | log
R-50-FPN (FP16)	Mask	pytorch	dconv(c3-c5)	-	1x	3.0		41.9	37.5	config	model | log
R-50-FPN (FP16)	Mask	pytorch	mdconv(c3-c5)	-	1x	3.1		42.0	37.6	config	model | log

Notes:

• dconv and mdconv denote (modulated) deformable convolution, c3-c5 means adding dconv in resnet stage 3 to 5. dpool and mdpool denote (modulated) deformable roi pooling.
• The dcn ops are modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch, which should be more memory efficient and slightly faster.
• (*) For R-50-FPN (dg=4), dg is short for deformable_group. This model is trained and tested on Amazon EC2 p3dn.24xlarge instance.
• Memory, Train/Inf time is outdated.