What is deformable convolution?

Deformable convolution is a CNN operation that learns where to sample input features instead of always using a fixed spatial grid. It helps a convolutional filter adapt to objects with different shapes, poses, and geometric distortions.

How does it work?

• Ordinary `3×3` convolution: Samples nine positions on a fixed 3×3 grid around each output location.
• Deformable `3×3` convolution: Predicts a horizontal and vertical offset for each of the nine positions, then samples at those shifted locations.
• A separate convolution predicts the offsets from the feature map. The offsets and filter weights are learned together during training.
• When a shifted location lies between pixels, bilinear interpolation estimates the feature value there.

Simple practical example

Imagine detecting a curved snake in a photograph. A regular 3×3 filter always samples a square pattern, even when parts of the snake curve away from that pattern.

A deformable filter can learn to shift some of its nine sampling points toward the snake's body. For instance, one point that would sample at (x + 1, y) could learn an offset of (1, 0) and instead sample at (x + 2, y). Other points can move in different directions.

This is learned from training data—the model is not explicitly told which pixels contain the snake.

When is it useful?

• Object detection: Accommodates objects with varying shapes, poses, and scales.
• Semantic and instance segmentation: Adapts sampling to irregular regions and object boundaries.

How is it different from dilated convolution? Dilation spaces filter positions apart by a fixed pattern. Deformable convolution learns input-dependent offsets that can vary by location.

Trade-off: Predicting offsets and performing interpolation add complexity and computational overhead. It is not automatically better than standard convolution for every task.

Research: Deformable Convolutional Networks (Dai et al., ICCV 2017). Deformable ConvNets v2 additionally introduces learned modulation to control the contribution of sampling points.