What is a Swin Transformer and why does it use shifted windows?

Swin Transformer (Shifted Window Transformer) is a hierarchical vision Transformer that computes self-attention inside small, non-overlapping windows rather than across every image patch at once. It shifts the window boundaries between successive blocks so patches in neighboring windows can exchange information.

How do the shifted windows work?

• First block (W-MSA): Partition the feature map into windows and calculate attention only among tokens inside each window.
• Next block (SW-MSA): Shift the window partition by about half a window. Tokens that were separated by an earlier window boundary can now attend to one another.
• Repeat: Alternating ordinary and shifted windows lets information spread beyond a single window over multiple layers.
• Build a hierarchy: Deeper stages merge neighboring patches, creating lower-resolution, richer feature maps useful for object detection and segmentation.

Simple numerical example

Imagine an 8 × 8 grid of 64 patch tokens, with a 4 × 4 window size.

• Ordinary windows: The grid becomes 2 × 2 = 4 windows, each containing 4 × 4 = 16 tokens.
• Local attention scores: Each window computes a 16 × 16 attention-score matrix. Across four windows that is 4 × 256 = 1,024 token-pair scores per attention head.
• Global attention comparison: One full attention matrix over 64 tokens would contain 64 × 64 = 4,096 token-pair scores per head.
• Shifted windows: The next block shifts the partition by 2 tokens vertically and horizontally. A token near the old boundary can now interact with tokens from the neighboring old window.

These score counts are illustrative; they exclude projections and other computation. Practical implementations use cyclic shifts and attention masks to prevent incorrect wraparound connections.

Why not keep windows fixed?

Fixed windows are efficient, but neighboring windows would remain isolated within each such attention layer. Shifting connects those regions without paying for global attention at every layer. With a fixed window size, the attention computation scales approximately linearly with the total number of image tokens, rather than quadratically as in global attention.

Practical use: For a street-scene image, details near the boundary between a car and the road can exchange information across windows. Hierarchical features also make Swin useful for image classification, object detection, and semantic segmentation.

Key takeaway: Swin combines local attention for efficiency, shifted windows for cross-window communication, and multi-scale features for vision tasks. It does not compute global attention over the whole image in each block.

Research: Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows — Liu et al. (ICCV 2021).