How does a masked autoencoder (MAE) learn image representations?

A Masked Autoencoder (MAE) learns useful image representations by hiding a large fraction of an unlabeled image, encoding only the visible patches, and using a small decoder to reconstruct the missing pixels. The original MAE architecture from He et al. uses an asymmetric encoder–decoder: the expensive encoder never processes mask tokens, while the lightweight decoder reconstructs a full patch sequence.

How does the original MAE work, step by step?

1. Split the input into patches: Convert a photograph into a sequence of fixed-size image patches, with position information. The original MAE uses a Vision Transformer (ViT) backbone.
2. Randomly mask about 75%: Shuffle patch indices, retain a random visible subset (typically 25%), and record which original positions are missing so they can be restored later.
3. Encode only the visible patches: The ViT encoder sees only the visible patch tokens, with positional information; mask tokens do not enter the encoder. Its contextual embeddings contain the visible image information.
4. Restore the full token layout for the decoder: Map encoded visible tokens into the smaller decoder width; insert learned mask tokens into the hidden positions and unshuffle to reconstruct the original patch order.
5. Add decoder position embeddings: The decoder now knows where each visible or masked token belongs in the image. A lightweight transformer decoder processes the full set.
6. Predict image pixels: The decoder's output head predicts pixel values for each image patch. The unmasked original image provides targets automatically—no human class labels.
7. Train using only masked regions: Compute a masked-patch pixel mean squared error (MSE); zero out contributions from visible patch positions. Update the encoder and decoder using backpropagation.
8. Discard the decoder after pretraining: Keep the encoder for recognition or transfer tasks. It can then process all image patches at inference or fine-tuning time.

End-to-end training flow: Image → Patchify → Random 75% mask → Visible tokens only → ViT encoder → Restore + insert mask tokens + positions → Lightweight decoder → Pixel predictions → Masked-only MSE

Beginner-friendly example: A 64 × 64 image

Imagine a color photograph of a bicycle with 64 × 64 pixels, split into 16 × 16-pixel patches.

Step 1 — Count image patches.

• Patches per side: 64 ÷ 16 = 4.
• Total patches: 4 × 4 = 16, labeled P1 to P16.

Step 2 — Randomly hide 75%.

• Masked patches: 16 × 0.75 = 12.
• Visible patches: 16 − 12 = 4.
• Suppose the visible positions are P1, P6, P11, P16. The model must reconstruct missing bicycle and background content from these four glimpses.
• Encoder processes 4 patch tokens, not 16 patch tokens. The visible-only encoder does not receive 12 fake mask tokens.

Step 3 — Build the decoder input.

• Project the 4 encoder outputs into decoder embedding space.
• Insert 12 learned mask tokens for the hidden patch positions.
• Unshuffle into the original P1 … P16 order; add decoder positional embeddings.
• The lightweight decoder processes all 16 positions and predicts a pixel vector for each patch.

For an RGB 16 × 16 patch, that prediction has 16 × 16 × 3 = 768 pixel-channel values per patch. Thus the decoder predicts 16 × 768 scalar values in this toy setup, although training loss only counts the 12 masked patches.

Numerical example: Masked-only reconstruction loss

For easy arithmetic, look at two of the twelve masked patches, using only two illustrative grayscale pixel values per patch. A real MAE averages errors over all pixels and channels in all masked patches, so the following is a teaching subset, not the full-image loss.

• Masked patch P2: Original pixels [120, 80]; predicted pixels [100, 90].
• Masked patch P3: Original pixels [30, 60]; predicted pixels [40, 50].

Step 1 — Square each prediction error.

• P2: (100 − 120)² = 400; (90 − 80)² = 100.
• P3: (40 − 30)² = 100; (50 − 60)² = 100.

Step 2 — Average within each masked patch.

• MSE(P2) = (400 + 100)/2 = 250.
• MSE(P3) = (100 + 100)/2 = 100.

Step 3 — Average over these two illustrative masked patches.

Illustrative masked-only MSE = (250 + 100)/2 = 175.

Step 4 — Ignore visible-patch errors. Even if the decoder's predicted pixels for a visible patch are imperfect, those errors are not included in the original MAE masked-only objective.

Important distinction: The number 175 is the illustrative subset loss. With the full image, the true training loss averages the per-patch pixel-channel MSE over all 12 masked patches, not just P2 and P3. An implementation may also use per-patch normalized pixel targets, which changes the numerical scale.

Why is the asymmetric encoder–decoder efficient?

• Heavy encoder handles fewer tokens: In the 16-patch example, the encoder operates on 4 visible patch tokens instead of 16. This reduces the most expensive part of pretraining.
• Self-attention comparison: Ignoring any special classification token, the visible patch-to-patch attention-score count is 4² = 16 versus 16² = 256 for all patches—1/16 as many patch attention-score pairs in this toy example.
• Do not confuse this with a 16× end-to-end speedup: Real runtime also includes projections, other layers, classification tokens, decoder work, hardware characteristics and memory overhead.
• Lightweight decoder does the full reconstruction: Only the smaller, shallower decoder sees all patch positions. The original paper reports 3× or greater training acceleration for its reported setups, not a universal multiplier for every model.

Why mask so much? At a high ratio such as 75%, recovering hidden patches becomes challenging enough to encourage reasoning about shapes, structure and context rather than trivial interpolation from nearby pixels. This is an empirical choice from the MAE paper, not a setting guaranteed optimal for every domain.

What does the encoder learn, and how is it used afterward?

• Visual structure: Identifying missing bicycle parts from a few visible regions can encourage the encoder to capture object shape, scene context and spatial relationships.
• Pretraining without manual labels: Missing pixel values supply the learning targets automatically.
• Keep encoder, drop decoder: After pretraining, use the learned ViT encoder on unmasked images. The reconstruction decoder is not required for ordinary downstream recognition.
• Evaluate and fine-tune: Fine-tune for image classification or adapt the backbone for detection and segmentation; frozen-feature linear probing can also assess transfer, but its quality depends on the training recipe.

Benefits, limitations and practical considerations

• Benefits: Scales to unlabeled image collections, uses a simple pixel-reconstruction target, and saves expensive encoder compute by omitting masked tokens.
• Reconstruction versus semantics: A low masked-pixel loss does not guarantee strong object recognition, robust retrieval or useful representations on a new domain.
• Mask-ratio tradeoff: Too little hiding can make the task easy, while too much may remove enough evidence that recovery becomes ambiguous; tune ratios by downstream validation.
• Resolution and patch-size cost: More image patches raise memory and compute, particularly in the decoder that processes all positions.
• Domain sensitivity: Tiny defects, text, medical details or fine boundaries can be lost with unsuitable patch sizes or pixel objectives.
• Implementation checks: Verify that the encoder truly excludes masked patch tokens, the decoder restores positions correctly, and the loss excludes visible positions. Test reconstructions alongside held-out downstream metrics.

Curriculum boundary: Question 273 (Masked Image Modeling) introduced the general hide-and-predict objective and alternative prediction targets. This article focuses on the specific MAE visible-only encoder, mask-token decoder, positional restoration and masked-pixel loss, without revisiting BYOL or DINO architectures.

Key takeaway: MAE pretrains a powerful visual encoder efficiently by encoding only visible patches and delegating full-image masked-pixel reconstruction to a lightweight decoder, which is discarded for downstream tasks.

Research: He et al., *Masked Autoencoders Are Scalable Vision Learners* (CVPR 2022); arXiv paper; official MAE implementation.