Your 1B parameter proxy model trains perfectly with a 1.2e-4 learning rate. You scale the model to 70B, and the training immediately explodes. What’s the most likely reason and how do you fix it without running a new, expensive hyperparameter sweep?

The first suspect is that the 1B model's learning rate and optimizer setup did not transfer safely to the 70B model. A peak learning rate of 1.2 × 10⁻⁴ can be perfectly reasonable for one configuration yet produce overly aggressive parameter updates in another—especially if initialization, normalization, model width/depth, global batch size, or optimizer parameter groups also changed.

Do not conclude that 70B models must always use a lower learning rate. Model parameter count alone does not determine an optimal learning rate, and exploding training can also result from numerical overflow or an implementation bug. The efficient approach is diagnose first, then run a short, controlled recovery experiment, rather than launching a large hyperparameter sweep.

1. Why might the same learning rate become unstable?

Scaling a model from 1B to 70B parameters changes much more than the number of weights:

• Parameter/update scale: Different layer widths, depths, initialization conventions, and residual paths change activation and gradient statistics.
• Optimizer behavior: Adam/AdamW adapt updates through moment estimates. Their effective per-parameter step sizes are not simply learning rate multiplied by raw gradient.
• Batch and schedule: Global batch size, gradient accumulation, warm-up duration, scheduler boundaries, and token budget may have changed along with model size.
• Precision and distributed execution: BF16/FP16 overflow handling, reduction errors, optimizer-state precision, and shard synchronization can turn a working small run into a failing large run.
• Architecture: Residual scaling, normalization placement and epsilon, and initialization all affect stability.

Research on LLM training instability finds that sub-layer and shortcut scaling can materially affect loss spikes. There is no universal safe formula such as "divide learning rate by 70" when parameter count grows 70-fold.

2. Beginner-friendly numerical illustration

Consider a scalar SGD thought experiment where an example parameter sees a larger gradient after a configuration change. This is *not* a model of AdamW's full adaptive update:

Original learning rate: η = 0.00012

1B example:
    gradient g = 0.02
    update magnitude |ηg| = 0.0000024

70B example (hypothetical 8× larger local gradient):
    gradient g = 0.16
    update magnitude |ηg| = 0.0000192

With η = 0.00003 for the second example:
    update magnitude = 0.0000048

Under these invented gradient values, keeping the same learning rate produces an 8× larger scalar SGD update; a 4× smaller trial rate reduces it. The arithmetic illustrates sensitivity, not a claim that 70B gradients are inherently eight times bigger.

With AdamW, inspect the actual optimizer step relative to parameter norms, such as ||ΔW|| / ||W||, as well as gradients. Clipping a raw gradient can help protect against spikes, but it does not guarantee small AdamW updates or fix an unstable architecture.

3. What should the engineer check first?

Before changing any setting, capture the first failing step:

1. Loss and gradient norms: Are norms rising gradually, spiking suddenly, or becoming NaN/Inf on the first backward pass? Log norms per layer as well as the global norm.
2. Actual learning-rate schedule: Is the 70B run accidentally starting at the peak 1.2e-4 without warm-up? Is a resumed checkpoint incorrectly resetting the scheduler?
3. Optimizer and effective step: Confirm AdamW betas, epsilon, weight decay, learning-rate groups, gradient accumulation, and optimizer-state initialization. Record weight/update norm ratios where possible.
4. Data and loss: Inspect the same failing batch for bad masks, unexpected long sequences, corrupt labels, and loss-normalization mistakes.
5. Numerical/distributed correctness: Look for overflow, non-finite optimizer moments, gradients reduced or scaled twice, mixed-precision misconfiguration, and inconsistent global batch size.
6. Architecture and initialization: Check residual scaling, normalization type/placement, embedding and final-projection initialization, especially if width or depth has changed.

Diagnosis matters: If the very first forward loss is non-finite, a smaller learning rate alone is unlikely to fix the underlying issue.

4. How do you recover without a costly sweep?

A. Establish a safe baseline. Reproduce the issue with a few fixed batches and a short training window. Check finite forward activations and loss before the optimizer step.

B. Make the smallest controlled change. If logs implicate optimizer-update instability, run one or two short trials with the same data and setup at illustrative peak rates of 6e-5 (half) and/or 3e-5 (quarter), with a sensible warm-up. These values are diagnostic starting points, not guaranteed rates for 70B.

C. Clip gradient spikes if appropriate. Global-norm clipping—for example max_norm=1.0, only after checking gradient scale and precision—is a defensive measure. PyTorch's clip_grad_norm_ returns the total norm and supports error_if_nonfinite=True.

PYTHON
# Simplified PyTorch step; adapt for distributed or mixed-precision training.
optimizer.zero_grad(set_to_none=True)
loss = model(**batch).loss
loss.backward()

norm = torch.nn.utils.clip_grad_norm_(
    model.parameters(), max_norm=1.0,
    error_if_nonfinite=True
)
optimizer.step()
scheduler.step()

For scaled mixed-precision gradients, unscale before clipping, and in distributed/sharded training use the framework's global clipping/optimizer interface so the result reflects all relevant shards.

D. Validate stability and learning progress. Compare loss, clipping frequency, layerwise norms, optimizer step/weight ratios, and throughput across the short trials. A run that stays finite but learns too slowly is not automatically a successful fix.

E. Resume a long run only after the signals look healthy. Restore or reinitialize optimizer/scheduler state deliberately; do not blindly resume from a checkpoint containing non-finite state.

5. Can we transfer learning rates from smaller models reliably?

Yes, under suitable methods—but not by default. Maximal Update Parametrization (μP) and μTransfer are research-backed techniques designed so that many hyperparameters can transfer more reliably as network width changes. Greg Yang and colleagues demonstrated small-model tuning transferred to larger Transformer settings.

Crucial constraints:

• μP requires the correct parameterization, initialization, per-tensor scaling, and optimizer configuration. Merely naming a proxy model "μP" or copying 1.2e-4 does not make transfer valid.
• Transfer evidence for width does not automatically guarantee transfer when depth, architecture, context length, optimizer, or data also change.
• A model trained under ordinary parameterization cannot be assumed to inherit μP's properties retroactively.

If the 1B proxy and 70B target were deliberately designed for μTransfer, first verify their configuration and parameter groups; if not, use brief evidence-driven LR trials rather than assuming zero-shot transfer.

6. How is this different from the earlier Post-LN vs Pre-LN question?

The earlier article "Why can post-norm Transformers become unstable when training very deep models?" focuses on normalization placement and residual gradient paths. Here the central issue is transferring training hyperparameters and optimizer behavior across model scales. A Pre-Norm architecture can still diverge with an inappropriate update scale or a numerical bug.

Interview-ready answer

> "I'd first suspect that the 1B proxy's LR and update scale do not safely transfer to 70B, but I would not assert that 1.2e-4 is universally too high. I'd inspect the first bad step, gradients by layer, LR warm-up, AdamW state, actual parameter-update ratios and precision/distributed correctness. If it looks like an update-scale problem, I'd run a couple of short lower-LR/warm-up trials, consider validated global-norm clipping, and compare stable learning—not just whether loss stops becoming NaN. For systematic hyperparameter transfer, I'd use a correctly implemented μP/μTransfer setup rather than copy rates between ordinary parameterizations."

In short: The 1B optimum is not automatically the 70B optimum. Diagnose the failing step, validate update scales and training-system correctness, make a few controlled adjustments, and only then commit to the full-scale training run.

References: Yang et al., *Tensor Programs V: μTransfer*; Microsoft Research: Maximal Update Parametrization; Takase et al., *Spike No More: Stabilizing LLM Pre-training*; PyTorch: clip_grad_norm_.