Weight Decay

Executive Definition: Weight Decay — A regularization technique applied during neural network training to penalize large weight values, preventing models from overfitting.

Weight decay adds a mathematical penalty to the loss function proportional to the size of the weights. This discourages the model from relying too heavily on any single feature or learning overly complex patterns that fail to generalize to unseen data.

Frequently Asked Questions

How does weight decay prevent overfitting?

By penalizing large weights, it forces the model to find simpler solutions that generalize better. Complex, overfitted models tend to have extreme weight values.

What weight decay value should I use?

Common values range from 0.01 to 0.1. Start with 0.01 for fine-tuning LLMs. Higher values provide stronger regularization but may prevent the model from learning complex patterns.

Is weight decay the same as L2 regularization?

They are closely related but not identical. In standard SGD they are equivalent. With adaptive optimizers like Adam, decoupled weight decay (AdamW) produces different and generally better results.