Loss Function

Executive Definition: Loss Function — A mathematical function that quantifies the difference between an AI model’s predicted output and the actual true values.

The loss function serves as the objective metric during the training of machine learning models. By calculating a ‘loss’ score, it provides a feedback signal that an optimization algorithm (like gradient descent) uses to adjust the model’s weights. The primary goal of training a neural network is to systematically minimize the loss function over time, resulting in more accurate predictions.

Frequently Asked Questions

What is the difference between loss and accuracy?

Loss is the internal mathematical value the model minimizes during training. Accuracy is a human-readable metric showing the percentage of correct predictions. They are correlated, but not the same.

What loss functions are used for deep learning?

Cross-entropy loss is standard for classification (including LLMs predicting the next token). Mean Squared Error (MSE) is common for regression tasks. Contrastive loss is used for training embedding models.

What happens if a loss function is poorly chosen?

The model will optimize for the wrong behavior. For example, using a standard accuracy-based loss on a highly imbalanced dataset might result in a model that simply predicts the majority class every time.