Gradient Descent

Executive Definition: Gradient Descent — An optimization algorithm used to minimize the error of a machine learning model by iteratively adjusting its parameters.

Gradient descent computes the gradient of a loss function and takes steps in the opposite direction to find the minimum error. The size of these steps is controlled by the learning rate. It is the fundamental optimization method for training neural networks and many other machine learning models.

Frequently Asked Questions

What variants of gradient descent exist?

Common variants include Batch Gradient Descent, Stochastic Gradient Descent (SGD), and Mini-batch SGD. Advanced optimizers like Adam, AdaGrad, and RMSprop adapt the learning rate dynamically.

What is a learning rate?

The step size taken during each iteration of gradient descent. A high learning rate may overshoot the minimum, while a low learning rate causes slow convergence.

Can gradient descent get stuck?

Yes, it can converge to local minima or saddle points instead of the global minimum. Techniques like momentum, learning rate scheduling, and adaptive optimizers help mitigate this.