Gradient Descent: 🗻Going downhill Pt 2

Flavors 🍧of Gradient Descent.

Photo by Jarritos Mexican Soda on Unsplash

🗻Gradient Descent

Gradient Descent is optimization algorithm that tweaks parameters iteratively in order to minimize a cost function J(θ) 📉.

📑 Types of Gradient Descent

There are three popular types of gradient descent that mainly differ in the amount of data they use.

1. Batch Gradient Descent

In Batch Gradient Descent also knows as Vanilla Gradient Descent, we sum over the entire training examples of dataset and then make update parameters such that to minimize the loss function.

2. Mini Batch Gradient Descent

In Mini-Batch Gradient Descent instead of summing over the entire set of training examples and passing it to algorithm we take a small batch of training examples and update parameters based on this small batch

3.Stochastic Gradient Descent

In Stochastic Gradient Descent we get a single training example from the dataset and then update parameters based on this single training example to minimize the loss function.

Other types of gradient descent:

* Adaptive Gradient Descent

In Adaptive Gradient Descent, we choose different alpha (α) values for each step.

* Momentum Gradient Descent

In Momentum Gradient Descent we keep track of previously computed gradient to give us momentum and accelerate the minimization process.The momentum is given by ρvt.

SGD Momentum(blue) vs SGD(black)

Check out: Gradient Descent: Going Downhill pt 1