Why Gradient Descent Zigzags and How Momentum Fixes It

The experiment you described illustrates the sensitivity of the momentum parameter (β) in gradient descent optimization algorithms. Here's a summary of key findings and insights:

Key Findings:

1. Convergence Improvement with Increasing β:

   • As β increases from 0 to around 0.95, the number of steps required for convergence decreases significantly.
   • This improvement is due to better smoothing of oscillations along the slow direction (θ₁) and accumulation of useful momentum in the fast direction (θ₂).

2. Deterioration Beyond Optimal β:

   • When β exceeds 0.95, particularly around 0.99, convergence performance deteriorates sharply.
   • High values of β lead to excessive reliance on past gradients, causing overshooting and instability.

3. Baseline Comparison (β = 0):

   • At β = 0, the optimizer behaves like vanilla gradient descent with no momentum effect.
   • This serves as a reference point for comparing the benefits and drawbacks of different β values.

Insights:

1. Role of Momentum:
   • Momentum helps in smoothing out oscillations along the slow direction (θ₁) by canceling them out over time.

Read the full article at MarkTechPost

---

Want to create content about this topic? Use Nemati AI tools to generate articles, social posts, and more.