Textbook
The differentiable objective $L:\mathbb R^d\to\mathbb R$ has current parameters $\theta_t$and a chosen step size $\eta>0$. One gradient-descent update is:
$$\theta_{t+1}:=\theta_t-\eta\nabla L(\theta_t).$$
The index t counts iterations. At each iteration, the gradient is recomputed at the current point, scaled, and subtracted. This specifies the next state; it does not establish arrival at a global minimum.
For a nonzero gradient, the directional derivative along $-\nabla L(\theta_t)$ equals $-\|\nabla L(\theta_t)\|_2^2<0$. A sufficiently small positive step therefore decreases the objective. The length of a finite step matters: the local guarantee does not cover arbitrary $\eta$.
For $L(\theta)=(\theta-2)^2$, the derivative is $2(\theta-2)$. From $\theta_0=0$, choosing $\eta=1/4$ gives next parameter 1 and reduces the objective from 4 to 1. Choosing $\eta=2$ gives 8 and objective 36. Convergence requires separate assumptions about the function and step choices.
For this quadratic, the update can be expressed through parameter error: $\theta_{t+1}-2=(1-2\eta)(\theta_t-2)$. A constant $0<\eta<1$ gives a multiplier of magnitude below one, so distance to 2 decreases geometrically. At $\eta=1$, the error changes sign without shrinking; at $\eta>1$, a nonzero error grows in magnitude. These conclusions concern this particular quadratic.
Definitions and notation: Mathematics for Machine Learning.
Related concepts: The gradient: a vector of local changes, A function’s minimum: the smallest attained value.
In plain language
A downhill route starts from the local slope: each step points against the gradient. The slope is evaluated again at the new position, so one old arrow cannot guide the entire route. Movement depends on both the gradient and step size; an oversized leap can cross the valley onto a higher slope. The direction was right and the jump length failed: an energetic stride does not establish progress toward the target.
Example
The constructed objective is $(\theta-2)^2$. The example shows two short steps with $\eta=1/4$, then separately a large step from the initial point.
Gradient descent: stepping against the gradient
Initial state
At 0, the gradient is −4 and the objective is 4.
Step 1 of 4: Initial state
Read all steps
Initial state
At 0, the gradient is −4 and the objective is 4.
- θ; gradient; objective
- 0current−4current4
First update
Subtracting one quarter of the gradient gives parameter 1.
- θ; gradient; objective
- 1current−2current1included
Second update
The new gradient is −2; the next parameter is 1.5.
- θ; gradient; objective
- 1.5current−1current0.25included
An oversized step
The initial point is 0 again. A step size of 2 gives parameter 8 and objective 36.
- θ; step size; objective
- 8excluded2current36excluded
At $\eta=1/4$, the error multiplier is $1/2$. The initial deviation $-2$ becomes $-1$ and then $-0.5$; the corresponding objective values are 4, 1, and 0.25. At the large step size $\eta=2$, the multiplier is $-3$: the deviation switches sides and triples. That explains the final objective 36 directly from the update. Both experiments subtract the gradient; that minus sign alone cannot prevent divergence.
The gradient and objective are recomputed after each move. A direction alone does not replace step-size control and convergence assumptions.