Textbook
For a scalar function $f:\mathbb R^d\to\mathbb R$, the gradient collects first partial derivatives. In column-vector notation: $\nabla f(x)=(\partial f/\partial x_1,\ldots,\partial f/\partial x_d)^{\mathsf T}$. Other books use a row, so the dimensions depend on the convention.
If $f$ is differentiable at $x$, then $f(x+h)=f(x)+\nabla f(x)^{\mathsf T}h+r(h)$ with $r(h)/\|h\|_2\to0$ as $h\to0$. The gradient gives the linear part; the remainder shrinks relative to step length. Partial derivatives alone do not establish this; their continuity in a neighborhood is sufficient.
Among directions $u$ with $\|u\|_2=1$, the directional derivative is $\nabla f(x)^{\mathsf T}u\le\|\nabla f(x)\|_2$. For a nonzero gradient, equality holds at $u=\nabla f(x)/\|\nabla f(x)\|_2$. This is fastest local growth specifically in Euclidean geometry.
For $f(x_1,x_2)=x_1^2+3x_2^2$, the gradient at $(1,2)$ is $(2,12)^{\mathsf T}$. Its negative gives a local descent direction, but a finite step requires length control. A zero gradient alone does not establish a minimum.
A local prediction requires consistent signs: increasing a coordinate with a negative partial derivative decreases the linear change. Components can be compared only in the declared coordinate units. Changing those units changes the gradient’s numerical components. The fastest-growth statement fixes the norm of the step; without that restriction, step length could be increased without bound.
Definitions and notation: Mathematics for Machine Learning.
Related concepts: Partial derivatives: changing one coordinate, Dot Products: Two Vectors Become One Number.
In plain language
The gradient describes how small turns of several knobs combine into one change in output. A coefficient of 12 for the second knob means a strong response at the current point and in the chosen units, not permanent superiority. Among steps of equal Euclidean length, the gradient points toward the greatest local increase. The arrow describes the immediate surroundings; a kilometer-long leap based on it remains ambitious navigation.
Example
The constructed example uses the function $x_1^2+3x_2^2$, point $(1,2)$, and small step $(0.01,-0.01)$. The linear prediction is compared with the exact change.
The gradient: a vector of local changes
Partial derivatives
At (1,2), the local coefficients are 2 and 12.
Step 1 of 4: Partial derivatives
Read all steps
Partial derivatives
At (1,2), the local coefficients are 2 and 12.
- Gradient
- 2current12current
A small step
The first coordinate increases by 0.01 and the second decreases by 0.01.
- Step
- 0.01current−0.01current
- Contributions
- 0.02included−0.12excluded
Linear prediction
The dot product gives change −0.10 and predicted value 12.90.
- Initial; prediction
- 1312.90current
Prediction remainder
The exact value 12.9004 differs by 0.0004, from the squared step terms.
- Prediction; exact; remainder
- 12.9012.9004included0.0004current
For the chosen step, the first contribution is $0.02$ and the second is $-0.12$, making the overall prediction negative. The exact remainder $0.0004$ comes from $0.01^2+3(-0.01)^2$; this function has no mixed term. Halving both step components halves the linear contribution but quarters the remainder. The example thus shows why approximation improves for small steps rather than merely displaying two similar numbers.
A gradient has meaning together with its point, dimensions, geometry, and the range where its linear approximation is accurate.