Textbook
The $n\ge1$ inputs $x_i\in\mathbb R^d$ have numerical targets $y_i$. The linear model $f_\theta(x)=\theta^{\mathsf T}x$ uses parameters $\theta\in\mathbb R^d$. For fixed $\lambda>0$, quadratic L2 regularization adds a penalty $\lambda\|\theta\|_2^2$ to mean squared error.
$$\theta^*\in\operatorname*{argmin}_{\theta\in\mathbb R^d}\left[\frac1n\sum_{i=1}^n(\theta^{\mathsf T}x_i-y_i)^2+\lambda\|\theta\|_2^2\right].$$
The first term measures agreement with observations; the second raises the objective for large parameters. Lambda controls the penalty’s weight and is held fixed within this optimization. Here every parameter coordinate is penalized; a separate intercept needs an explicit penalty convention.
For this quadratic problem over all $\mathbb R^d$, $\lambda>0$ gives a unique attained minimum: the penalty makes the objective strictly convex and grow without bound as the parameter norm increases. This conclusion does not automatically extend to arbitrary regularized models.
In the scalar case $x_1=1$, $y_1=2$, $\lambda=1$, the objective is $(\theta-2)^2+\theta^2=2(\theta-1)^2+2$. The optimizer shifts from 2 to 1. The penalty changes the problem itself; its presence alone does not guarantee quality on new data.
The norm is squared in this objective: $\|\theta\|_2^2=\sum_j\theta_j^2$. Positive and negative coefficients of equal magnitude incur equal penalties. Rescaling a feature changes the coefficient required for the same prediction and therefore changes the penalty. Feature units belong to the problem specification; coefficient magnitudes from different scales cannot be compared solely as bare numbers.
Definitions and notation: Mathematics for Machine Learning; Deep Learning: Notation.
Related concepts: Argmin: the set of minimizing inputs, Vector Norms: Choosing a Notion of Length.
In plain language
A close fit to data can come at the price of large coefficients. L2 regularization adds a second bill for their squared magnitudes, so the objective accounts for both expenses. The penalty weight sets the balance without forbidding large parameters or guaranteeing quality on new data. Changing a feature’s units can change the required coefficient and penalty while preserving its effect. The cashier follows the formula faithfully without noticing that centimeters have dressed up as meters, so scales matter.
Example
The constructed example has one input $x=1$, target $y=2$, and $\lambda=1$. The example compares error, penalty, and total at several parameter values.
L2 regularization: penalizing parameter size
Zero parameter
Parameter 0 pays no norm penalty but incurs squared error 4.
Step 1 of 4: Zero parameter
Read all steps
Zero parameter
Parameter 0 pays no norm penalty but incurs squared error 4.
- θ; error; penalty; total
- 0current4excluded0included4
Exact fit
Parameter 2 removes the fitting error but pays penalty 4.
- θ; error; penalty; total
- 2current0included4excluded4
The compromise
Parameter 1 gives one unit of each cost, total 2.
- θ; error; penalty; total
- 1current1included1included2included
Optimality argument
The completed square is nonnegative, so the objective is never below 2.
- Minimizer; minimum
- 1included2included
Checking three points alone would not exclude a better parameter between them. The expression $2(\theta-1)^2+2$ closes that gap for every real $\theta$: its first term is nonnegative and vanishes only at 1. Keeping the single observation but allowing any $\lambda>0$ gives optimum $\theta=2/(1+\lambda)$. This shows how increasing the penalty moves the solution toward zero in this particular illustrative problem.
Training error, the regularized objective, and quality on new inputs are distinct: this problem optimizes only its stated sum.