Textbook
Underfitting occurs when a fitted model does not adequately capture a useful pattern. It can result from an overly restricted model class or insufficient fitting. Substantial training error therefore calls for checking both expressiveness and the learning procedure.
A simple expressiveness limit is the class of constant classifiers $h_c(x)=c$, with $c\in\{0,1\}$. If both labels occur in the training data, no classifier in this class can answer every object correctly, regardless of how often the constant $c$ is selected.
With underfitting, errors can remain high on both training and representative held-out data. Two similar numbers alone do not establish the diagnosis: noisy labels, insufficient features, or a genuinely optimal simple rule are also possible. Diagnosis first requires establishing what pattern can be recovered.
When the restriction is truly the model class, additional flexibility may help. This does not promise that greater complexity is always better: the new model must also be evaluated on held-out data. Overfitting is a different failure mode; this article concerns failure to express a useful rule.
Repeating training within the same restricted class cannot remove errors the class is fundamentally unable to express. Conversely, poor parameter fitting cannot automatically be blamed on the class: a suitable rule may exist but remain undiscovered. Diagnosis separates two questions: can the class describe the pattern, and did training approach the result available within that class? High error motivates this investigation without, by itself, explaining its cause.
Related concepts: Overfitting.
In plain language
A thermometer with two marks cannot measure tenths of a degree merely because its scale receives a stern stare. Similarly, a constant model cannot represent distinctions between objects: every input receives the same answer. Repeated fitting does not add expressive power to a restricted class. A coarse scale and poor calibration nevertheless remain different causes of bad results. The analogy separates capacity from adjustment without replacing checks of features, labels, and the fitting procedure.
Example
In this toy task, the true label is 0 for $x<2$ and 1 for $x\ge2$. Training coordinates are 0, 1, 2, 3; held-out coordinates are 0.5 and 2.5. The initial model is restricted to the constant answer 0.
Underfitting
Constant training answers
The model predicts 0 everywhere and misses coordinates 2 and 3.
Step 1 of 3: Constant training answers
Read all steps
Constant training answers
The model predicts 0 everywhere and misses coordinates 2 and 3.
- Coordinates
- 0123
- Predictions
- 0 ✓included0 ✓included0 ✗excluded0 ✗excluded
The same limit on new objects
The answer is correct at 0.5 and incorrect at 2.5. New data do not repair the inability to change labels.
- Coordinates
- 0.52.5
- Predictions
- 0 ✓included0 ✗excluded
Input-dependent answers
An illustrative threshold at 2 can express the specified pattern. It makes no errors on the listed points.
- Training
- 0 ✓included0 ✓included1 ✓included1 ✓included
- Validation
- 0 ✓included1 ✓included
Changing the constant answer from zero to one does not solve the toy task: the first two training points become wrong instead of the last two. Even the best constant choice therefore leaves two errors out of four. The threshold rule specifically removes the ban on different answers on opposite sides of a boundary. Zero errors on the displayed points illustrate that restriction’s removal in the stated model, without promising zero error on arbitrary real data.
Source of the original notes: lecture notes.