Textbook
Overfitting occurs when a model adapts to accidental features or noise in its training sample that do not recur in new data. Familiar-object performance improves without a corresponding improvement on future objects.
One characteristic pattern is low training error and substantially higher held-out error. The comparison is meaningful when held-out observations represent the intended future use and did not guide fitting. The gap alone does not prove the cause: the samples may differ in composition too.
A flexible boundary may accommodate every labelled point, including an unusual observation, while representing the general pattern less well. An outlier is an atypical observation, not necessarily a labelling mistake: a wealthy borrower can still default. An inconvenient case cannot be discarded simply because the model dislikes it.
Generalisation means retaining performance on new objects. Held-out data assess that ability. Model complexity is selected using validation; the final test is reserved for the completed choice. There is no universal error-gap threshold here that proves overfitting by itself.
Related concepts: Training, validation, and test splits, Classification error.
Errors are compared for a fixed prediction rule on different observation sets. Adjusting the model after viewing held-out answers changes the comparison’s meaning: those answers have now influenced the decision. On a small sample, one error changes the fraction substantially, so differences require cautious interpretation. Zero errors on a few new objects likewise do not establish performance across every future case.
In plain language
Memorized answers to six exam questions can reproduce even the lecturer’s typo perfectly. Repeating those questions mainly tests memory, while a new question may reveal the absence of a general solving method. A model can likewise collect familiar exceptions that turn out to be souvenirs once the examples change. One difficult new question does not establish overfitting; another subject or different data may explain it. The analogy illustrates a memorization risk rather than diagnosing the problem from one awkward answer.
Example
This toy comparison evaluates two models on the same six training and four validation objects. These numbers illustrate a possible overfitting pattern; they are not results from a real experiment.
Overfitting
Perfect training answers
The flexible model answers all six familiar objects correctly. This describes only its training sample.
Step 1 of 3: Perfect training answers
Read all steps
Perfect training answers
The flexible model answers all six familiar objects correctly. This describes only its training sample.
- Training
- A ✓includedB ✓includedC ✓includedD ✓includedE ✓includedF ✓included
Validation results
On four new objects, the same model makes three mistakes. Zero training error did not carry over.
- Validation
- G ✗excludedH ✓includedI ✗excludedJ ✗excluded
A more constrained model
Another model makes one training mistake and one validation mistake. Validation prefers it in this example.
- Training
- 1 of 6excluded
- Validation
- 1 of 4current
The flexible model is error-free on six familiar objects but misses three of four validation objects. The simpler model makes a training error yet has validation fraction $1/4$, below $3/4$. Under the displayed selection criterion, this favors the simpler model. It does not establish future accuracy percentages, however: four observations illustrate checking unfamiliar data rather than exhausting that assessment.
Source of the original notes: lecture notes.