Textbook
A hyperparameter specifies a learning algorithm or model variant and is not fitted by the same procedure as the model’s learned parameters. Linear-model weights are fitted using training data, for example, while the neighbour count in k-nearest neighbours is chosen as an algorithm setting.
For a finite candidate set $K$, each $k\in K$ defines a model $h_k$ built using training data only. Candidate selection minimizes error on the same validation sample. A tie-breaking rule is fixed in advance.
After selection, the final test evaluates the chosen candidate rather than guiding another search. Repeatedly adapting to a competition’s public score likewise turns it into a selection signal. A private test portion matters because the public score no longer supplies an independent final check.
The original lecture compares $k=1$ and $k=15$; its illustrated test-error curve has a minimum at $k=7$. This describes that example, not a universal value or a recommendation to tune on the test. Validation provides the selection data for a particular task.
Related concepts: Training, validation, and test splits, The k-nearest neighbours method.
Candidates must use the same error criterion and data-preparation procedure. Otherwise a score difference cannot be attributed solely to the hyperparameter. A validation score also depends on a finite sample: winning on it does not prove optimal performance over the future population. The selected setting records the procedure’s decision; an independent test evaluates that decision, including the selection already made.
In plain language
Hyperparameter selection resembles fitting shoes: comparable conditions help choose among available pairs, and a walk after purchase evaluates the choice. If every walk changes the choice again, it has become part of the fitting. Seven neighbors from another task offer as little justification as another person’s size ten shoes. Validation selects a setting on comparable data, while a separate test evaluates the frozen candidate. Neither a fitting room nor a finite sample guarantees success on every future walk.
Example
The toy candidates $k=1,3,5$ use the same training data and five shared validation objects. The errors are constructed to illustrate selection, not measured on a real task.
Hyperparameter selection
Validation comparison
The candidates make 3, 1, and 2 mistakes out of five. Zero training error for the first candidate does not determine selection.
Step 1 of 3: Validation comparison
Read all steps
Validation comparison
The candidates make 3, 1, and 2 mistakes out of five. Zero training error for the first candidate does not determine selection.
- k
- 135
- Validation
- 3/51/5current2/5
- Training
- 0/101/103/10
Selected k
The lowest validation error belongs to k = 3. The setting is chosen before opening the test.
- Candidates
- 1excluded3included5excluded
- Chosen
- k = 3included
One final test
The selected model makes two mistakes on ten illustrative test objects. That is the final estimate, not a reason to retrospectively choose k = 5.
- Fixed
- k = 3included
- Test
- 2/10current
All three candidates in the demonstration face the same five validation objects, so their error fractions share a denominator. The winner is $k=3$; the test result $2/10$ describes that chosen variant on another sample. It need not equal the validation fraction because the samples differ. Choosing another $k$ after viewing it would make the test participate in selection and abandon its original role as the final assessment.
Source of the original notes: lecture notes.