Textbook
Labelled data are separated by purpose. The training sample fits model parameters. The validation sample compares fitted candidates and selects algorithm settings. The test sample evaluates the final selected candidate.
In this protocol, test answers influence neither fitting nor selection. If a test result repeatedly guides the next candidate, those data have become part of selection and no longer provide an independent final check.
A split must match future use: related observations do not become independent merely by entering separate files. For time-dependent data or repeated objects, the partition must account for those relationships. Having three files alone does not establish a sound evaluation.
The original lecture’s 80% training and 20% validation example is not a universal rule. In competitions, public scores may effectively guide selection while the private portion retains the final-evaluation role. An external evaluator may keep test labels hidden.
A split’s role depends on the whole decision process, including feature preparation and choosing settings from comparison results. In a held-out-data protocol, transformation parameters estimated from observations are fitted on training data and then applied elsewhere. Otherwise held-out observations have already influenced the constructed procedure. Before final testing, the complete process that turns an original object into a prediction is frozen, including the particular model.
Related concepts: Classification error.
In plain language
Exam preparation separates a practice book for learning, a mock exam for choosing an approach, and a final exam for evaluation. Final answers retain that role only while they remain outside another round of preparation. Adapting to those answers turns the exam into another mock, even if its file is still named “test.” The data are not physically damaged; the selected procedure has become dependent on their answers. Renaming the folder “secret” cannot make the file manager erase memory of those hints.
Example
Cards A–I represent nine independent hypothetical observations. The 4/2/3 split is chosen only for illustration. Each card receives one role.
Training, validation, and test splits
Candidate fitting
Only cards A–D are available for fitting parameters. The rest remain held out.
Step 1 of 3: Candidate fitting
Read all steps
Candidate fitting
Only cards A–D are available for fitting parameters. The rest remain held out.
- Training
- AcurrentBcurrentCcurrentDcurrent
- Held out
- EFGHI
Candidate selection
Cards E and F compare candidates. Test cards G–I remain unused.
- Training
- AincludedBincludedCincludedDincluded
- Validation
- EcurrentFcurrent
- Test
- GHI
Final evaluation
After the chosen candidate is fixed, test cards G–I become available. Their results do not guide another selection round.
- Selection fixed
- modelincluded
- Test
- GcurrentHcurrentIcurrent
Cards E and F guide selection, so their result no longer provides an independent final assessment of the selected candidate. G–I retain that role until the procedure is frozen. The last step evaluates that frozen procedure. If an error on G prompts a rule change and another check on the same cards, the new procedure has used the test. The toy split demonstrates separate roles, not recommended partition sizes for a real project.
Source of the original notes: lecture notes.