Textbook
Classification assigns each object a label from a finite set of classes $Y$. Features represent the object as $x\in X$, where $X$ is the space of valid descriptions. A labelled sample contains pairs $(x_i,y_i)$: features and an observed correct label.
Supervised learning uses those pairs to construct a classifier $h:X\to Y$. For a new description $x$, the model produces the prediction $\hat y=h(x)$. The hat distinguishes a prediction from the true label, which may still be unknown when the prediction is made.
Features must have the same meaning during learning and prediction. A customer might be described by income and card balance. A $4\times4$ image can become a vector of 16 pixel intensities, with the depicted digit as its label. These are different representations of the same label-assignment task.
Observed answers do not provide a lookup table for every future object. A classifier must carry a learned pattern to new descriptions; data volume alone does not remove errors in features or labels.
The input description and correct label have different roles even when both are numerical. Label one may name a category rather than a quantity to approximate by numerical distance. Binary classification has two classes; the chosen category names do not alter the task. Applying an existing $h$ produces a prediction for each admissible input, but its correctness requires comparison with the label or another independent checking procedure.
In plain language
Sorting messages into “ordinary” and “spam” starts with examples whose correct labels are already known. Features describe information available before the answer, such as link counts, while the label specifies the required folder. For a new message, the model must apply its learned rule without the correct label already supplied; otherwise the task merely reads an answer printed on the envelope. Remembering familiar addresses is insufficient for recognizing new cases. The analogy assumes a consistent meaning of the labels; a confident classifier alone cannot settle annotators’ disagreement over “spam.”
Example
This toy message model uses two features: link count and word count. Prediction follows an illustrative rule: at least three links means “spam,” otherwise “ordinary.” This applies a given $h$; it does not claim the rule was learned or is suitable for a real filter.
Supervised classification
Features of a new message
The true label is hidden. Four links and twenty words are observable.
Step 1 of 3: Features of a new message
Read all steps
Features of a new message
The true label is hidden. Four links and twenty words are observable.
- Links
- 4current
- Words
- 20current
- True label
- ?
The input description
The two features form the input. Coordinate order is fixed: links, then words.
- x
- 4 linksincluded20 wordsincluded
The predicted class
The given rule predicts spam. This remains a prediction until checked against the true label.
- Prediction
- spamcurrent
- True label
- ?
In $x=(4,20)$, the first coordinate counts links. Swapping coordinates without changing the model’s convention would incorrectly treat word count as link count. The final “spam” prediction follows from the illustrative threshold, while the true label remains a question mark. A confident prediction does not remove that question mark: the demonstration computes an answer without establishing that it matches the correct category.
Source of the original notes: lecture notes.