Textbook
A fixed classifier $h$ is evaluated on $n>0$ labelled observations $(x_i,y_i)$. The indicator $\mathbf1\{h(x_i)\ne y_i\}$ is 1 for an incorrect answer and 0 for a correct one. The classification error on sample $S$ is the fraction of incorrect answers:
$$\operatorname{err}_S(h)=\frac1n\sum_{i=1}^{n}\mathbf1\{h(x_i)\ne y_i\}.$$
The sum counts mistakes; division by $n$ converts the count to a fraction between 0 and 1. Every observation has equal weight. For $n=0$, the formula is undefined: performing no checks does not establish zero error.
The same formula applies to training, validation, and test samples, but their roles change its interpretation. A zero value on data used to fit the model does not establish performance on new objects.
If new pairs $(X,Y)$ come from distribution $P$, error on that distribution is the expected indicator: $R_P(h)=\mathbb E_{(X,Y)\sim P}[\mathbf1\{h(X)\ne Y\}]$. This is the probability of a wrong label, not an automatically known number. Error on an independent representative sample estimates it and can differ because a finite sample has a particular composition.
Related concepts: Supervised classification, Training, validation, and test splits.
If exactly $k$ observations are misclassified, empirical error is $k/n$, while the fraction correct is $1-k/n$. This rule uses only label agreement: prediction confidence and the numerical distance between class codes do not enter it. Renaming classes consistently in both true and predicted labels leaves the result unchanged. The pairing of each prediction with the correct observation must still be preserved throughout the calculation.
In plain language
Classification error resembles marking a dictation: every wrong answer receives one red mark, then the mark count is divided by the answer count. Two mistakes out of four differ from two out of a hundred despite identical demands on the red pen. The evaluated set matters; a perfectly memorized sheet does not characterize the next dictation. Each mistake has equal weight here, regardless of confidence or consequences. The marks count misses honestly without deciding their individual costs or the conditions of a future exam.
Example
This toy sample has four objects. True labels are 0, 1, 1, 0; predictions are 0, 0, 1, 1. The comparison pairs labels belonging to the same object.
Classification error
Answer comparison
Predictions differ from the true labels on the second and fourth objects.
Step 1 of 3: Answer comparison
Read all steps
Answer comparison
Predictions differ from the true labels on the second and fourth objects.
- Truth
- 0110
- Prediction
- 0 ✓included0 ✗excluded1 ✓included1 ✗excluded
Error indicators
Correct answers contribute 0; incorrect answers contribute 1. There are two ones.
- Indicators
- 01current01current
- Mistakes
- 2current
Fraction of errors
Two out of four predictions are wrong. The error is 0.5, or 50%.
- Mistakes
- 2excluded
- Total
- 4
- Rate
- 0.5current
Positions 2 and 4 are wrong. Subtracting the sum of true labels from the sum of predictions would fail: both sums equal 2, producing a false zero as opposite errors cancel. The indicator first converts each mismatch into its own one, and only then are those ones added. The resulting 50% describes these four observations rather than guaranteeing the same fraction on the next sample.
Source of the original notes: lecture notes.