Overfitting, explained simply
Overfitting happens when a model fits its training examples too closely. It learns the real pattern and also every accident and mislabelled example, called noise.
An overfitted model scores brilliantly on the examples it trained on and poorly on new ones. That is why testing on held-out data matters so much.
Ways to reduce overfitting include using more examples, fixing wrong labels, and choosing a simpler model. In kNN, listening to more neighbours (a larger k) often helps, because one noisy neighbour no longer decides the answer alone.
Example: Learning a route to school
- Good learning: "turn left at the park, right at the library".
- Overfitting: "turn left where the red car was parked on Tuesday".
- The second rule matched Tuesday perfectly and fails on every other day.
Try this
- Memorise the answers to ten maths questions, then try ten new ones. How is that like overfitting?
- Draw scattered points and compare a wiggly line through every point with a smooth line through the middle.
- Find one wrong label in a set of examples and think about how it might confuse a model.