Testing a model, explained simply
After training, you need to know whether the model actually learned the pattern or just memorised its examples. The only fair way is to test it on examples it has never seen before.
So before training, data scientists set aside part of their data as a test set, also called held-out data. The model never trains on it. Afterwards they count how many test examples it gets right.
Testing with the same examples used for training is like marking a spelling test where the pupil saw the answers beforehand. The score looks great but tells you very little.
Example: A fair test
- You have 50 labelled photos of cats and dogs.
- Keep 10 aside and do not show them to the model.
- Train on the other 40.
- Test on the hidden 10: if it gets 9 right, that is a much more honest score than testing on the 40 it already saw.
Try this
- Teach a family member to recognise three kinds of bird from a few pictures. Test them with the same pictures, then with new pictures of the same birds. Which score is more honest?
- Split a pile of cards into "study" and "test" piles before playing a guessing game.
- Think of a test result that looked perfect but was not fair. What made it unfair?