Comparing & Selecting Models
Two spam filters score 98% and 97%, so you ship the first, then flip one line, random_state=3, re-shuffle the same rows, and now the 97% model leads. Nothing about either model changed.
- ▸Freeze both models, re-shuffle only the split, and which of two close rivals leads can flip
- ▸Rotate the held-out slice so every row is tested once, average, and the flickering ranking holds still
- ▸Peek at the sealed test set to tune, and the score you'd publish drifts above the score you'd really get
Two spam filters, A and B, come back from testing at 98% and 97%. A wins, so you ship it. Then you re-shuffle which emails landed in the held-back pile, nothing about either filter changes, only the rows they are graded on, and B pulls ahead. The score you trusted was part skill and part luck: it rode on which emails happened to fall in the test slice. That wobble has a name, a lucky split, and it makes comparing two close models off one split close to a coin toss.
RE-ROLL THE SPLIT, WATCH THE WINNER FLIP