Bias & the Limits of a Model
Feed a hiring model ten years of past resumes with model.fit(past_hires) and let it decide who advances. It copies the pattern in that pile faithfully, including the one where nearly everyone hired before was a man.
- ▸Feed the model a lopsided hiring history and its decisions lean the same way, with no rule to blame
- ▸A 97% next to a rejection is the model's mood, not a measured hit rate
- ▸A model is often most sure about the very applicants it has never seen anything like
An automated resume screener learns by example. Show it the resumes the company advanced over the last ten years and it hunts for the pattern that separates the people who got hired from the people who did not. No one writes a rule like if gender == "F": reject. The screener just copies whatever pattern is already sitting in that pile of past decisions.
That is the whole problem. If the last ten years of hires were almost all from one group, the strongest pattern in the pile is membership in that group, and the screener learns it like any other signal. Its advance-or-reject line quietly tilts toward that group, not because anyone told it to prefer them, but because the examples did. This tilt, inherited straight from skewed data, is what bias means. The arithmetic has no opinion. It faithfully reproduces the slant baked into the history it was trained on.