His defenses have given up enough points to lose six games.
I think that is an issue. We have exceedingly high expectations for the defense and I imagine most people on this board would be upset if White leaves, even though he "has given up enough points to lose six games."
We are the #18 scoring defense in the country, and that takes into consideration all the times the defense was put into a bad position. That's a stark change from the defenses from the previous years and it's largely with the same players from those teams.
Contrast we are the #118 scoring offense. Averaging 18.9 points a game. I don't think it's unreasonable to ask for 21 points in regulation from a guy who gets paid ~$1.4M.