It's HOW the team loses, and how things are trending that concerns me the most. If "new guy" comes in, and has losing records his first two years, but they are competitive and passionate in every loss, and little improvements seen each week/year, I'm willing to give "that product" more time to get pieces in place to turn the competitive losses into eventual wins.
When you lose to NIU and are underdogs to Turdue in your 3rd year, you gots to go.
Edit: Adding the fact that they aren't favored in any remaining game against mediocre teams... gots to go.