I love football - it's the only sport I will watch a game even if I have no rooting interest. However, even I draw the line at some of these bowl games. I want meaningful matchups that create national attention, and many of the bowl games don't because the teams are uninspiring.
While I agree there may not be a quantifiable downside in some cases, there's certainly a qualitative one. But, bowl game ratings for the first week or two were down compared to the same time last year by a fair margin. Nationally, fewer people just didn't give a damn this year and I think a lot of this is because the quality of some of the games stunk. Were they worse than last years? In some ways it matters, in others it doesn't.
The more average and 5-7 teams you allow to compete, the more the quality of the opponents drop as does the national interest, imo.