It shouldn't ruin his career. But how he responds is important. I think the appropriate response for someone in that position today is to legitimately recognize that he did something stupid yesterday, in a time where a lot of people didn't understand how stupid it was. Just because it may have been more socially acceptable then doesn't make it okay, but society has evolved. We are more enlightened now and need to do better about recognizing our blind spots and confronting bias. ...something like that.
The privilege that they are talking about is the ignorance of why blackface and those kinds of caricatures can be offensive, because perhaps we have never been forced to face those issues. Ignorance and privilege often go hand in hand. When I was in my early 20s, as a white middle class male, I'm sure I did and said all sorts of insensitive and offensive things, because of my own ego-centrism, because I didn't really understand the world outside of my own upbringing, etc. I bet most of us can say the same thing.