I guess I don't come across this very often in the circles I frequent, though I do hear about it from time to time. I think it depends largely on what we're talking about. By and large, I don't care if someone wants to dress a certain way, have their hair a certain way or wear certain jewelry. If it's not harming me, I don't care. But, there are instances where cultural appropriation goes too far (the improper adoption of certain iconography is a really good example).
Damned if you do, damned if you don't. If you ignored their cultures they'd be offended by that too, and white people would be accused of culture killing. Personally, I love experiencing other cultures. We all should. There's so much to appreciate.