It's not so much the skin color, it's the culture of their ethnicity. Or perhaps it's a localized, colloquial thing.
I'm Asian, and I also don't gravitate toward Asian women, though most of my female friends are beautiful Asian women. I think I'm immune to them from so much exposure.
Maybe...