I do believe there is a Darwinian basis for it.
Natural selection works better the more variations there are to be tested by nature. Natural selection would work very slowly if everyone is the same as there would be few new characteristics giving an advantage. If there is a lot of interaction among diverse groups, there would be more differing characteristics that could lead to a feature that is favored by the environment.
It makes sense that exotic looking people would be sexually attractive.