If my goal is to hire e.g "the best physicists" in the world then its probably going to be low on my list. Particularly in stem where disagreeableness is high in general.
Also if a University is optimizing for say "Most High Impact Research published" by a prospective hire, their skin color is going to be pretty irrelevant.
So assuming that's what the university is optimizing for, I can see your point.
As noted in peer comments in these threads though, universities aren't solely in the business of pure research. They optimize for a wide band of goals, including teaching, grant acquisition and use, and publication. Those are human interaction skill-spaces, not particle-behavior-comprehension skill spaces.
And even in the realm of pure research: of the top ten big physics and astronomy institutions, four are in Europe (across three countries), one in China, one in Japan, two in Russia, and two in the United States. The ability to work cross-culturally with diverse people is extremely important for physics collaboration in the 21st century.
Also if a University is optimizing for say "Most High Impact Research published" by a prospective hire, their skin color is going to be pretty irrelevant.