Not everyone wants the results of concentrating on a certain sport/job/hobby, even if they are naturally great at it. For example, many women (or men for that matter) might not want the physical result of being the world's best body builder, even if they are genetically predisposed for it. Being the world's best horse hockey might also not appeal to many, the prestige probably goes most too the horse and not the jockey. I'm sure there are other activities which are lacking in some aspect, which would make people who would excel at it turn away.

Also, as it takes some time to get past the amateur level and explore whether one actually had generic potential to be great at a sport, and that there are more sports than time, there's an element of chance whether someone hits upon the right sport.