I was thinking about that a while ago and came to the conclusion that it's likely massively helped out by the narrow search space. They aren't trying to match between every single person, just the ones in your photo library which is an extremely small group compared to what most facial recognition is doing.