You can whitelist from GMail while still greylisting from everyone else. Simply query DNS for their SPF records and use that for the whitelist, Google is good about keeping that up to date.

GMail sends from a farm, it's random where the exit ends up being.

I only have that problem from GMail, Hotmail and Yahoo don't seem to have this problem.