I wish I could uprate you with more than one vote.
Seriously, https performance sucks from start to finish. If the average user doesn't care about https, but does care about performance, who are they going to go with - you or your faster competitor?
Probably depends on if their account gets hijacked at your site or the site of your fastest competitor.
I agree that anonymous or non-logged-in activity has no need of HTTPS, but anything that's transferring cookies, passwords, or other important user or session information should be HTTPS.
Seriously, https performance sucks from start to finish. If the average user doesn't care about https, but does care about performance, who are they going to go with - you or your faster competitor?