One could argue that it affects your Google organic search traffic. The better your HTML, and lighter your Javascript, the more likely the mighty Google will send people to you.
For whatever it's worth: Patrick's original strategy for ranking (and perhaps still the strategy that accounts for most of his revenue) isn't the Google search [bingo cards]. It's 1000 closely related but technologically distinct searches.
I heard that alot, but haven't actually seen that proven yet.
It's a very good excuse to force people on improving their HTML and website speed, but I still would like to see the impact of the Google organic results after only speed improvement and bad HTML fixes.