If there were no users, presumably there would be no network (at least if "users" include people who are in it for the financial speculation), so it is meaningful to attribute energy usage to users. But how to quantify that? One reasonable approach that shows up a lot is to say that usage is making transactions, and just consider the energy usage of all transactions to be that of the entire network (since mining exists to add credibility to transactions). Then estimates will say that on average, any transaction uses more than 1 MWh of electricity.
The network uses the same amount of energy whether a block has transactions or not. Surely you see how that would actually help your “I hate energy use” argument if you actually made that one. Your argument is a common argument though, its incompatible with the more accurate argument.
Secondly, trading for speculation occurs offchain, meaning most speculators are not transacting on the network.
