What's even scarier is when your browser "clicks" on the link by itself [1] and you don't even realize that you've just sent a request to some 'random' server and that server now has the log of your IP. If you browse 'questionable' sites, definition of which is growing by the day, you should be careful in which log your IP might end up in.
I now disable page prefetch on every browser I use. Some browsers don't even use it (which is a sensible thing in 2020 given the risk vs rewards of having it turned on).
It took me a while to find the setting -- search Chrome preferences found nothing for "prefetch," and a how-to article I found claimed the setting was called "Use a prediction service to load pages more quickly," which also returned nothing, but I finally found it under Advanced -> Privacy -> Preload pages.
I was quite pleased, however, to find that this had already been turned off for me, and enforced, by uBlock Origin: https://imgur.com/a/9seIFEU
Moral of the story is actually to use a VPN regardless of what you do, so you can't be held accountable for someone else's actions or lack of security of IT department of some company.
I now disable page prefetch on every browser I use. Some browsers don't even use it (which is a sensible thing in 2020 given the risk vs rewards of having it turned on).
[1] https://en.wikipedia.org/wiki/Link_prefetching