Exactly. If someone goes after you personally, they'd need several of your password (at least three or four) if you have a decent algorithm. Then they'd have to find that pattern.

Most password leverage comes from breaches and people running larger scale operations for scamming and spamming.