You can have a financial manager control your accounts for you and just keep a small checking account, (plus they'll help you grow your balances) but they're not free. Well, they're not free if you want them to be unbiased. Given, what's going to keep them from getting scammed? Maybe what you're looking for is several safe deposit boxes.
I still want my money invested into the economy. I just want Chase/Fidelity/etc to have an understanding that I am never going to withdraw money from these accounts without planning for it. So, “I” should never be authorized to drain the account at a moments notice without extensive approval. Anything to cause friction for would be scammers and only once-a-year (?) pain from me to triply confirm the money can move.
I don't have direct access to my long-term savings and retirement accounts— I have to go through my financial manager who'll works in a small, local firm, and so would anyone trying to impersonate me. He would probably recognize my voice, knows where I live and what's going on in my life, to whom I'm married, etc. because we have bi-annual check in meetings. He'd definitely contact me through his existing contact info if there was anything weird going on with one of my requests, especially if it involved a different address or account than he's used to dealing with. As anyone in that compliance-and-accuracy-focused line of work should be, he's very intent on making sure all of the Ts are crossed and Is are dotted. He charges a flat percentage of my modest retirement savings annually (I'm far behind most white collar workers my age, coming from a working class early adulthood) so he has a financial interest in my investments, and does a really solid job managing them. The accounts are in a large investment-focused bank which I believe only he can access. I think it's about as safe as you could get while still keeping your money active in the economy and not having a rich person's resources.