Typically you have your own IP pbx such as an asterisk system which lives in one place on static v4/v6 IPs, it connects to your upstream sip trunk. Your own clients such as zoiper on Android connect to that.
I thought about using asterisk or something but does that noticeable latency when connecting remotely? Right now I connect directly to my sip provider on the go via an android sip client whereas running my own PBX would add another hop and was concerned how much this impacts latency.
I'll open myself up for correction on the matter, but it's been my experience that these days Asterisk is more useful as and often more commonly deployed as a feature server (hunt-groups, ring-groups, call trees etc), at least for a broad majority of use cases where SIP is even a part of the conversation. Personally I'd not recommend one try rolling their own Ast based phone system for production/enterprise unless you just really want to and have literally no other projects to deliver back to the org.
Otherwise, what you're doing is perfectly fine and well enough: connect to your SIP trunk via credentials using a local client and you've more or less got a working, accessible phone number. Unless you truly have a need for Asterisk features, don't bother.
I happen to know first hand of a "voice company" that has a pretty sizeable footprint in the travel/hospitality industry making several million dollars a year and one of their products amounts to nothing more than configuring IVRs and charging through the nose to host them using what is (in my opinion alone) a thoroughly overly-complicated Asterisk infrastructure and similarly overly-complicated dial plans.
