For clarity, you allow the open source client app (which you can audit and compile yourself) to securely share your contact list with a secure enclave (which is open source so you can audit it as well).
The client-side contact list is one of two lists: (a) device’s contact list if Signal is given access — or (b) list of numbers that have manually been add to the App and Signal is not given access to device’s contact list because user declined providing it access.
Also, might be wrong, but appears Signal’s secure enclave, which is based on Intel’s SGX technology, is known to be vulnerable to side-channel attacks and Intel’s SGX code & hardware is not open source:
No, you have to trust anyone with access to the server; secure enclave provides no security given it appears it is vulnerable to side channel attacks; see comment you just replied to for related link.
Basically all it does is given Signal plausible deniability for public search warrants:
It would do nothing to stop national security letters. Basically, you should assume the functionality provided by the secure enclave is only in name; given the NSA ATT national security letters enabled both physical access and system modifications, see no reason to believe others would not be required to do so as well:
Again, average person does not have a threat model that makes this relevant, but to say Signal does not have access to the SGX data is purely based on trust, not mathematical proofs.
At that point, your phone is vulnerable. As is your computer, and your connected fridge.
If your threat model does not allow it, use manual e2ee with pigeons. Otherwise, well secure enclave will get better with time, which will make Signal better too.
Signal should make threats clear to users, just as they should have notified all users that multiple 3rd parties downloaded all the phone numbers in Signal. Offering federated servers would also allow users to secure their own servers.