You can use any part of C++ without having to use the rest of it. For example, you don't have to write your own std::allocator to use std::vector, nor do you have to use exceptions.
You can use which bits of C++ that you want - that's its great strength, permitting low-level behaviour whilst supporting high level abstractions (and as simple or as complex as you like).
While technically you don't need exceptions and can disable it, the reality is the design of the STL is such that exceptions are required. It makes no affordance to signal errors in any other way. You can get farther if you decide OOM is a crash - but otherwise much of the STL is out of bounds.
I haven't seen it in practice, but OOM is conceivably recoverable in certain cases (e.g., exponential vector growth with large capacities fails on a 32 bit system, but switching to linear growth as a recovery strategy succeeds in allocating sufficient additional memory). But OOM errors aren't the only type of errors that you can get in ctors. Disabling exceptions makes all those errors either fatal or your class invariants become silently broken, which makes exceptions and RAII kind of a package deal in robust software.
The point was that STL is unusable without exception because of OOM. If you're not using exception anywhere in your codebase and you don't plan to recover from OOM, is this still an issue?
(I don't have much experience with exception handling in large codebase, I've mostly worked on codebase that disabled exceptions)
Realistically even our operating systems aren't OOM safe. Linux will happily over commit swap but kill you when you try to use it.
Windows actually provides a stronger gurantee here. But if any of your dependencies are not OOM safe then neither are you. And much of windows userland is not OOM safe.
You can use which bits of C++ that you want - that's its great strength, permitting low-level behaviour whilst supporting high level abstractions (and as simple or as complex as you like).