In the kinds of applications that require a systems language you need to know the object layouts, it isn’t avoidable. Algorithm selection over those objects is dependent on the physical object layout and hardware architecture based on the use case. The compiler doesn’t do any of this and largely can’t because it doesn’t understand what you are trying to do. It has nothing to do with “hand-optimized assembly code”.

You are making a classic “sufficiently smart compiler” argument. These types of problems can’t be automagically solved without strong general AI inside the compiler. See also: SIMD, auto-parallelization, etc. We don’t have strong general AI, never mind inside the compiler.

Until we have such a compiler, you will be dependent on people caring a lot about physical data layout to make your software scalable and efficient.