You change the compiler to emit different things like a reverse stack or whatever your protection model is. Far as implementation, they describe it in p5 (PDF p7) of paper above (not MULTICS paper). It's actually brilliant now that I read it as the naive thing they avoid is, IIRC, what the other academics did on Linux/GCC. The performance overhead hit 10% easily due to x86's stack approach. I think worst-case was even higher. This team effectively tricks the CPU with simple instructions (eg addition/subtraction) without invoking memory to get it to worst-case of 2%. Clever.
Note: I'm not saying this is sufficient to stop stack smashing. Just that reverse stacks are a better idea than the ludicrous concept of making unknown amount and quality of data flow toward the stack pointer. Definitely reduced risk a bit but how much takes more assessment.
Note: I'm not saying this is sufficient to stop stack smashing. Just that reverse stacks are a better idea than the ludicrous concept of making unknown amount and quality of data flow toward the stack pointer. Definitely reduced risk a bit but how much takes more assessment.