The Mac keyboard layout system is good; it's basically a tree of states, so it's both simple and capable. Ukelele is a decent layout editor; the only down side I recall is that it's not easy to create chains analogous to X11 Compose. Editing or generating keyboard layout XML directly is also possible.

Karabiner and hidutil are at a lower level; they modify the key codes that are the input to layout.