Thanks for the comment! I will look into this since I certainly think the ideal scenario would be the core mermaid project supporting customizations in their engine. I imagine that before that I would have to refine the project further after which it would be interesting to see if they would be open to supporting it!
have you considered implementing your algorithm as a (better) auto-layout engine for mermaidjs ?