SWE-Bench is also not stellar. "It's important to remember" that:

- they are only evals

- this is mostly positioned as a general consumer product, they might have better stuff for us nerds in hand.