Which version of swebench is using by sb-cli?

[daa233]

I run SWE-Bench Multimodal both by OpenHands and sb-cli. However, I got diffrent results:

• By OpenHands eval_infer.py, the final result is 25 / 94. (26.60%)
  • using swebench = { git = "https://github.com/ryanhoangt/SWE-bench.git", rev = "fix-modal-patch-eval" }
  • or OpenHands-Versa, using swebench = "^3.0.8" link
• By sb-cli submit according to here, the final result is 14 / 94. (14.89%)

The differences are also described by this issue OpenHands/OpenHands#10452

Could you please tell the which version of swebench is using by sb-cli?

Thanks a lot!

[daa233]

@carlosejimenez Could you please give me some information? Thanks!

[daa233]

Could you please tell the which version of swebench is using by sb-cli?

@carlosejimenez @john-b-yang @klieret @ofirpress Any update plz? 😢