I believe this is possible ATM.
Comment From: jorisvandenbossche
I think we should also consider in looking into asv (airspeed velocity), if it would be possible to replace our current vbench suite to that (issue for that is: #8361).
Airspeed velocity is actively developed and maintained, other projects like scipy did also start using it, and it has some nice extra features (eg memory benchmarking, interactive web frontend). So maybe it is more worth the effort looking into that, than trying to improve vbench.
Apparently @qwhelan did already some work on this: https://github.com/spacetelescope/asv/issues/194 @qwhelan, maybe you can share some thoughts about this?
Comment From: rockg
I agree. I figured I would see how much work it would be to get vbench working in Python3 and then go from there. It was a few simple changes on the vbench side and definitely the harder part is going to be getting the benchmarks compatible. However, this would probably need to be done anyways for asv so I figured it was worth proceeding with vbench. The suite can be run in Python3 now with my vbench changes (note that I didn't do a full test of plotting, but nothing there seemed to stand out as an issue).
Comment From: qwhelan
@jorisvandenbossche Definitely have some comments but I'll have to get back to you guys this weekend. They've added a lot of support for vbench features I thought were significant but I haven't had a chance in the last few weeks to see what's fully/partially/not at all supported.
Comment From: qwhelan
@jorisvandenbossche I have a short script that converts vbench Benchmarks into asv ones with a little bit of ast use. There's a bit of information being lost (at least comments and start dates) but I don't see any issues that would prevent us from having things in parallel for a bit. I'm still working through some compilation issues that I'm fairly sure are OSX-specific, but I do have a subset of our vbench cases running under asv via my translation script. If I don't run into any issues, I'll upload a cleaned-up version tomorrow night.
Comment From: jorisvandenbossche
@qwhelan And what about some of the other things you noted: is it possible to keep the benchmarks in the pandas repo or does it have to be a separate repo? And are the similar abilities to our test_perf.sh script?
Comment From: jorisvandenbossche
And BTW, thanks a lot for working on this!
Comment From: qwhelan
@jorisvandenbossche They've changed it so everything can be stored in the same repo. Also, the asv run and asv compare commands should be able to mostly replace test_perf.py, though there may be some specific commands missing. Notably, it doesn't have the daily sampling vbench does - it will just uniformly slice the commit range based on the number of commits.
Comment From: jorisvandenbossche
good to hear! Just open a PR with what you have, that is the easiest to try out/give feedback
Comment From: jorisvandenbossche
Closing as it is probably not worth to put more work in vbench if we are going to use asv (see #9715)