BOMR takes a lot of time to look for library updates. Looking for a single version can take up to 17 seconds in my tests. This can be sped up by using multithreaded lookups.
While we're at it, we could improve another thing: right now, BOMR looks for update, and if it finds one, it waits for user input if this update should be applied before continuing looking. This can be improved by first collect all available updates, and then prompt the user. The first phase takes longer but the user can then do something else, and when the long phase is done, the user is asked for all the update choices without further waiting time in between the choices.
Comment From: mhalbritter
BOMR library update search now uses 8 threads by default. Running gradle with --info now shows which library search took how much time.
See the commit message of https://github.com/spring-projects/spring-boot/commit/e0b7720b9e218ea0e85e67a9f255f0694c40a0d3 for further details
Comment From: wilkinsona
Unfortunately, I think we should revert this. Using multiple threads to resolve the versions in the way that we do is pushing the limits a bit for what's acceptable client behavior. I learnt this the hard way in the early days of Bomr when I managed to get my IP address blocked due to making too many requests that looked like a scraping tool.
I do like the idea of speeding this up though. I wonder if we could use the Maven Central Search REST API instead. The search index lags Central itself a little so we may miss some very recent releases but the benefits may well outweigh that.
Comment From: mhalbritter
Oh noes :/ I agree that 8 threads may be too much, but what about 2 threads? This way if one thread blocks a long time, the other can still resolve.
I'll take a look at the Search REST API.
Comment From: mhalbritter
There's a call to the search API which returns all versions:
https://search.maven.org/solrsearch/select?q=g:com.google.inject+AND+a:guice&core=gav&rows=100&wt=json
But the search API is very unreliable, I got some 502 Bad gateways or requests taking very long. I don't think it's stable enough for us to use.
Comment From: mhalbritter
Ah, looks like I just hit the API in a service degregation:
So maybe it's usable after all :)
Comment From: mhalbritter
Another idea would be to cache the POM files locally with their Etag value. Subsequent requests can then send the If-None-Match header with the etag value and when the server responds with a 304 Not modified we used the cached version. That should save some bandwidth and maybe speed up the requests.
Comment From: mhalbritter
There's an update on search.maven.org: https://central.sonatype.org/faq/what-happened-to-search-maven-org/
Comment From: wilkinsona
This seems to be OK with 2 threads. We can revisit if it becomes a problem in the future.