> There’s a big downside though in that landing patches to Rust is serialized on running the test suite on every patch, and it takes a particularly long time to Run the Rust test suite in all the configurations we care about.
One solution is speculative batched testing. Kubernetes uses this to keep up with its very high rate of PRs (>40 merges/day) and ~1hr maximum test times.
Given a queue of patches A, B, C, D, start the normal testing against A, but also start testing the merge of A+B+C+D against the master. If the batch passes first, merge them all. If A is merged first, it's still "compatible" with A+B+C+D, so they can go ahead. Doing a batch in parallel with a single PR like this slightly increases testing load, but ideally doesn't cause any slowdowns versus the fully serialized case.
Doesn't this process break when, e.g., master + A passes tests, master + A + B fails, but master + A + B + C + D passes again (having been fixed by C + D accidentally)?
I mean, it's pretty unlikely to happen, but it's possible. Plus it doesn't matter if all you cared about was that the HEAD of master passes tests, but not the whole history.
However, the current approach, if rebasing is used to squash every merge request into a single commit on top of master, allows you to have a linear history of the master branch where every commit is known to pass the whole test suite that was in existence at that moment, which, I think, helps a lot when bisecting.
I'm not sure if Rust compresses merge requests into a single commit on top of master, though.
In Rust the "verified product" is the chain of bors merges - we even save build artifacts for each node in the chain for easy bisection of issues. A rollup creates a single bors merge for all the commits in them, so it's only 1 link in the chain.
Yes, it's theoretically possible. When it happens in Kubernetes, it's far more common that m+A+B failed because of a flaky test than because B was actually broken, and C+D fixed it.
What we often see is m+A fails because of a flake, and m+A+B+C+D+E passes and merges as a batch. We have a lot of flaky tests. :(
I'd at some point drawn up a proposal for this, back when Rust started having problems with this. At the time I think we didn't really have the budget for such testing either.
bors' rollup support only lets one mark things for rollup, which show up differently in the queue. It's effectively a priority=-1. This is for things which probably won't break the build, and contributors make a batch rollup every now and then (bors has a button for this). Generally when I do rollups I go through these and ensure they're green on travis.
I've considered giving homu (the engine behind bors) better support for automatically doing this (at least automatically showing travis statuses) but I don't have much time.
You already trigger Travis tests for PRs before bors does an auto run, right?
In that case, it wouldn't double expenses. Assuming batches are slightly effective, it should reduce expenses-- with >50% success rate with batches of >=3 PRs, you don't have to do individual testing for the later PRs in the batch.
The Travis smoke tests against PRs are only done in a single configuration, so are relatively cheap.
I think it would double expenses because our CI runs at capacity. To do parallel builds we would have to contract for double the compute resources. (With a different purchase structure for our CPU time you may be right about that, but not sure).
If cost is a factor then you could use batch testing as your default mode of operation and not bother testing individual commits at all unless a batch fails.
Using batch testing as a default would both reduce costs and increase merge speed, assuming that your commits usually pass testing.
The downside of using batching as a default is that it wouldn't test every commit in isolation. That means it wouldn't necessarily be safe to roll back to a particular commit if that commit was tested as part of a batch. E.g. if patches A, B and C were tested together, then it's not certain that patch A by itself would pass the tests.
If your test-suite pass rate is significantly greater than 70% (and statistically independent), you can gain throughput by grouping PRs into batches.
For example, if the pass rate is 80%, then by batching 3 PRs together there's a 51.2% chance to pass with all three, which would save 2 test runs. and a 48.8% to fail, which would cost 1 extra run. That's a pretty substantial increase in throughput.
Similar, but different, yeah: it gives you an easy button to make a new PR out of A + B + C + D together. We tend to batch up documentation changes this way.
One solution is speculative batched testing. Kubernetes uses this to keep up with its very high rate of PRs (>40 merges/day) and ~1hr maximum test times.
Given a queue of patches A, B, C, D, start the normal testing against A, but also start testing the merge of A+B+C+D against the master. If the batch passes first, merge them all. If A is merged first, it's still "compatible" with A+B+C+D, so they can go ahead. Doing a batch in parallel with a single PR like this slightly increases testing load, but ideally doesn't cause any slowdowns versus the fully serialized case.
It looks like bors already supports something like this with "rollups", where simple fixes can be marked for testing together: https://internals.rust-lang.org/t/batched-merge-rollup-featu...