Awesome, for some reason I couldn't think of the workaround of committing to a temporary branch.
> Hour-long test suite runs are a problem in and of themselves.
I'm not sure I agree. Let's say I have a data integration system that has 70 modules. Each module have some integration tests and some baseline benchmarks.
At the time of development I'm only concerned with the 1 or 2 modules I change whose tests can finish in < 5m easily.
But when I push the change I want to be sure nothing else broke. Maybe because the change was in a base module used as a library by some other modules. Maybe I forgot to run tests on 2 of the dependent modules. So running the entire suite on merges (and optionally on PRs if required) is a necessity in my opinion.
Thanks for the link though - I'll take a look.
> You either need multiple levels of testing so you get some feedback quickly, or you need some sort
> of parallel build system that breaks the tests up so that the whole set completes in a reasonable
> amount of time. Without that, you're going to have people either sitting around waiting on test
> runners or skipping the tests entirely.
100% agreed. With most of these points you can now ignore my previous paragraph. By a 1 hour test suite I mean the wall-clock time of a single build-test cycle on CI runners. i.e. to run ~130 test suites in parallel against various containerised databases and some DBaaS (like maybe Redshift, BigQuery etc.) takes 1 hour where each individual module takes 5 to 10 mins. Some may take longer depending on the depth of things you are testing.
> Effort and resources spent on a massively parallel CD buildbot sounds like a good
> way to get things done to me. Computers are cheap; developer time is expensive.
100% agreed there again. In my opinion the organisations that treat CI improvements as technical debt and not as first class citizens of the software development flow are doomed to move slowly and have a hard time getting things done.
> Hour-long test suite runs are a problem in and of themselves.
I'm not sure I agree. Let's say I have a data integration system that has 70 modules. Each module have some integration tests and some baseline benchmarks.
At the time of development I'm only concerned with the 1 or 2 modules I change whose tests can finish in < 5m easily.
But when I push the change I want to be sure nothing else broke. Maybe because the change was in a base module used as a library by some other modules. Maybe I forgot to run tests on 2 of the dependent modules. So running the entire suite on merges (and optionally on PRs if required) is a necessity in my opinion.
Thanks for the link though - I'll take a look.
> You either need multiple levels of testing so you get some feedback quickly, or you need some sort
> of parallel build system that breaks the tests up so that the whole set completes in a reasonable
> amount of time. Without that, you're going to have people either sitting around waiting on test
> runners or skipping the tests entirely.
100% agreed. With most of these points you can now ignore my previous paragraph. By a 1 hour test suite I mean the wall-clock time of a single build-test cycle on CI runners. i.e. to run ~130 test suites in parallel against various containerised databases and some DBaaS (like maybe Redshift, BigQuery etc.) takes 1 hour where each individual module takes 5 to 10 mins. Some may take longer depending on the depth of things you are testing.
> Effort and resources spent on a massively parallel CD buildbot sounds like a good
> way to get things done to me. Computers are cheap; developer time is expensive.
100% agreed there again. In my opinion the organisations that treat CI improvements as technical debt and not as first class citizens of the software development flow are doomed to move slowly and have a hard time getting things done.