BUIP052: Set up dedicated Continuous Integration
Proposer: freetrader
Submitted on: 2017-04-15
Status: draft
Get a paid-for dedicated Continuous Integration (CI) service to do daily
/ nightly builds which cover the full test suite.
This would be an addition to the existing free service provided by
Travis.
Right now, BU is running on free instance of Travis CI [1], which is a
great service for open source projects. However, this is limited in the
build minutes, which means that BU cannot run the full suite of software
tests (which can take up to a few hours to run through depending on
hardware).
Instead, it runs a limited subset of quicker tests (a kind of “smoke
testing”).
Practically, individual developers are limited in the build platforms
available to them, and do not always run the extended tests, which
results in a lack of awareness of whether changes have broken some parts
of the software on some supported platforms.
Indeed, it is difficult if not impossible to currently obtain an
accurate picture of the overall project test status (all applicable
tests on supported platforms). A dedicated CI service could run the full
test suite once a day (testing ‘dev’ and ‘release’ branch heads for
various build platforms).
Contributing developers need information on how reliably a test is
expected to perform - when you develop a feature and the test fails, you
want to know whether the test was occasionally known to fail before you
started, or whether it’s likely due to changes you’ve made, etc.
Acts as a ‘health check’ for users who want to run BU. They can look at
the test status to gauge whether they are satisfied with the software’s
health and prepared to take the risk of running it.
Helps the project obtain better quality by making sure they don’t break
the software accidentally (daily feedback on merged code ‘dev’ and
‘release’ branches).
This BUIP does not intend to prescribe a particular service or provider,
but strongly recommends going with an existing, well known service with
good reputation and public interface.
A ‘Startup’ Travis instance (described as ‘best for small teams’) costs
$129 / month, i.e. ~$1548 / year, and allows 2 concurrent jobs and
unlimited build minutes / repositories / collaborators.
A ‘Small Business’ Travis instance (described as ‘best for growing
teams’) costs $249 / month, i.e. ~$2988 / year, and allows 5 concurrent
jobs and unlimited build minutes / repositories / collaborators.
$3K / year for reliable full-spectrum daily test status information
accessible to all project members would be a good investment in the
author’s view.
The existing free Travis service which we are using to test PRs and
merge commits should remain in operation unchanged by this BUIP (this is
a recommendation). The free service is useful as-is, a lot of effort has
been invested and there is no major benefit to changing it / moving
everything to another service.
The daily / nightly tests can also be run quite differently, requiring a
different CI configuration. For example, for tests that fail, one could
re-run them several times to gauge whether it’s a random failure and
what the probability of failure is. One could also set up repetitions of
passing tests with different execution orders, various random seeds etc.
There’s a lot that can be done to get more mileage out of the existing
tests.
The author is unaware of Bitcoin Core or other Satoshi-based clients
publishing full test suite run data to the public. This is certainly not
optimal, and a good chance for BU to improve upon current practice and
provide a good example.
As mentioned by @solex in the bitco.in forum thread [2], Travis has
made an offer of upgrading the standard CI service that BU is currently,
giving developers access to faster build times for the regular builds.
This is offered at $2750 p.a.
Developers could set up dedicated test boxes at home / work and feed
back test results into a common Git repository.
This would lack the nice interface etc., and would be more costly
overall as people might have to acquire dedicated test hardware
(although some of this could be obtained through sponsorship or
donations).
One could make a separate BUIP for a project to aggregate this “ragtag”
test data into something that’s easily digestible for project members.
I’m assuming this would end up costing more and not performing as well
as using an experienced CI provider. The advantage is that it would be
more decentralized.
[1] https://travis-ci.com/plans
[2]
https://bitco.in/forum/threads/buip052-set-up-dedicated-continuous-integration.2034/