BUIP052: Set up dedicated Continuous Integration
Submitted on: 2017-04-15
Status: draft


## Summary

Get a paid-for dedicated Continuous Integration (CI) service to do daily
/ nightly builds which cover the full test suite.

This would be an addition to the existing free service provided by
Travis.

## Motivation

Right now, BU is running on free instance of Travis CI [1], which is a
great service for open source projects. However, this is limited in the
build minutes, which means that BU cannot run the full suite of software
tests (which can take up to a few hours to run through depending on
hardware).

Instead, it runs a limited subset of quicker tests (a kind of “smoke
testing”).

Practically, individual developers are limited in the build platforms
available to them, and do not always run the extended tests, which
results in a lack of awareness of whether changes have broken some parts
of the software on some supported platforms.

Indeed, it is difficult if not impossible to currently obtain an
accurate picture of the overall project test status (all applicable
tests on supported platforms). A dedicated CI service could run the full
test suite once a day (testing ‘dev’ and ‘release’ branch heads for
various build platforms).

## Benefits

Contributing developers need information on how reliably a test is
expected to perform - when you develop a feature and the test fails, you
want to know whether the test was occasionally known to fail before you
started, or whether it’s likely due to changes you’ve made, etc.

Acts as a ‘health check’ for users who want to run BU. They can look at
the test status to gauge whether they are satisfied with the software’s
health and prepared to take the risk of running it.

Helps the project obtain better quality by making sure they don’t break
the software accidentally (daily feedback on merged code ‘dev’ and
‘release’ branches).

## Implementation

This BUIP does not intend to prescribe a particular service or provider,
but strongly recommends going with an existing, well known service with
good reputation and public interface.

A ‘Startup’ Travis instance (described as ‘best for small teams’) costs
$129 / month, i.e. ~$1548 / year, and allows 2 concurrent jobs and
unlimited build minutes / repositories / collaborators.

A ‘Small Business’ Travis instance (described as ‘best for growing
teams’) costs $249 / month, i.e. ~$2988 / year, and allows 5 concurrent
jobs and unlimited build minutes / repositories / collaborators.

## Alternatives

Developers could set up dedicated test boxes at home / work and feed
back test results into a common Git repository.

This would lack the nice interface etc., and would be more costly
overall as people might have to acquire dedicated test hardware
(although some of this could be obtained through sponsorship or
donations).

One could make a separate BUIP for a project to aggregate this “ragtag”
test data into something that’s easily digestible for project members.
I’m assuming this would end up costing more and not performing as well
as using an experienced CI provider. The advantage is that it would be
more decentralized.