(this will become part of my upcoming booklet on beautiful builds)
One of the things that kept frustrating me when I was working on various types of build configurations in our CI server, was that each build in a rolling wave would take longer than the one before it.
The CI build was the fastest, the nightly build was slower, because it did all the work of CI (compile, run tests) plus all the other work a nightly needed (run slow tests, create installer etc..). The Deployment to test build would maybe just do the work of the CI and then deploy, or, in other projects, would do the work of nightly, then deploy, because wouldn’t you want to deploy something only after it has been tested?
And lastly, production deploy builds took the longest, because they had to be “just perfect”. so they ran everything before doing anything.
That sucked, because at the time, I did not discover one of the most important features that my current CI tool had : the ability to share artifacts between builds. Artifacts are the output of the build actions. these could be binary compiled files, or they could be configuration files, or maybe just a log file, or just the source files that were used in the build , from source control.
Realization
Once I realized that build configurations can “publish” artifacts when they finish successfully, and that other build configurations can then “use: those artifacts in their own work, things started falling into place.
I no longer needed to re-do all the things in all the builds. Instead, I can use the DRY (Don’t repeat yourself) principle in my build scripts (remember that build scripts are kept in source control, and are simply executed by a CI build configuration that provides them with context, such as environment parameters, or the artifacts from a previous build).
Instead, I can make each rolling wave build, only do one small thing (single responsibility principle), that gets added on top of the artifacts shared by the build before it.
For example:
- The CI Build Only gets the latest version from source control, compiles in debug mode, and runs the fastest tests. Then it published the source and compiled binaries for other builds to use later on. Takes the same amount of time as the previously mentioned CI build.
- The Nightly build gets the artifacts from the latest successful CI Build, and only: compiles in RELEASE mode, runs the slow tests, creates installers, and publishes the installer, the source, and the binaries for later builds. Notice how it does not even need to get anything from source control, since those files are already part of the artifacts (depending on the amount of source this might not be a good idea to publish source artifacts due to slowness, but that depends on the size of the project). takes half the time of the previously mentioned nightly build.
- The Deploy (to test) build, gets the installer from the nightly build that last passed successfully, and deploys it to a machine somewhere. It does not compile, or run any tests. It publishes the Installer it used. takes 30 seconds.
- The Deploy (to Staging) will just get the latest successful installer build artifacts from (deploy to test) builds, and deploy them to staging, and also publish the installer it used. Takes 30 seconds.
- The Deploy (to Production) will just get the latest successful installer build artifacts from (deploy to staging) builds, and deploy them to production, and also publish the installer it used. Takes 30 seconds.
Notice how with artifact reuse, I am able to reverse the time trend with builds. The more “advanced” a build is along the deployment pipeline, the faster it can become.
And Because each build in the deploy pipeline is only getting artifacts of successful builds, we can be sure that if we got all the way to this stage, then all needed steps have been taken to be able to arrive at this situation (we ran all the tests, compiled all the source…)