This is the fourth post in the series - Modeling Deployment Pipelines. In our previous post, we saw the advantages of reusing pipelines (using Templates). In this post, we’ll dive into ways to move your build through the deployment pipeline. Typically, once a build has gone through the various tests, it is deployed onto a set of environments such as UAT, pre-production and production (for example). This concept is called “Build Propagation”.
Fan-in / Fan-out
To implement the idea of build propagation in your pipeline, we can use a concept called fan-in / fan-out. In the image below, there is a “fan-out” from the end of the “Build” pipeline. It fans out into three functional test pipelines. The opposite of a fan-out is “fan-in”, where multiple (upstream) pipelines lead into a single (downstream) pipeline.
Here’s how that looks:
Click image to zoom in
In this scenario, the “UAT” pipeline will wait for all three of its upstream functional test pipelines to finish successfully before it starts. This doesn’t happen just because there are three pipelines fanning into the “UAT” pipeline, but because those three pipelines trace back to the same code repository. This is important to understand. If all three pipelines traced back to different code repositories, for instance, then every successful pipeline run would lead to the “UAT” pipeline starting. Fan-in ensures consistency by making sure that all upstream pipelines which trace back to the same repository have all finished successfully for the same changeset.
In practice, this means that if one or more of the functional test pipelines fails, then the UAT pipeline doesn’t get triggered. This makes sense since that changeset is not “good enough” to go to UAT.
That’s the real beauty of a CD pipeline - it prevents bad changes from reaching important environments (such as production).
The green arrow in the image represents “Build Propagation” in the pipeline. The installer artifact from the “Build installer” stage is fetched and used. This is the same installer artifact used by the functional test pipelines as well. So, the same artifact, built once, is being pushed further along in the pipeline.
Highlighting Dependencies
What we don’t see in the image above, are the dependencies between
- the functional test pipelines and the code repository
- and between the UAT pipeline and the code repository.
For the sake of completeness, we added these dependencies in and this is what the pipeline would look like.
Click image to zoom in
As you can see, it complicates the image quite a bit and so, has been omitted from other images. Owing to GoCD’s fan-in ability, adding that dependency doesn’t change how and when the pipelines are triggered. If a commit is made to the code repository, it does not mean that the functional test pipelines are triggered. Since each functional test pipeline has two dependencies now, both of them have to be compatible (i.e. be the same version) before it triggers.
For example, if a commit was made in the repository, the “Build” pipeline would build first with that commit. If the "Build" pipeline is successful, then both the dependencies (often called materials) of the functional test pipelines are compatible and the functional test pipelines will run.
Unlike the example in the image above, maybe your repositories are structured so that functional tests and deployment scripts are in their own repositories. If so, they can be modeled as shown below:
Click image to zoom in
The advantage of this approach is that a commit to the functional test repository does not trigger the “Build” pipeline (which makes sense). The latest successful instance of the “Build” pipeline will be verified using the new functional test code. In this case, fan-in does not apply, since the two dependencies of the functional test pipelines do not trace back to the same repository or repositories. Similarly, a change to the deployment scripts will not trigger the build and functional tests.
Depending on your business case, you might find it necessary to have a manual approval process, especially to deploy into sensitive environments such as production, and often pre-production (mostly because it is being used by several people and needs co-ordination). We’ll dive into how to create a manual approval process to propagate your build in our next post. Stay tuned.