Automated Builds - A Simple Step to Avoid a Time Bomb

2017-11-06

Your CI build has a hidden flaw ####

After you get your new continuous integration build initially configured you get to sit back and enjoy the fruits of your build labor. You check in your code and watch your build pipeline magically spring into action. As you progress you might add a branch spec or two and now you are down the road to thinking about an automated release pipeline as well. During the life cycle of your project your team is committing many times daily on topic branches, pull-requests and your watching release branches land in production. Possibly you are even deploying your infrastructure from your pipeline as well. For this back to basics post we are just talking about the continuous integration portion of the process.

As your project moves forward you hit your launch date, your customers are thrilled with the service you are providing and the business is thrilled with the value you have provided. Your project goes into a get a little bit of a stability and you're on another project or at least a little more time passes between feature releases. This is a very common scenario for companies with just a few developers who are tasked with writing customer software, supporting out-of-the-box products as well as helping help desk with ticket in various capacities. What ever the reason a bit of a drought appears in your commit graph.

You either will get lucky and you will not notice anything. You start committing again and don't notice anything. As all luck has a shelf life yours will run out and you will return to your little project with broken build in your hands and you're not sure why it had mysteriously broke.

So what happened?

During that period of no commits your operations folks patched your build agent, or some of your version unspecific project dependencies were updated, or your hosted build agent on your CICD pipeline service updated a a SDK for which you were only calling out for a I-do-not-care default.

There are a number reasons that can can cause this but since you were not building anymore you didn't get notification of any build failures to warn you about the changed situation until you were airdropped back into the project for a quick fix. Instead of cutting a hotfix branch to to get fast tracked into production the night before Black Friday you are now you're looking for the razor to clean up that certain bovine.

Yak Shaving is something that's tangental to what you should be doing, and not necessary.

The Simple Fix

Add a scheduled trigger to your builds so that at a very minimum your master branch builds on a nightly basis. If you are using a branching strategy where you having some other unmerged branches you might consider adding scheduled builds for those as well. Some projects I will have builds scheduled for develop, feature/* and hotfix/* but those types of deciesions are a case-by-case basis.

Repeat after me

I will put my builds on a scheduled trigger along with my commit

Side Note I just ran into a CI build that has not been run in 130 days and I'm scared to run it.