Build Machine Virtualization

So hopfully by now you've heard the term 'virtualization'.  There is a definate trend toward running production servers at virtual machines to fully utilize the capabilities of new servers, to help achieve 0 downtime, to reduce the amount of hardware needed, etc.  There's also a trend towards quality testing being performed more and more on virtual machines rather than physical machines where possible to reduce the number of physical test machines required which thus reduces price, to more easily facilitate reproducing problems, saving snapshot of problem environments, to help facilitate automated testing, etc.

So why not go virtual with build environments?

Where I work they have the concept of freezing build boxes with every release.  So at the end of a release, that build box is frozen and a new box is purchased for the next version.  This was a new concept to me.  However due to some events that unfolded when we did not freeze a build box and decided to build that version's two service packs on the same box.  Bad things unfolded.  Suffice to say, I now am all for the freezing of build boxes.  The problem is that process gets expensive and time consuming (setting up and maintaining those boxes and providing disaster recovery for all of them).

So when we got close to the end of our last version, I proposed to our project manager that we go virtual with our build environment.  I recommended buying a beefy server capable of storing multple environments and running a number of them too.  So we got a dual Xeon, 4 GB Dell server with a good amount of storage and a license of VMware GSX Server (only to find out shortly after purchasing the license that VMware is offering its Server software free in Q2 '06 - oh well, we still need a support contract).  Here are the benefits that running a build machine in a virtual environment:

• Multiple machines utilizing the same hardware - the hardware for a good VMware server may be expensive but if you freeze build boxes with each version or have multiple products, the cost can easily be made up.
• The build environment can be run on any VMware Server or Workstation (of compatible version).  This means that you don't need high-end hardware to run a virtual build environment.  Even a simple workstation computer would do.
  
• The build environment can easily be copied - for testing reasons, build machine redundancy, etc.  Obviously you'll need to worry about software licensing on the build machine if you copy it.
• Since the build machine can be run on any VMware Server or Workstation, if the physical machine dies, you can easily transfer it to another machine and be back up and running more quickely.
• Since you can just copy build machines, you can easily set up base build machine images or copy the build machine as a starting point for the next version.
• Here's my favorite: Easily archive the build machine at the end of a version.  Simply burn the image to DVDs (it is highly recommended that you have the virtualization software to span the hard disk files at 2GB - VMware definately has this functionality).  Then if you have a disaster were you lose the build machine somehow, just retrieve the DVDs, find a VMware Server or Workstation (or download VM Player) and you'll be back up in a short time.  No need to wrry about compaitble hardward.  The problem with images (such as Ghost images) is that you have to install on the same hardware or you're going to have major issues.

I highly recommend running a virtual build environment if you can.

Tags: VMware, virtualization

Build Reproducibility

When designing a build process, you should design a way that the build process can be re-run for a particular build.  The easiest way to do this is to divide each step into one of three categories:

1. Master
2. Build
3. Test

Build category

Any step that is required to build the software, documentation and installers are those that fall under this category.  These steps are heart of the build process.  The build steps should be unaware of source control, versioning, build output distribution, etc and should assume that everything it needs is on the box it is running on.  These build steps should ideally also contain only relative paths making them location independent.

If you had all of the build steps contained within its own build file or able to be executed by themselves, then someone who had all the build software installed on their computer should be able to fetch the source code himself, execute the build and have the same output that he would if he ran the same thing on the build computer.  Also, in the event of a catastrophe where you lose a build, you simple restore the source code for that particular build number and run this file again, then when it is done running, you should have the same output as you did when that build was originally run.
 

Test category

Any step that tests the build belongs in this category.  These steps should, like the those steps in the build category, be unaware of source control, versioning, build output distribution, etc and should assume that everything it needs is on the box it is running on.  These build steps should also ideally contain only relative paths making them location independent.

Master category

The steps contain in this category are those that fill in the gaps between the build and test category.  These steps include maintaining and incrementing the version number, pre-build modification of the source code (for example updating resource files with the most recent translations), tagging or labeling the source code, fetching the source code (preferably from the tag or label), executing the build and test steps and distributing the output of the build (such as making the installers publicly available or at least available for QA).  The master build file should also be the one who actually updates any build status application such as a build status web page.  It is often beneficial to have notices on the individual steps of the building of the applications, so you may have to develope your build category steps to take in a flag as to whether it should output its status to an external source but should default to not update that external source.  This way, only when the Master build steps execute the Build steps will they actually notify others of its status.

Example:
Here's a quick example of how this works.  Say I have a .NET project that I build 1 solution, generate 1 help file and make 1 installer.  This project has a database that the build process automatically generates the install scripts for.  This application also works for multiple languages so I have a database in which I keep all the translations for my software which the build process reads and updates all of the resource files in the project.  Below is how I would define the build process.

Master Build File

• Increment the build number
• Read the latest translation data from the database and update the resource files.
• Commit changes to build number and resource files to the database.  Note: at this point, since I have truely consumed another build number, I can consider the build to actually have started.
• Tag/label the source code with the new build number.
• Fetch the source code from the tag/label.
• Execute the Build file
  
• Save the installers and binaries from the build to a public share for others to pick up
• Execute the Tests file
• Distribute the results of the tests.

Build File

• Generate Database install scripts
  
• If executed by the Master build file, update status
• Build the .NET solution
• If executed by the Master build file, update status
• Build documentation
• If executed by the Master build file, update status
• Build installer
• If executed by the Master build file, update status

Tests File

• Test Database install scripts
  
• If executed by the Master build file, update status
• Build and run unit tests on .NET solution
• If executed by the Master build file, update status

Reproducibility
Now imagine that catastrophe strikes and our build machine dies.  Luckily we have instructions for setting up a new build environment to be the same as the old one.  The instructions contain all of the software and their versions along with the keys needed for them.  So within a day we have a new build environment up and running.  Now I need to recreate version 1.0 of our software so I create a revision for it.  I fetch the source from the tag that we made for v1.0.  If I've defined the build process properly, I can simply run the Build File and have output that is nearly the same exact output that we had for v1.0.  Hopefully the only difference will be dates of files.