Science is difficult. Developing a strict testing system for a science experiment that is repeatable in other lab environments is even harder.
Science and Software
In a science lab, testing is done following the scientific method. This means setting up a scientific control. This type of experiment increases the reliably of a test by comparing a control scenario with a near duplicate scenario that has only had a single independent variable changed. Scientists must strive to define a test that is auditable and scalable while creating data that is repeatable and free from error.
In software development, testing means creating a simulation that matches production in every way possible to ensure reproducibility and judge the quality of software. With many tests, it’s important to set up a reproducible environment for regular, automated testing. In science this is referred to as a cleanroom.
What is a Cleanroom?
A cleanroom is used in lab environments for testing to prevent contamination. Contamination in a lab is the appearance of unwelcome materials or substances entering the environment. Contamination in software testing can be corrupted packages, non-standard network configurations, or a number of other issues not seen in production. Contamination is bad wherever it occurs in the development pipeline.
In a lab, cleanrooms must be built and maintained to precise measurements and requirements. That means high standards of control for preventing contamination from items as small as 0.5 microns (200 times smaller than the diameter of a human hair).
Testing in a clean environment free of contamination is just as important in software development. It allows for rapid iteration of tests in a fast-paced industry. Developers can create a simple cleanroom environment for software testing and re-use it often for consistent results.
Testing with Virtualization
One way software developers test in contamination-free environments is using virtualization. There are many different types of virtualization available, but one prominent way is using virtual machines within VMware. Testing in virtualized environments allows developers to emulate a specific hardware scenario like the production environment.
Combining a cleanroom environment with virtualization is the perfect mix of variable control and efficient testing. This is a cleanroom VM.
The Cleanroom VM, Templates, and Cloning
With a cleanroom VM, software developers can have a non-conflicted environment that is easily reproducible. Through VMware vSphere, developers can setup a master template virtual machine for deploying reusable testing environments.
The master template VM will be the base image that a cleanroom VM is recreated from. It’s imperative to account for every variable and factor when developing the master template VM. That means deciding on and accounting for all software installed as well as ensuring the proper versions are used.
Versioning is essential because software dependencies change all the time with new software and tests can be invalidated if a different version of software is used than expected for testing.
Once the master template VM is created, developers now have a base image for testing. VMware vSphere offers developers the ability to clone the master template and create fresh cleanroom VM’s quickly and easily.
Cloning is very handy in virtualization. It allows developers to decide on the exact pieces of a VM to copy and create a new VM from. In this example, the entire master template VM is cloned into a new cleanroom VM. We can make very minor adjustments to the VM for identification purposes like changing the VM hostname and network settings without affecting the cleanroom environment.
Cleanrooms with Dependencies
The first argument against a cleanroom VM template is losing dependencies. With a cleanroom VM, most developers believe that you have to start from scratch completely, even with constant variables not changing. That means extra time to reload static assets and wasted bandwidth in some cases.
Advancements in virtualization have negated this argument. It’s now possible to attach static assets and cache dependencies externally to lower VM boot times and reduce network traffic. Travis-CI, an open source, continuous integration build tool, uses all of these methods with great success on their OS X and iOS builds running at MacStadium.
“We have strong guarantees for consistency. We create a fresh clone of a base VM image for every single job,” stated Dan Buch, a builder at Travis CI. Travis CI can deploy VM’s via a VMware Linked Clone image in seconds, with an overall time to green for new build VM’s averaging close to 60 seconds.
Advanced features like dependency caching and arbitrary directory caching allow consistent testing without wasting resources. “We offer these primarily to our pro customers,” Buch added.
Travis-CI uses cleanroom VM’s with advanced caching of dependencies to dramatically decrease build times for iOS and OS X build jobs on Mac hardware. Since moving to MacStadium in the spring of 2015, they’ve gained full access to the VMware vSphere client on their private cloud environment.
This allows Travis-CI to manage VM templating through custom scripting and launch the bare essentials to create a cleanroom VM for iOS and OS X build jobs. They’re able to control high-level actions like dependency caching for multiple jobs that require persistence of assets across builds.
With EMC SSD-based VMAX SAN storage array hosting builds, Travis-CI efficiently boots up cleanroom VM’s in close to 60 seconds for iOS and OS X builds. As the service expands and more companies take advantage of Travis-CI’s hosted offering, the service will be able to dynamically scale and offer the same fast delivery of cleanroom VM’s under heavy use.
Science and Virtualization
Effective use of cleanrooms in a scientific setting is essential for proper testing. The technique translates well to virtualization. It allows for companies like Travis-CI to take advantage of efficient testing methods to rapidly deploy virtual machines on Apple hardware using VMware vSphere to meet customer demand.
One goal in the current software development ecosystem is a move to an automated workflow with agile development. Using VMware, developers looking to test software are able to set up virtual machine templates. These are useful for quickly cloning the template into a new virtual machine — a cleanroom VM. These virtual machines can be created and destroyed rapidly for an efficient workflow in tools with high demand for speed and consistent behavior.
Learn how Travis-CI implemented cleanroom VM’s in their continuous integration tool for iOS and OS X testing at MacStadium, follow @MacStadium on Twitter and subscribe to the MacStadium Blog RSS feed.