Benchmarking with Multiple vSphere Builds Using Jenkins

MacStadium customers use a variety of virtualization layers – a choice that is driven by a number of factors that you can read about here. One such virtualization provider is VMware's vSphere. Currently the most popular method of virtualizing at MacStadium, vSphere is a longstanding giant in the field that has had time to develop a rich ecosystem of compatible tech. Perhaps not surprisingly, Jenkins – also the most popular option in its respective field at MacStadium – is among that tech.

As such, we've set out to discover best practices as they pertain to this duo when paired with MacStadium hardware. We're not convinced that the best approaches for use in a Windows or Linux environment apply equally to macOS or iOS builds with Xcode. In fact, we often see customers set out with a given best practice for the Windows and Linux worlds, only to later change that policy for Apple-based builds.

To that end, MacStadium has started to conduct internal benchmarking of builds, and in the process are building out example code for what we believe to be the best way to run VMs with Jenkins. We believe that ephemeral VMs will run best and that utilizing the Instant Clone feature is the best way to produce ephemeral VMs on-demand with vSphere.

Instant Clones have already had a good history of success. VMware has published quite a few articles on the subject, and all MacStadium internal testing confirms Instant Clones instantiate much faster than the old linked clones, making the ephemeral concept viable.

This article from VirtuallyGhetto.com describes, fairly accurately, the scripts needed to launch with the VMware vSphere as the only interface. Moving towards Jenkins, a combination of extra scripts, set up, or packages will be required for feature parity. In future blogs and resource releases, this code will become available for the general community.

As for early results, multiple VMs seem to be the way to go. The first tests have produced the following results:

Running two VMs results in a slower overall real build time per image, but much higher productivity. In this case, a net savings of about 8 minutes per build was achieved.

This 20% extra time cost for running multiple builds has been confirmed by a MacStadium customer. In the customer's production environment, one VM build takes a very consistent 27 minutes. In the off-peak hours test, this customer ran three VMs on the same node. All three completed with a build time of 32 minutes, or 10.7 minutes per build.

More to come!