How to Connect Storage for Ephemeral Build Testing
Leveraging ephemeral builds as part of your CI pipeline greatly improves consistency, scalability, and ease of updating. In this post, learn how to connect your build server to shared storage.
Ephemeral builds – continuous integration (CI) builds that are executed on a brand-new, single-use VM image that was cloned from a master image – are all the rage in modern infrastructure for a number of good reasons:
- More consistent behavior (avoiding any subtle changes to base settings over time)
- Ease of updating (update one template versus every machine in your fleet)
- Scalability (scaling up is as simple as adding hardware)
Interestingly, although setup is the most labor-intensive part of running an ephemeral build -- like most complex workflows comprised of multiple platforms -- the real power of ephemeral VMs is harnessed through a process of fine adjustments that are meant to wring every ounce of performance out of the system as a whole.
One of the adjustments you can make to maximize the power of your ephemeral builds is through shared storage. This can be achieved in two ways: cloud storage or locally shared storage. While there are pros and cons to both options, the main trade-off is speed. As everyone knows, transmitting data via the internet takes time. Storing data locally to the same system where it was generated offers a significant opportunity for time savings. Many companies prefer to store artifacts locally, test locally, then make the choice to upload artifacts that pass to their repo of choice for distribution. This architecture is effective, but requires locally shared storage to the Mac-based build site.
Setting up Cloud Network Connections
First, connecting to MacStadium from other clouds is an easy process. In general, the process of accessing an external data source inside of your MacStadium private cloud works like this:
- Setup a VPN/VPC gateway with your cloud provider
- Configure your MacStadium firewall to accept traffic from that gateway
On the MacStadium documentation site, you can find easy-to-follow walkthroughs with screen shot examples:
Setting Up Local Storage Connections
MacStadium provides local dedicated storage arrays in a few forms, from USB drives to enterprise flash blades. Using that storage requires some configuration, but it isn’t too difficult. We’ll cover how to connect storage to the two cloud-based virtualization packages in use at MacStadium: ESXi (usually with vSphere by VMware), and Orka (Orchestration on Kubernetes with Apple).
MacStadium works closely with the VMware teams in the interest of providing the best possible VM on Mac experience. For example, a local data store is mandatory for VMware-based VM templates and clones to run, so it is set up for you as part of every vSphere deployment. This looks like:
Connecting to the storage of choice for local writes is fairly straightforward. Just have the usual /mnt/ path defined for any networked storage, and write as normal when you’re ready.
NFS storage in Orka is similar. Kubernetes has a persistent volume (PV) system defined, where a PV is made for space, then local machines have a claim (PVC) which associates the storage area to that VM at that time. Full tutorials of this are on the Kubernetes website and include references to storage areas like ElasticSearch, which replicates and shards data for use by multiple VMs.
Remember: one of the Kubernetes value-adds is the ability to use your preferred tool on a management platform optimized by Google, written in YAML. Always ask yourself, “How can I simplify my stack?” Then do a search for example Helm charts or *.yaml files to handle the deployment of services for you.