Dropbox: Virtualizing Mac Infrastructure at Scale

Growing Pains

While their CI pipeline worked very well, Dropbox started running into problems as they increasingly added more hosts to their cluster. “One of the issues we saw was a lot of ESXi crashes on the Macs. We didn’t know why, but it was causing us a lot of headaches. That made coordination between the teams a challenge,” said Ruan. The ESXi service crashing for an unknown reason didn’t inspire a lot of confidence in their staff developers. When a test failed, there was some lingering doubt as to whether it was the infrastructure or the code that was causing the failure. “It’s confusing when you can’t trust the underlying infrastructure,” said Ruan. “Because then you’re questioning, ‘is my test actually flaky or is it the infrastructure’? So it’s really important to have reliable infrastructure that you can trust.”‍



Dropbox didn’t have a tremendous amount of expertise in-house for running VMware on Mac computers, and there were multiple teams involved in the process which added even more complexity. “If you think about the entire system, it’s not actually one team that’s managing everything,” said Ruan. “We have a team working on CI, there’s a team managing vCenter, and then there’s a team working on Mac hardware in the data center. As we scaled up, we found that we were stepping on each other’s toes.”‍



The struggles associated with growing an in-house build farm, and the expertise that it requires, led Dropbox to MacStadium. “It was a mix of having to deal with all these issues while constantly hiring engineers because we needed to add capacity quickly,” said Ruan. Experimenting with different storage, networking, and configuration options, Dropbox aligned on an improved version of their original in-house Mac build cluster based on VMware, but in a more scalable cloud environment at MacStadium.



With MacStadium, we have more expertise available to us for Macs.

Impact on Dropbox

With a MacStadium cloud based on VMware, the Dropbox team is able to add Macs to their CI clusters with a simple request to MacStadium’s engineers. “If we were to try to add racks in our old data center, we would have needed to plan for the space much earlier and we would have had to figure out how to buy more Macs from Apple. Whereas with MacStadium we talk to our finance department, we get it approved, we come to MacStadium, and we get Macs in a few weeks,” said Ruan. “We have so much more flexible scaling now.”‍



That translates to increased flexibility elsewhere in the pipeline as well, and the ESXi crash headaches are gone. They effectively cut the workload in half for the team’s in-house CI pipeline, with half of the workload being funneled into MacStadium. Dropbox engineers built a small API wrapper around VMware that could load balance between the two environments. “We’re reducing load on our internal vCenter by spreading the load across two vCenters now,” said Ruan. “We already had this extra layer on top of vCenter, so we just had to change the code a little so that we can allocate VMs to either our internal vCenter or our MacStadium one.”

Conclusion