How Nubank cut build times with MacStadium
Nubank is a rapidly growing FinTech company changing online banking for over 5 million customers in Latin America and around the world.
Headquartered in Brazil, Nubank is a $4 billion company, recently raising $180 million from Tencent. They use bleeding-edge technology, design, and data to fight complexity and empower people to take control of their finances. Nubank knows that a great iOS app is critical to serving its customers and has invested in building a solid mobile development team and supporting them with a team of dedicated DevOps engineers. Their iOS app is developed with robust continuous integration (CI) processes that generate multiple releases per month and supports over 1.5 million users (and has a great App Store review, too).
Nubank’s iOS continuous integration efforts weren’t always so smooth. Before starting to work with MacStadium in early 2017, Nubank was handling their iOS development in-house running Xcode server. As they grew, they realized they needed to transform the way they developed their iOS applications. CI outages and other issues impacted build schedules and caused delays for the entire team. Updating Xcode (and Ruby, fastlane, etc.) caused painfully long downtimes, and had to be repeated for each machine in the in-house solution. There were also issues with Xcode caching, which meant caches had to be manually cleared by the DevOps engineers every few days.
At least one day a month, if not more, had to be devoted to ‘fixing the iOS builds.’
As their CI needs grew, so did the time required to manage their environment. At least one day a month, if not more, had to be devoted to “fixing the iOS builds.” Manually managing their builds with no virtualization was not only time-consuming but also prone to breaking. With 13 developers working on application development, concurrency was an enormous issue. Far too often, too many pull requests were submitted at once, so builds started to queue. Developers were waiting for the build to compile and see the results of their tests. Scaling their development efforts on their DIY infrastructure was going to be very challenging.
If they were going to continue to grow their business, Nubank needed to change. The team started by reviewing several possible solutions including Mac colocation services and hosted CI services. However, colocation options were complex and expensive and hosted CI solutions didn’t offer the ability to apply customizations that their build process required. Ultimately, Nubank chose MacStadium’s private Mac cloud to enable their DevOps transformation because it balanced simplicity, flexibility, performance, and affordability.
MacStadium quickly helped Nubank meet their growing Mac infrastructure and iOS CI needs by moving them from their in-house Macs running Xcode server to a private cloud running on Mac Pros with all-flash SAN and virtualizing their build machines with VMware. Nubank’s DevOps team partnered with MacStadium engineers to determine the optimal size and number of build VMs for their app, team, and development workflow. They were quickly using Linked Clones to automate their infrastructure allowing them to use truly ephemeral, single-use build VMs. The Nubank team also created a set of scripts to automate access to their vCenter and maintained a file that defined every aspect of how their infrastructure should behave.
Impact on Nubank
These changes gave Nubank the power and flexibility to update its entire iOS development process. The most immediate impact came from having more dedicated build VMs, which in turn let them run more frequent simulations without negatively impacting performance. Nubank was able to transition from a monolithic codebase to a new scheme with more modular feature sets. Nubank could only make this change because of the concurrency MacStadium infrastructure provided. It enabled each developer to work more independently because as a group they could now coordinate changes, which in turn increased throughput and grew productivity. This also reduced build times significantly compared to what they were before.
Further, after moving their CI to MacStadium infrastructure, Nubank implemented Bluepill to run as many iOS tests in multiple simulators as possible in parallel. Test procedures that once took half an hour or more now took 10 minutes. Nubank was also able to implement pull request integrations, only merging code if it passed all tests. Their current development process now ensures that every developer also writes and runs tests. In turn, that helped maintain stability, which helped prove to developers that the solution worked. Overall, testing takes a fraction of the time it did before Nubank switched to MacStadium. Developers now get feedback from testing exponentially faster, usually in 5-10 minutes and developers know as soon as possible whether their changes worked or caused problems.
This reduced build times to one-third of what they were before.
Nubank has also seen quantifiable improvements in code quality. Being able to test with each code check-in, and just as importantly getting developer support, was a tremendous improvement on former processes. As Nubank iOS engineer Victor Maraccini said, “We’re building better code because we’re able to test it.” MacStadium enables the structure of the team to be what it is because of the computing power and virtualization support it offers behind the scenes. According to the Nubank team, developers aren’t even aware that MacStadium infrastructure supports their CI efforts – all they know is that “it just runs.”
Nubank learned valuable lessons along the way about the best way to optimize their CI practices for their MacStadium environments. Primarily, moving away from physical hardware and adopting a philosophy of “Write Once, Deploy Everywhere” opened a world of possibilities. Virtualization matters. Running VMs on top of Mac infrastructure let Nubank move to Linked Clones instead of having to manage each image manually via scripts that had to be run each time. And the benefits of having the same tooling that builds/runs the tests run the infrastructure can’t be understated. Put another way, being able to treat infrastructure as code creates more possibilities and better applications.
When deciding on how to upgrade their iOS continuous integration setup, Nubank evaluated several options. They looked at continuing in-house operations, but their need to scale made that difficult. They looked at other IaaS solutions, but all were lacking in some key factors ranging from security to concurrency. Ultimately, MacStadium offered benefits no other IaaS provider could match.
MacStadium has supported Nubank as they’ve matured their iOS CI endeavors. At first, Nubank only leveraged MacStadium IaaS for product releases, but not for final release to the App Store. Now, Nubank uses MacStadium from commit all the way to the App Store. With the support of MacStadium, Nubank feels confident that they are on the right development track and can scale to meet their future iOS CI needs. They know they’ve got a true partner willing to develop the best solutions for their unique concerns and environment.