Improving compilation speed by order of magnitude
All of our improvements allowed us to decrease build times of freshly checked out workspace – order of magnitude less.
We are working with one of the biggest and most innovative Tier-1 financial institutions and the project consists of one of the largest Scala codebases in the world.
In such a big and complicated project compilation times used to be significantly over one hour. We were working on many fronts to radically bring it down.
We changed our build system and IDE to use the same incremental compiler (Zinc). That allowed us to generate compile caches during CI builds, and use them on-demand later. Thanks to it, average build time went down from 1h to 25 min, and to just a few minutes with good cache hits.
That also significantly speeded up setting up new workspaces or catching up with master – no need for expensive recompilation saves us up to 1h every time.
We introduced user-specified, limited working sets that would reduce mental and compile overhead. Developers can now focus only on what is important to them.
We designed and developed several improvements for the Scala compiler. One of the examples could be flat classpath which in our environment decreased memory footprint by order of magnitude. It was contributed back and it’s now part of the mainline scalac.
We helped develop a custom build system that would ensure efficiency consistency between CI and IDE builds. Furthermore it gave us a chance to introduce several performance improvements such as no-op compilation faster by order of magnitude, much more effective cache shared between machines, and significantly quicker compilation.
All in all, all those improvements allowed us to decrease build times of freshly checked out workspace from over one hour to just a few minutes – order of magnitude less.
That not only hugely increased developers’ productivity but also significantly reduced the costs of running our build farm – we are saving hundreds of hours per day.
