Towards Cloud Build Systems with Dynamic Dependency Graphs
Project leader
Dates
September 2017 - August 2021
Sponsors
Royal Society Industry Fellowship
Description
A build system is a critical component of most software projects. It compiles the source code written in various programming languages. It then produces executable programs - te end software products. Build systems sound simple, but they are not. In large software projects, it grows from simple beginnings into a huge, complex engineering artefact. This is the case because:
- it evolves every day as software developers add new features
- it builds programs for a huge variety of target hardware configurations (from mobile to cloud)
- it operates under strict correctness and performance requirements
- it stands on the critical path between the development of a new feature and its deployment into production
Build systems can take up to 27% of software development effort. Thus, improvements to build systems have rapid pay-offs. Despite its importance, this subject is under-researched. Major companies, such as Microsoft, Facebook and Google, invest significant internal resources to make their own bespoke build system frameworks.
Two requirements are important for complex large-scale build systems. But no existing build system can support both.
Distributed cloud builds
A codebase of large software projects comprises millions of lines of code spread across thousands of files. Thousands of developers build each file independently on thousands of machines. A distributed cloud build system shares build products among developers. This dramatically speeds up builds (and saves energy).
Dynamic dependency graphs
Many existing build systems need to know all the dependencies between software components at the start of the build process. This makes it possible to analyse the graph ahead of time. But it is limited. Parts of the dependency graph can often be discovered only during the build process. In other words, the dependency graph is dynamic, not static.
Some build systems that support cloud builds, such as Microsoft’s new build system, Facebook’s Buck, and Google’s Bazel. Others support dynamic dependencies, including Shake and Jane Street’s Jenga. But, at present, no systems support both. During this Fellowship, we will develop the first cloud build system to support dynamic dependency graphs. We will work with Microsoft Research to evaluate opportunities for its integration into Microsoft’s build infrastructure.