Friday, January 2, 2015

Software Builds at EA: The 5000' View

A couple of days ago this tweet by John Carmack popped up. In case the link ever goes away, he says, "Dealing with all the implicit state in the filesys and environment is probably going to be key to eventually making build systems not suck."

At EA we have spent a lot of time developing build systems that don't suck. They're not perfect but as we develop applications and games targeting platforms from Android to Xbox, plus everything in between, they work incredibly well for us.


The cornerstone of our build infrastructure is a version of NAnt that was forked eons ago and has undergone essentially a complete rewrite. Although the base functionality is still there, and still used, a lot of effort has been put into establishing it as a base for the higher level components.

NAnt is just one piece of a system called Framework. In addition to NAnt, Framework includes features for managing build configurations, SDKs, and software distribution. There are a few core concepts in Framework that I think have made it much easier to manage the build morass: modules, packages, dependencies, masterconfigs, build configuration management, SDK packages, and the package server.


A module is Framework's representation of an artifact producing build process. At a minimum a module will have a type (static/dynamic library, executable, C# assembly, etc.), a name, and a set of input files. Modules have a number of optional components such as custom compile options, custom preprocessor definitions, custom warning suppression, sets of resource files/include files/object files/libraries, and dependencies.


Packages are a collection of modules. Every package in the Framework world has a version, even if that version is "unversioned". A package contains a description of how to build its modules, its public exports (such as public include directories, build artifacts, etc.), as well as a manifest which allows the package author to include structured metadata that can be inspected by the package server, continuous build/test systems, etc.


Dependency handling is one of Framework's killer features. Dependencies can be made onto other modules in the same package, onto other packages, or onto other modules in other packages. For example I can specify that my module has a dependency on EASTL and the build system will ensure that I am using the correct version, build it if it is not already built, and set up the necessary include and library paths.


Versioned packages lose their utility if there is no way to enforce the versions being used. The masterconfig file is the bill of materials for the application and specifies the version of each dependency, each SDK, and each configuration component.

One of NAnt's script concepts is the property. Essentially a global variable, properties are used for tweaking the state of the build system. Some are set in build files as crude variables, others like the particular configuration to build are set on the command line. A third way to set properties is to encode them in the masterconfig. This ensures that the whole development team is using the same Android SDK level, for example, as the SDK package may provide several.

Most build system state can be controlled in the masterconfig file(*), and checking the masterconfig file into source control can be a easy and effective way of ensuring the team is building with consistent settings.

(*) Provided that the build script writers do not start using environment variables on the side :-)

Build Configuration

Framework also provides global build configurations that describe how to perform a build. These build configurations take high level build goals (ie: optimized 32-bit Windows build) and use it to determine the required compiler invocation command lines.

These build configurations are used as the base for building a project and all of its dependencies. Although a particular dependency may override its build settings for warning suppression, custom optimization settings, etc., for the most part these are slight modifications made to the global configuration.

SDK Packages

SDK packages are not handled any differently by Framework than other packages, but Framework's versioning forces consistent use of SDK versions by all developers. Many SDKs are packaged in ready-to-go forms that the build system will download automatically and use to build, skipping the hassle of having to find the right SDK package and installing it. Or installing it and its numerous components.

Package Server

Where versioned packages come into their own is as a basis for sharing technology between development groups and teams. To make this sharing easier we run a server for people to post their packages on, not entirely unlike Freecode/Freshmeat. If Framework cannot find locally a package listed in the masterconfig it will try to fetch it from the package server.

Less Make, More CMake

Owing to its NAnt heritage Framework has the ability to perform builds itself but with Visual Studio support demanded by most developers it is being used more and more as just a front end for generating project files for Visual Studio or Xcode. 

No comments:

Post a Comment