On Bazel and Open Source (2015)

This is a rare post because I don’t usually talk about Google stuff here, and this post is about Bazel: a tool recently published by Google. Why? Because I love its internal counterpart, Blaze, and believe that Bazel has the potential to be one of the best build tools if it is not already.

However, Bazel currently has some shortcomings to cater to a certain kind of important projects in the open source ecosystem: the projects that form the foundation of open source operating systems. This post is, exclusively, about this kind of project.

For this essay more than ever: the opinions in this post are purely my own and I have no affiliation to the Blaze team. But yes, I have used Blaze for years.

And for those that don’t know me, why am I writing this? Because, first and foremost, I am a “build system junkie” and thus I have general interest in this topic. And second, because I have written various open source software components and packaged countless projects for various operating systems, including NetBSD, FreeBSD, and Fedora; all this for longer than I’ve been at Google. In fact, I was NetBSD’s sole Gnome 2.x maintainer for about 3 years—yeah, call me masochist. These activities led me to learn a lot about: build systems; the way a great bunch of upstream maintainers think and behave; and a ton on how to write portable software that can be built and installed with minimum fuss. I’m far from an expert on the topic though.

About three weeks ago, Google released Bazel: the open source variant of Google’s internal build system known as Blaze. During the six years I have been at Google, I have heard various individuals wishing for an open source version of Blaze and, finally, it has happened! This is a big milestone and, all things considered, a great contribution to the open source community. Kudos to the team that pulled this off.

What I would like to do with this post is, for the most part, guide you through how a sector of the open source world currently builds software and, to a lesser extent, present why Bazel is not yet a suitable build system for this specific use case. By “open source world” I am specifically referring to the ecosystem of low-level applications that form a Unix-like operating system these days, the majority of which are written in C, C++, and interpreted languages such as Python. There certainly are plenty of other use cases for which Bazel makes a lot of sense (think higher-level apps, Android, etc.), but I won’t be talking about these here because I do not know their needs.

Bazel, just as Blaze, is an exemplary build system. As its tagline {Fast, Correct} - Choose two claims, Bazel is a fast build system and a correct build system. Correct in this context means that Bazel accurately tracks dependencies across targets, and that it triggers rebuilds whenever the slightest changes. Fast in this context refers to the fact that Bazel is massively parallel and that, thanks to accurate dependency tracking, Bazel only rebuilds the targets that really need to be rebuilt.

But the above two qualities are just a byproduct of something more fundamental, which in my opinion is the killer feature in Bazel.

Bazel build rules are defined in BUILD files, and the build rules are specified at a very high semantical level. Compared to make(1), where you specify dependencies among files or phony targets, Bazel tracks dependencies across “concepts”. You define libraries; you define binaries; you define scripts; you define data sets. Whatever it is that you define, the target has a special meaning to Bazel, which in turn allows Bazel to perform more interesting analyses on the targets. Also, thanks to this high level of abstraction, it is very hard to write incorrect build rules (thus helping enforce the correctness property mentioned above).

cc_binary( name = "my_program", srcs = ["main.cpp"], deps = [":my_program_lib"], )

cc_library( name = "my_program_lib", srcs = [ "module1.cpp", "module2.cpp", ], )

cc_test( name = "module1_test", srcs = "module1_test.cpp", deps = [ ":my_program_lib", "//something/unclear/googletest", ], )

This simple BUILD file should be readable to anyone. There is a definition of a binary program, its backing library, and a test program. All the targets have an explicit “type” and the properties they accept are type-specific. Bazel can later use this information to decide how to best build and link each target agains the others (thus, for example, hiding all the logic required to build static or shared libraries in a variety of host systems).

Yes. It’s that simple. Don’t let its simplicity eclipse the power underneath.

The open source world is a mess of build tools, none of which is praised by the majority; this is in contrast to Blaze, about which I have not heard any Googler complain—and some of us are true nitpickers. There are generic build systems like the autotools, cmake, premake, SCons, and Boost.Build; and there are language-specific build systems like PIP for Python, PPM for Perl, and Cabal for Haskell. (As an interesting side note, Boost.Build is probably the system that resembles Bazel the most conceptually… but Boost.Build is actively disliked by anyone who has ever tried to package Boost and/or fix any of its build rules.)

Of all these systems, the one that eclipses the others for historical reasons (at least for the use case we are considering) is the first one: the autotools, which is the common term used to refer to the Automake, Autoconf, Libtool, and pkg-config combo. This system is ugly because of its arcane syntax—m4, anyone?—and, especially, because it does a very poor job at providing a highly semantical build system: the details of the underlying operating system leak through the autotools’ abstractions constantly. This means that few people understand how the autotools work and end up copy/pasting snippets from anywhere around the web, the majority of which are just wrong.

However, despite the autotools’ downsides, the workflow they provide—configure, build, test, and install for everyone, plus an optional dist step for the software publisher—is extremely well-known. What’s more important is that any binary packaging system out there—say RPM, debhelper, or pkgsrc—can cope with autotools-based software packages with zero effort. In fact, anything that does not adhere to the autotools workflow is a nightmare to repackage.

The autotools have years of mileage via thousands of open source projects and are truly mature. If used properly—which in itself is tricky, although possible thanks to their excellent documentation—the results are software packages that are trivial to build and that integrate well with almost any system.

What I want to say with all this is that the autotools are the definition—for better or worse—of how build systems need to behave in the open source world. So, when a new exciting build tool appears, it must be analyzed through the “autotools distortion lenses”. Which is what I’m doing here for Bazel.

Blaze was designed to work for Google’s unified codebase and Bazel is no different. The implication of a unified source tree is that all dependencies for a given software component exist within the tree. This is just not true in the open source world where the vast majority of software packages have dependencies on other libraries or tools, which is a good thing. But I don’t see how Bazel copes with this yet.

Actually, the problem is not only about specifying dependencies and checking for their existence: it is about being able to programmatically know how to use such dependencies. Say your software package needs libfoo to be present: that’s easy enough to check for, but it is not so easy to know that you need to pass -I/my/magic/place/libfoo-1.0 to the compiler and -pthread -L/some/other/place/ -Wl,-R/yet/more/stuff -lfoo to the linker to make use of the library. The necessary flags vary from installation to installation if only because the Linux ecosystem is a mess on its own.

The standard practice in the open source world is to use pkg-config for build-time dependency discovery and compiler configuration. Each software package is expected to install a .pc file that, in the usual case, records the compiler and linker flags required to use the corresponding library. At build time, the depending package searches for the needed library through the installed .pc files, extract the flags, and uses them. This has its own problems but works well enough in practice.

I am sure it is possible to shell out to pkg-config in Bazel to integrate with other projects. After all, the genrule feature provides raw access to Python to define custom build rules. But, by doing that, it is very easy to lose Bazel’s promises of correct builds because writing such low-level build rules in a bulletproof manner is difficult.

Ergo, to recap this section: the first shortcoming is that Bazel does not provide a way to discover external dependencies in the installed system and to use them in the correct manner. Providing an “official” and well-tested set of build rules for pkg-config files could be a possible solution to this problem.

Another very common need of open source projects is to support various operating systems and/or architectures. Strictly speaking, this is not a “need” but a “sure, why not” scenario. Let me elaborate on that a bit more.

Nowadays, the vast majority of open source developers target Linux as their primary platform and they do so on an x86-64 machine. However, that does not mean that those developers intentionally want to ban support for other systems; in fact, these developers will happily accept portability fixes to make their software run on whatever their users decide to port the software to. You could argue that this is a moot point because the open source world is mostly Linux on Intel… but no so fast. The portability problems that arise between different operating systems also arise between different Linux distributions. Such is the “nice” (not) world of Linux.

The naïve solution to this problem is to use preprocessor conditionals to detect the operating system or hardware platform in use and then decide what to do in each case. This is harmful because the code quickly becomes unreadable and because this approach is not “future-proof”. (I wrote a couple of articles years ago, Making Packager-Friendly Software: part 1, part 2, on this topic.) It seems to me that, today, this might be the only possible solution for projects using Bazel… and this solution is not a good one.

The open source world deals with system differences via run-time configuration scripts, or simply “configure scripts”. configure scripts are run before the build and they check the characteristics of the underlying system to adjust the source code to the system in use—e.g. does your getcwd system call accept NULL as an argument for dynamic memory allocation? configure-based checks can be much more robust than preprocessor checks (if written properly).

I suspect that one could use a traditional “configure” script with Bazel. After all, the main goal of configure is to create a config.h file with the settings of the underlying system and this can be done regardless of the build system in use. Unfortunately, this is a very simplistic view of the whole picture. Integrating autoconf in a project is much more convoluted and requires tight integration with the build system to get a software package that behaves correctly (e.g. a package that auto-generates the configure script when its inputs are modified). Attempting to hand-tune rules to plug configure into Bazel will surely result in non-reproducible builds (thought that’d be the user’