A Bazel Persistent Worker for Rust
Bazel persistent workers are a cool feature that allow Bazel to start up “compiler” instances that can accept multiple build requests. This brings benefits like saving startup time, saving the time to parse a standard library or share some cache across compiler invocations. This allows slight speedups in rebuilds, which can be valuable in speeding up the developer iteration cycle.
This is best exemplified in the existing persistent workers:
- The Java and Scala rules benefit from paying the cost of process startup only once (warming up the JVM and so on.)
- The TypeScript compiler benefits from parsing all the JS standard library type definitions only once instead of on each re-compile.
I have just released a similar
compiler wrapper for Rust. The unimaginatively named
rustc-worker does not get any of
the above benefits, since
rustc does not have a “service”
Instead it brings the speed up due to incremental compilation that is already
the default in Cargo builds to Bazel.
Since Rust 1.24, rustc
has a notion of incremental
incremental=/a/directory is passed to it (as Cargo does1), intermediate state
is saved in that directory. This allows it to rebuild the crate faster.
By default Bazel uses sandboxing to guarantee more hermetic builds. This means
rustc invoked does not have access to data written by prior
invocations of itself, so it cannot take advantage of incremental compilation.
This means Bazel rebuilds of Rust code are slower than their Cargo equivalents.
Introducing a persistent worker allows enabling incremental mode because the worker process can introduce a cache that is shared across builds. The worker is responsible for making sure this cache does not violate hermeticity completely. Of course, we rely on rustc’s notion of incrementality being sound.
Rebuilds of ninjars are roughly 2x faster with workers. This is not a benchmark by any means, but clearly there is an improvement.
cargo build (incremental by default) 1.65s bazel build (without worker) 2.47s bazel build (with worker) 1.2s
How do I try this out?
Give it a shot and file issues if something goes wrong.
Writing a persistent worker is fairly easy. One has to watch for the special
--persistent_worker and then read protocol buffers on stdin. Using
Protocol Buffers in Rust is
really easy using Prost. The only
annoying part is reading length-delimited messages from stdin. Since Prost
reads from a byte buffer, and not a stream, we need to do some careful
There are a few more things I need to fix, like using the path to rustc and the Bazel compilation mode in the cache path. I’m really hoping this will integrated into rules_rust so everyone benefits.
- See the directory called
target/debug/incrementalin any Cargo built Rust project. [return]