Boosting Python with Rust: the case of Mercurial (Raphaël Gomès) [FOSDEM 2020]

link slides

Mercurial is written in Python, but they have started integrating some Rust source code to improve performance.

Mercurial is a VCS that was started at around the same time as git. It has a very powerful plugin system. It is written mostly in Python, with some C modules to improve performance.

Rust is a low-level language with a very powerful type system and compile-time memory safety. Maintainability is rust is much better than in C. Better signal-to-noise ratio, less boilerplate code needed. Standardized tooling: cargo for packaging and dependencies, test suite. It is “safe” by default. Performance is comparable to C for sequential code, probably better for parallel code.

Valentin Gatien-Baron did a small experiment to compare performance of Rust with Python. He re-implemented a very small subset of mercurial in rust. Performance on a large repository improves by two-three orders of magnitude.

rust-cpython is a pair of rust crates to make python and rust interact. A low level crate for the basic interaction. A high-level crate for really integrating.

However, when moving parts of the Python code to rust, is doesn’t get faster. For example, after moving a single function to Rust, it gets twice as slows. That’s because the data structures have to be translated from one language to another. Also there is a lot of complex interface code. For example, stat’ing 100K files from rust takes 30ms, but passing the results to Python takes 300ms.

One possible solution is not to go through the Python layer to go to the existing C layer. There is already the concept of a PyCapsule to wrap a C module. Let’s reuse this for rust code as well. However, there are a bunch of missin features. For example, inheriting from a class written in rust is not possible. Also instance attributes are not possible, only class attributes.

(Skipped a bunch of details that were hard to follow.)

With the current state, some C modules ported to rust, there is some improvement, but not dramatic: about a factor of two. To improve further, a number of things are considered:

  • Do more in parallel.
  • Improve conditional execution, i.e. focus on hot paths to optimise.
  • Make use of the rust type system to rethink the order of execution. This may expose more parallelism.
  • Reduce the exchanges between Python and rust.
  • Fewer allocations, etc.
  • Get rid of Python entirely? In particular, don’t start python if it won’t be used, because python has a large startup time.

Working so much with Rust has given Raphaël a new appreciation for python. In python you get working code very quickly. It allows experimentation. It is easy to break things by changing something somewhere in a large project, but on the other hand you can often get away with it without changing a lot.

Wouldn’t it be better to better to avoid FFI entirely? E.g. always use either rust or python, not call one from the other: this would be difficult because it would break e.g. plugins or would need a lot of rewrite. E.g. use rust code as a server and communicate with a socket: this probably wouldn’t help because we’re shaving off such small amounts of performance that the socket overhead would kill it.