Universal rubygem precompilation

2025-09-10

Context

Precompiling Nokogiri and SQLite was an unmitigated success for users in terms of installation speed and reliability, but the intensive effort required to set up precompilation and the testing matrix makes it inaccessible for many maintainers.

Tools like uv have demonstrated how amazing it is to be able to install dependencies quickly, but without a broad set of precompiled ruby gems, the benefits of a similar rv project would be limited.

This brief attempts to answer:

Current state

The current tooling, maintained in part by your humble author, relies primarily on cross-compiling on Linux for (mostly) historical reasons. Maintaining this Linux toolchain (cross-compilers and libraries to support various Windows and macOS (Darwin) versions, a hacked Ruby build system, all wrapped up in OCI images) is complex and brittle compared to the more simple process of compiling on the target platform.

The current gemspec has a few drawbacks which limit the platforms that can be targeted, and require that binary gems must package multiple ruby versions together ("fat gems").

Discussions have been ongoing, primarily between myself, Ruby Core member Soutou Kouhei (@kou), and Rubygems contributor Samuel Giddins (@segiddins), on evolving the gem specification and the tooling to better support precompiled libraries.

Evolving the specification

Some recent discussions have been driving decisions about evolving both the gem specification and the tooling for precompiling gems.

These discussions have been ongoing for a few years, but Sam's paid position at Ruby Central allowed him to focus on this long enough to generate a reasonable RFC inspired by Python's Wheels specification and informed by the "Wheel Next" project.

Participants in the discussions, in the above links and elsewhere, include:

Highlights from the recent RFC:

Open question: Rubygems and Bundler support

There's obviously quite a bit of work that will need to be done in the rubygems and bundler projects to support the new gem specification. It's unclear, given Sam's departure from his salaried position at Ruby Central, whether anyone will have the resources to execute on this in the near future.

Evolving the tooling

In the 2025-04 Rubygems discussion linked above, kou and myself also discussed how to evolve the tooling.

I think we achieved consensus around a few points:

Inspiration from the Python community is the cibuildwheel project which provides CI-focused support for a large matrix of Python versions and platforms.

Open question: Improving the tooling

The execution path here is pretty clear, but will require focused effort to get great, well-documented software that will work for a wide variety of gems.

Open question: Binary library issues

There are known issues when shared libraries are statically linked in a Ruby gem (e.g., Nokogiri with libxml2, OpenSSL with libopenssl or LibreSSL) which are currently being debated and worked around in various ways.

Similarly, the variations on ways to build Ruby can make it challenging to support any arbitrary Ruby (e.g., ENABLED_SHARED can be set on or off). See Luis Lavena's comment on this here.

There is also the challenge that each Ruby install build differently, ruby-build, rvm and others use different flags that affect how extensions are loaded. So much fun!😅

Ruby namespaces, due to be introduced as an experimental feature in Ruby 3.5, may provide better tooling around this (e.g., this commit in Ruby calling dlopen with RTLD_LOCAL instead of RTLD_GLOBAL).

Build farms

Prior art: Luis Lavena's proof-of-concept

In 2020, Luis Lavena (former maintainer of the Windows RubyInstaller project) posted some tweets demonstrating a proof-of-concept precompilation of native gems:

There were enough complications around binary compatibility to make Luis suspend his efforts, as reflected in his followup tweets here and here:

There are several challenges on this (build infra & cost & gem authors) but http://rubygems.org not allowing different Ruby's ABI for same gem version (forcing the fat-binary thing) which complicates build process 😿 Been working on something for this.

There is also the challenge that each Ruby install build differently, ruby-build, rvm and others use different flags that affect how extensions are loaded. So much fun!😅

What's needed

If we had improved CI-focused tooling in place, and the more flexible gem specification was adopted, then a build farm is feasible if someone has enough money.

In my mind, this would take the form of a build process that would automate something like the testing matrix described at flavorjones's ruby-c-extensions-explained project:

Some developer experience things we should think about:

Open question: Do we really need a build farm?

I suspect that many gems' native extensions are going to be difficult to automate, either because

If the CI-focused tooling is easy enough to use, especially on Github, could we make it trivial to opt-in to the tooling to build and test precompiled, fully attestated binary package? I think we probably can.

Declaring external dependencies to avoid compilation

Many (most?) C extensions exist only to integrate with a C/C++/Rust shared library. For many years, there have been ad-hoc methods of declaring a gem dependency on an external library, to either make compiling the C extension very quick, or to avoid it altogether (e.g., using FFI from a pure Ruby library).

Timeline:

Trust

2021-04-16

In 2021, I posted on Twitter asking people about their trust in precompiled gems.