Linkers & Ruby C Extensions

June 7, 2020

I recently learned that linkers are really cool. It all started when I saw an error message that looked something like this:

❯ rake test
symbol lookup error: /home/jez/.../foo.so: undefined symbol bar

I already wrote about finding where this error was coming from. The tl;dr is that it was coming from GNU’s libc implementation:

❯ rg -t c 'symbol lookup error'
dl-lookup.c
876:      _dl_signal_cexception (0, &exception, N_("symbol lookup error"));

That led me to a fun exploration of how linux linkers work, and how Ruby C extensions rely on them.

I always knew that Ruby C extensions existed (that they break all the time is a constant reminder…) but I never really connected the dots between “here’s some C code” and how Ruby actually runs that code.

Ruby C extensions are just shared libraries following certain conventions. Specifically, a Ruby C extension might look like this:

#include "ruby.h"

VALUE my_foo(VALUE self, VALUE val) {
  return rb_funcall(self, rb_intern("puts"), 1, val)
}

// This function's name matters:
void Init_my_lib() {
  rb_define_method(rb_cObject, "foo", my_foo);
}

The important part is that the name of that Init_my_lib function matters. When Ruby sees a line like

require_relative './my_lib'

it looks for a file called my_lib.so (or my_lib.bundle on macOS), asks the operating system to load that file as a shared library, and then looks for a function with the name Init_my_lib inside the library it just loaded.

When that function runs, it’s a chance for the C extension to do the same sorts of things that a normal Ruby file might have done if it had been require’d. In this example, it defines a method foo at the top level, almost like the user had written normal Ruby code like this:

my_lib.rb
def foo(val)
  puts val
end

That’s kind of wild! That means:

I was pretty shocked to learn this, because my mental model of how linking worked was that it split evenly into two parts:

There’s actually a third option!

Then I looked into what code Ruby actually calls to do this. I found the code in dln.c:

dln.c
/* Load file */
if ((handle = (void*)dlopen(file, RTLD_LAZY|RTLD_GLOBAL)) == NULL) {
    error = dln_strerror();
    goto failed;
}

→ View on github.com

Ruby uses the dlopen(3) function in libc to request that an arbitrary user library be loaded. From the man page:

The function dlopen() loads the dynamic shared object (shared library) file named by the null-terminated string filename and returns an opaque “handle” for the loaded object.

— man dlopen

The next thing Ruby does with this opaque handle is to find if the thing it just loaded has an Init_<...> function inside it:

dln.c
init_fct = (void(*)())(VALUE)dlsym(handle, buf);
if (init_fct == NULL) {
    const size_t errlen = strlen(error = dln_strerror()) + 1;
    error = memcpy(ALLOCA_N(char, errlen), error, errlen);
    dlclose(handle);
    goto failed;
}

→ View on github.com

It uses dlsym(3) (again in libc) to look up a method with an arbitrary name (buf) inside the library it just opened (handle). That function must exist—if it doesn’t, it’s not a valid Ruby C extension and Ruby reports an error.

If dlsym found a function with the right name, it stores a function pointer into init_fct, which Ruby immediately dereferences and calls:

dln.c
/* Call the init code */
(*init_fct)();

→ View on github.com

It’s still kind of mind bending to think that C provides this level of “dynamism.” I had always thought that being a compiled language meant that the set of functions a C program could call was fixed at compile time, but that’s not true at all!

This search led me down a rabbit hole of learning more about linkers, and now I think they’re super cool—and far less cryptic! I highly recommend Chapter 7: Linking from Computer Systems: A Programmer’s Perspective if this was interesting to you.

Read More

Search Down the Stack

I’ve found it useful to search though the source code of things lower in the stack lately. Continue reading

Sorbet Does Not Have FixMe Comments

Published on February 11, 2020