Lachlan's Scala Blog

Using a C library from Scala Native

Using a C library from Scala Native

Going deep on a non-trivial library

Lachlan O'Dea's photo
Lachlan O'Dea
·

14 min read

Table of contents

I've recently been experimenting with Scala Native. The goal I've been working towards is to make the libuv library usable from Scala Native. libuv is a C library for asynchronous I/O that is at the core of Node.js. There is a lot of work left to be done, but I'll share what I've learned so far.

My experience and knowledge of Scala Native is still pretty limited, so please let me know if I get any facts wrong.

Introduction to Scala Native

At the time I'm writing this, the latest version of Scala Native is 0.4.16, and all my observations here apply to using that version. The pre-1.0 version number is definitely justified. It's not terribly difficult to get a Scala Native project up and running, but there's a lot of rough edges, rougher than "normal" JVM Scala, which itself isn't always the smoothest experience.

Scala Native works by compiling ahead of time to native code using LLVM. The amazing thing about it is that it really is "just Scala". Type in whatever Scala code you like and you get a native executable out with the same semantics as you'd get on the JVM, but no JVM needed. There are a few differences, the only really significant one being there's no support for threads right now.

But there's a catch. In most real scenarios where Scala is used you don't just use Scala, you also use a bunch stuff from the Java ecosystem, especially in the app's lower layers. Consider how many Scala projects rely on Netty for their network I/O. On native we get all of Scala, but no Java libraries.

Scala Native does offer a subset of the JDK API, which has been re-implemented in Scala. This is very useful because there's so much Scala code out there that uses JDK APIs.

Using native libraries

On the JVM, our Scala code has to eventually end up calling JDK APIs as they're the only way to actually do anything I/O related on that platform. On native, we need to call C APIs to do our I/O. We could use the native APIs of our OS, but then to support multiple OS platforms we'll have to deal with the many differences between the platforms. This is why something like libuv is so useful, it takes care of the OS differences and provides us with a C API for fully asynchronous I/O that (mostly) works the same way on Windows and any Unix flavour.

As we're about to see, using a C library from Scala Native is much more difficult than using a Java library from Scala JVM. While Java APIs often aren't ideal for use from Scala in terms of their design, they can still be used just like a Scala library, and objects can be passed back and forth freely.

In the native world, all this is much more difficult. There are three reasons for this, going from most significant to least:

  • The C ABI leaves a lot to be desired in terms of interoperability between languages, but it's all we have, see C isn't a programming language anymore

  • Scala is very high-level (for example, it has full GC), which puts it further away from C than, say, Rust is

  • Scala Native's tooling isn't yet as developed as some other languages

The problems

The docs on how Scala Native interoperates with native code can be found here. I'm not going to go into how to define bindings to the C functions in Scala and mapping the C types, I think the docs cover that well. Overall it works pretty nicely, but there's a number of problems we have to deal with that don't exist on the JVM.

Memory management

Scala is a language that requires garbage collected memory, and this is what you get with Scala Native. But native code has no knowledge of the Scala heap and GC. To interoperate with native code, we need to switch to native memory management like we would do in C. Scala Native supports allocating memory on the stack, or the native heap via malloc.

Scala Native also has this neat "zone allocation" API which allocates memory on the native heap like malloc, but once the execution leaves Zone block, the memory is automatically freed:

Zone { implicit zone =>
  val ptr = alloc[Int](200)
  // use ptr
}
// the memory at ptr is now freed

Much nicer than malloc/free, but there are often situations where the lifetime of an allocation doesn't fit neatly into a lexical scope like this. This comes up all the time with libuv in fact, because its asynchronous nature forces you to split logic over multiple callbacks. There's no alternative but to manage memory manually in such cases.

One neat thing about Scala 3 is the inline keyword makes stack allocation easier to use. One of the (many) classic ways to blow your own foot off in C is to return a pointer to memory on your own function's stack:

def foo(): Ptr[Int] = {
  ptr = unsafe.stackalloc[Int](10)
  p(0) = 1
  p(1) = 2
  p
}
val badPtr = foo()
// badPtr now points to stack space that is considered free
// and can be overwritten at any time

but make foo inline and the stack allocation happens on the callers stack, and everything's fine:

inline def foo(): Ptr[Byte] = ...

Things only the C compiler can know

While we have a standard C ABI, actually using that in practice requires knowing a few things, like

  • The values of enums, integer codes with special meaning, error codes, etc

  • The precise size and layout of any structs that we need to allocate or use

While in principle you can just create Scala definitions for all this, in practice this can be difficult if either of these things are true:

  • you don't control the native code you're using

  • you want to support multiple platforms

In the case of libuv, both of these are true.

For example, if you want to open a file for writing, you need to pass the O_WRONLY value in the int oflags parameter of the open function. When writing in C you can just do open(path, O_WRONLY) and you get the right value for the O_WRONLY constant. But to write that in Scala, we need to somehow figure out the actual numeric value to use. To open a TCP socket you need to allocate and initialise a sockaddr_in structure, meaning you need to know the exact size and layout of that structure.

So what's the problem? Just read the docs, or the source and figure this out. Well, the issue with docs is they typically don't specify enough details to make these APIs usable across the C ABI, they only specify enough details for the API to be used from from C source code. Docs will say "there are flags named X and Y", but not their literal values. Docs often specify that a structure will have fields X and Y, but that doesn't tell you all the fields, or how they're laid out.

Ok, read the source then. Yes this works, but it's not always easy. You have to find which header file these definitions live in, and there can be all manner of C preprocessor directives that determine what you actually end up with on a given platform. You can deal with this by running just the preprocessor on your platform and seeing what you end up with. I'm not sure I actually want to do that on a regular basis.

But the real tough problem comes if you need to support more than one native platform. Now you have to figure out if these things are defined in exactly the same way on all the platforms you care about. And if they aren't identical (which is common, even a basic structure like sockaddr_in has minor variations that may matter), then what? Make your Scala code platform-specific? I'd rather not.

All languages that do C interop solve all the above in the same basic way: integrate an actual C compiler into your toolchain, as that's the only practical solution. Python has this neat thing where you can specify just the fields of a structure that are known, along with ..., and the Python tools will use the C compiler to figure out the exact memory layout of the structure on the platform you're building for, and fill that in for you. Scala Native doesn't have that level of sophistication, not yet at least, but it does have a solution that works: write "glue" code in C.

Glue code

The way to solve the above problems in Scala Native is instead of trying to call the third-party native code directly from Scala, you write your own C code to provide an API that is known and consistent across platforms, which is easy for your Scala code to interface with.

While this is generally a tedious task, Scala Native does make it pretty easy. You put a .c file in your project's src/main/resources/scala-native directory and define an @unsafe.extern object with a matching name, along with the function bindings, and it just works.

Constants

As an example, libuv defines a set of error code constants. The names of these constants are the same on all platforms but the values are different on Windows and Unix. To "import" these into Scala, I created a file src/main/resources/scala-native/errors.c that defines a bunch of single line functions to return every error code (libuv conveniently provides a C macro that can be used to generate these). Then in Scala, we have:

@extern
private[scalauv] object errors {

  def uv_scala_errorcode_E2BIG(): ErrorCode = extern
  def uv_scala_errorcode_EACCES(): ErrorCode = extern
  ... and so on

A lot of boilerplate to be sure, but it works on every platform!

Structures

Another example is accessing fields of libuv's uv_connect_t structure. The docs do not specify this structure's layout, only that it has certain "public" fields like uv_stream_t *uv_connect_t.handle. So to make an API to read the handle field that works regardless of structure layout, we write:

uv_stream_t *uv_scala_connect_stream_handle(const uv_connect_t *req)
{
    return req->handle;
}

Glue like this is needed for any language, not just Scala. As libuv is commonly used from languages other than C, the authors have actually built some of these glue functions into libuv itself. For example, libuv provides the following function:

size_t uv_req_size(uv_req_type type)

which returns the size of the structure for a particular request type. This function is useless when writing in C, as you can just use the appropriate structure directly. Similarly for field access functions like uv_req_get_data. Unfortunately these built-in functions don't cover all the cases I needed, so I had to write several glue functions myself.

A consequence of needing to resort to all this glue code for libuv's semi-abstract, platform-dependent structures is that none of them are defined in Scala using unsafe.CStruct. On the Scala side, they're all typed as Ptr[Byte] and we rely on native code for all operations on them.

Pass by value

There is one additional case where glue code is needed, which is specific to Scala Native. Currently, Scala Native cannot pass C structures by value. This isn't a very common practice in C, but it does come up. In libuv, there is a constructor function for its uv_buf_t structure that handles layout differences between Windows and Unix, and this returns the structure by value. There's currently no way to write an @extern binding for this. The solution is to write a glue function that passes the structure by reference:

void uv_scala_buf_init(char *base, unsigned int len, uv_buf_t *buffer)
{
    uv_buf_t buf = uv_buf_init(base, len);
    buffer->base = buf.base;
    buffer->len = buf.len;
}

With Scala 3 inlining, at least we can make this transparent to the caller:

  inline def stackAllocateBuffer(
      ptr: Ptr[Byte],
      size: CUnsignedInt
  ): Buffer = {
    val uvBuf = stackalloc[Byte](structureSize)
    helpers.uv_scala_buf_init(ptr, size, uvBuf)
    uvBuf
  }

Function pointers

libuv supports asynchronous operations, and as a result it makes heavy use of callbacks. To do callbacks in C you need to pass a pointer to the C function to be called back on. In order for a library like libuv to be useful for Scala programs, it is essential that the native code can callback to code written in Scala, which can then update state on the Scala "side".

Scala Native supports defining C function pointers using the standard function literal syntax we all know and love:

type StreamWriteCallback = CFuncPtr2[WriteReq, CInt, Unit]

def onWrite: StreamWriteCallback = {
  (req: WriteReq, status: ErrorCode) =>
    status.onFailMessage(setFailed)
    val buf = Buffer.unsafeFromNative(uv_req_get_data(req))
    stdlib.free(buf.base)
    buf.free()
    stdlib.free(req)
}

This is great, but despite the syntactic similarity to a normal Scala function, C function pointers have limitations that don't apply to Scala functions. The docs don't currently mention these limitations, and that I only discovered them by trial and error.

If your C function pointer closes over local scope, it won't compile:

object Foo {
  def test = {
    val foo = "s"
    def cFunc: CFuncPtr0[String] = { () =>
      foo
    }
    cFunc
  }
}

gets you Closing over local state of parameter foo$1 in function transformed to CFuncPtr results in undefined behaviour.

Worse, if you close over a field of a class, it will compile but then crash with a segmentation fault or similar. What's going on?

From my experiments, I think it works as follows. Even though Scala Native allows you to define C function pointers anywhere you like, they always end up as globals, in the top level object or companion object of whatever lexical scope you're in. This means that they can only close over global values. I think this is likely just a limitation of how function pointers work in C, as it doesn't support closures at all. A C function pointer literally points at the first instruction of your function in the TEXT segment of your program's binary, so it's necessarily global in scope.

I think the best way to define function pointers is to always put them in a top-level object. My number one request for the Scala Native documentation would be for someone who actually knows how function pointers work to describe the restrictions and best practices.

This global nature is quite a painful thing to deal with in practice. For a callback where you need to alter the state of the Scala world, you have two options:

  • Operate on global state. The downsides of this are well-known, but in certain situations it might make sense.

  • Somehow pass a reference to the relevant Scala context into the callback.

The second mechanism is very common in callback systems, which usually offer a way for the user to pass arbitrary data through to their callback. In the case of libuv, the structures passed around to the callbacks have a data field that we can fill in with whatever pointer we like.

Which leads us to the next problem...

Passing Scala references to native code

We have to pass through to our callback a reference to whatever Scala context it needs to operate. Maybe it's a data structure to be updated based on what we read from a TCP socket. But the libuv native code can't work with Scala references, we need to pass it a value that we can convert back to the original Scala reference in our callback. The only way to do this that I know of is to convert the reference to a raw pointer value.

Scala reference → convert to raw pointer → write raw pointer to data field when initiating request → read raw pointer from data in callback → convert from raw pointer → original Scala reference.

If the Scala Native docs mention how to convert Scala references to raw pointers, then I missed it. But there is an API to do it, maybe it's considered internal? I'm not sure. But when doing this, the issue of memory management hits us again:

  • The Scala garbage collector can't tell that native code effectively has an active reference to the object, so we have to make sure that it isn't collected too early

  • Some garbage collectors can move objects to different memory locations, and if this happens it would render the raw pointer invalid

My understanding is that the second issue can be ignored for Scala Native as the GC it uses does not move objects. But I think the first problem remains, as creating a raw pointer does not create a GC reference (and which would also necessitate a way to manually release that reference, which doesn't seem to exist). So we need to somehow keep a reference to our context on the Scala side until the native code has completely finished using the raw pointer we made from it.

This is definitely something that Scala Native needs to make easier, somehow. If there is an easier way that I've missed, I'd love to know about it!

Conclusion

My attempt at Scala bindings for libuv is imaginatively called scala-uv and can be found here. It is at the point where you can do some basic file I/O and can connect and use TCP sockets, but much of the libuv API isn't covered yet.

By creating the Scala bindings we end up with an API that works exactly like the original C version. As a result, when we use it we end up basically doing C programming with Scala syntax. I'm not sure if you're aware of this, but programming in C is terrible.

We really need to go further, to build a truly Scala-friendly API on top of the raw C bindings. We don't want Scala developers to need to worry about C-isms like function pointers and manual memory management. scala-uv includes a few conveniences to make some things like error handling easier than the C equivalent, but it's far from a nice Scala API. This is something I hope to work on in the future.

This process did remind me of some of the JVM's good points. These days I tend to gripe about having to run my production Scala code on the JVM, but wow it really does let you skip a lot of extremely tedious low-level issues.

But regardless, I'm hopeful for Scala Native. What the Scala Native team have achieved so far is quite remarkable. It's still early days, and using native libraries is definitely not trivial, but the potential outcome seems very compelling to me.

scala-nativeScalascala3,