Calling Go functions from LUA

A few days ago, I came across an interesting article, Calling Go Functions from Other languages. In it, Vladimir Vivien creates a small shared library in Go, which he then invokes from C, Java using Java Native Access, and from Python, Node, Ruby using Foreign Function Interface libraries. I thought it would be a fun exercise to interface LUA, or more accurately LUAJIT with the library which was written here.

A little background - why LUA? I’ve been using LUA in production for several years, to provide low latency (0-1ms) API endpoints. I’ve used it to add some sort of CGI capability to nginx - to process LESS, SASS, SCSS and even ES6 on server-side. Recently I’ve used it to build image crop/resize API endpoints, using the magick library, which also uses FFI to load imagemagick or magickwand libraries and invoke their functions/methods. I’ve had to fork and add things to my needs, because the original library and forks only exposed a part of the library API which was needed.

After all this work with LUA, I was already aware that LUAJIT has FFI, and that I could now use LUA to invoke Go functions.

After looking at the code of the other FFI interfaces I wasn’t exactly sure what I was in store for. I just started to “hack” it together, and see how far I would get. These are some of the lessons I learned when I was using LUAJIT to interface with Go.

I can’t use the .h file directly

You build your C shared library from Go code like this:

go build -o awesome.so -buildmode=c-shared awesome.go

When go builds your shared library, it also generates a awesome.h header file, which has all the definitions a program needs to invoke your functions and expose your data structures. You can use these definitions with FFI, and declare them in LUA by wrapping it in a ffi.cdef block:

ffi.cdef([[
  // your C definitions here
]]);

So simple! So, is FFI a simple bridge to C in the background? Can we just include the .h file and basically have a one-liner that would expose the Go functions and structures to LUA? No. Unfortunately, no preprocessor is included, so things like #define will not work. That leaves us to copy-paste most of the .h file to LUA.

local ffi = require("ffi")
local awesome = ffi.load("./awesome.so")

ffi.cdef([[
typedef long long GoInt64;
typedef unsigned long long GoUint64;
typedef GoInt64 GoInt;
typedef GoUint64 GoUint;
typedef double GoFloat64;

typedef struct { const char *p; GoInt n; } GoString;
typedef struct { void *data; GoInt len; GoInt cap; } GoSlice;

extern GoInt Add(GoInt p0, GoInt p1);
extern GoFloat64 Cosine(GoFloat64 p0);
extern void Sort(GoSlice p0);
extern GoInt Log(GoString p0);
]]);

I removed all the unused types (8, 16, 32 bit precision ints), and typedef __SIZE_TYPE__ GoUintptr; because it was unrecognized by LUAJIT. It’s enough for what we need.

Simple datatypes, integer and float

Things get a bit more interesting when we’re dealing with data types - as they are returned, and as they are passed to Go. Most of this is just what I could google and find, with more of a trial-and-error approach until everything “fits”.

The first thing, functions invoked from awesome library return cdata (a C data object). By virtue of the fact that this object is not a LUA native type, it needs to be converted to one.

n = tonumber(cdata)

Converts a number cdata object to a double and returns it as a Lua number. This is particularly useful for boxed 64 bit integer values.

So, with our first case, we need to use tonumber to get a LUA number from the cdata object, and then use math.floor to convert it to an integer. With the Cosine function, we don’t need to cast to int.

io.write( string.format("awesome.Add(12, 99) = %f\n", math.floor(tonumber(awesome.Add(12,99)))) )
io.write( string.format("awesome.Cosine(1) = %f\n", tonumber(awesome.Cosine(1))) )

Declaring Go tables

When it comes to tables, this is the C declaration of the data that needs to go into a Slice:

typedef struct { void *data; GoInt len; GoInt cap; } GoSlice;

So, the slice data is referenced by a simple pointer (void *), and the other two values are the slice length and capacity. When it comes to the Node implementation, the declaration of the Slice was quite simple:

nums = LongArray([12,54,0,423,9]);
var slice = new GoSlice();
slice["data"] = nums;
slice["len"] = 5;
slice["cap"] = 5;

But, the complexity is hidden behind the LongArray declaration, which is a construct of the library ref-array. In comparison, the Ruby declaration is more accurate in terms what’s being done here:

nums = [92,101,3,44,7]
ptr = FFI::MemoryPointer.new :long_long, nums.size
ptr.write_array_of_long_long  nums

In Ruby, a long long * pointer is created and allocated to the size of the nums array. The array is copied into this memory by a call to ptr.write_array_of_long_long. So, it seems when it comes to constructing the table, we should do something similar with LUA FFI.

local nums = ffi.new("long long[5]", {12,54,0,423,9})
local numsPointer = ffi.new("void *", nums);
local typeSlice = ffi.metatype("GoSlice", {})
local slice = typeSlice(numsPointer, 5, 5)

I came to this point after realizing that I can’t set a LUA table object to the Slice.data; And then I also realized that I can’t set cast a LUA table to a pointer (ffi.new void * …), and only after reading the other FFI examples I realized that the table type was abstracted away in the Node example. Of course it’s only logical that C code can only consume complex types that are declared in C space.

This is the final state that I came up with. It declares a long long[5] in C space, and sets the values from the LUA table. The declaration is a bit simpler than what we have in Ruby. As the C code from the Go .h file expects a void * pointer, I declared this as well, so I can then pass it to the Sort function.

Declaring Go strings

Thankfully, strings are a much more normal data type.

typedef struct { const char *p; GoInt n; } GoString;

As such, they consist only of a pointer to a string (const char *) and an integer with the length of the string.

local typeString = ffi.metatype("GoString", {})
local logString = typeString("Hello LUA!", 10)
awesome.Log(logString)

Putting it all together

You can review the full source of the LUA FFI interface to the Go shared library on this gist; And of course, here’s the obligatory money shot of the thing in action:

$> luajit client.lua
awesome.Add(12, 99) = 111.000000
awesome.Cosine(1) = 0.540302
awesome.Sort([12,54,9,423,9] = 0, 9, 12, 54, 423
Hello LUA!

Update: I submitted a PR to vladimir’s github repo, so you may now review my LUA example along with all the others.

While I have you here...

It would be great if you buy one of my books:

I promise you'll learn a lot more if you buy one. Buying a copy supports me writing more about similar topics. Say thank you and buy my books.

Feel free to send me an email if you want to book my time for consultancy/freelance services. I'm great at APIs, Go, Docker, VueJS and scaling services, among many other things.