How to build a VR retro arcade: do it yourself!

In part 1, you got an intro and team retrospective of the “retro arcade” hack week project, where the whole team banded up and built a fun project together that really put Alloverse to the test. In part 2, you got an architecture overview, explaining the components involved in bringing a SNES emulator into a collaborative VR environment.

In this part 3, you’ll get a step-by-step tutorial in building it yourself! This is going to be intermediate-to-advanced, and a lot of code. The point is to demo the capabilites of the Alloverse platform, and show you some of the really advanced stuff that is possible with LuaJIT and AlloUI. If that sounds appealing to you, strap in and let’s go!

Let’s get to coding!

We start out by creating an AlloUI project somewhere on our computer. (This is the same as the first step in our Getting Started guide for the Lua language. If you want an easier starter project, I recommend following that guide!)

$ mkdir myarcade
$ cd myarcade
$ git init
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/alloverse/alloapp-assist/master/setup.bash)"
$ git commit -am “initial"
$ ./allo/assist run alloplace://sandbox.places.alloverse.com # just to confirm it worked

If you open up the Alloverse app and connect to sandbox.places.alloverse.com, you should now be seeing your app in there as a flat surface with a button on it.

Stubs and dummies

This flat slab is an eyesore. Let’s spice it up and add some basic UI, make it feel like a real object that the user will want to walk up to and interact with. Replace lua/main.lua with the code below. It instantiates stubs for Emulator and RetroMote; loads a fancier model file; creates the “main UI” view that everything will attach to; places game controllers at some good default location; and kicks off the runloop for the app.

local client = Client(arg[2], "myarcade")
app = App(client)
local Emulator = require("Emulator")
local RetroMote = require("RetroMote")

assets = {
    arcade = ui.Asset.File("models/220120-arcade.glb"),
}
app.assetManager:add(assets)

local main = ui.View(Bounds(0.2, 0.1, -4.5,   1, 0.2, 1))
main:setGrabbable(true)

local emulator = Emulator(app)

local cabinet = main:addSubview(ui.ModelView(Bounds.unit():scale(0.3,0.3,0.3), assets.arcade))

local controllers = cabinet:addSubview(View())
controllers.bounds:scale(5,5,5):move(0,5.6,-1.4)
emulator.controllers = {
    controllers:addSubview(RetroMote(Bounds(-0.15, -0.35, 0.6,   0.2, 0.05, 0.1), 1)),
    controllers:addSubview(RetroMote(Bounds( 0.087, -0.35, 0.6,   0.2, 0.05, 0.1), 2))
}

app.mainView = main

app:connect()
app:run(emulator:getFps())

You’ll need three more files before this runs.

1st file: Download and place the arcade cabinet model file into models/. This’ll be the fancy arcade cabinet model that the user will want to walk up to.

2nd file: Create lua/Emulator.lua, where we just stub out an Emulator class. We will later use this to wrap all of libretro in a self-contained class.

local class = require('pl.class')
Emulator = class.Emulator()

function Emulator:getFps()
    return 60
end

return Emulator

3rd file: Create lua/RetroMote.lua, which will be our game controller object that we can pick up in VR to control the retro game being played. For now, this is also just an empty stub subclassing ui.View.

local class = require('pl.class')

class.RetroMote(ui.View)

return RetroMote

Run it…

$ ./allo/assist run alloplace://sandbox.places.alloverse.com 

and you should have a blank cabinet in the Sandbox place, yay!

Using FFI to hook up libretro to Emulator.lua

Now to make the arcade machine actually DO something. This is going to be a bit of a journey: we’ll be using LuaJIT’s FFI (”foreign function interface”) to talk directly to libretro’s C API. This way, we’ll be able to tell libretro to emulate games, send controller input to it, and receive back audio and video from the emulated system.

The libretro API is thankfully all in a single header, but LuaJIT can’t read it as-is, since the LuaJIT FFI lacks a preprocessor (and also a few other C features). I’ve gone ahead and preprocessed the header for you using the advanced tool “my brain and hands”, so that LuaJIT can read it. It’s too long to paste here; instead, download it from here, and put it into lua/cdef.lua.

Head over to lua/Emulator.lua. Splat this at the top:

local class = require('pl.class')
local tablex = require('pl.tablex')
local pretty = require('pl.pretty')
local vec3 = require("modules.vec3")
local mat4 = require("modules.mat4")
local ffi = require("ffi")
local RetroMote = require("RetroMote")

ffi.cdef(require("cdef"))

(we’ll be needing all those other requires later, so might as well get them in there now).

That last ffi.cdef line is all that’s needed for you to be able to call all of libretro as if it was a lua library.

We just need to dynamically load it. Which means we need libretro somewhere on your system…

Installing libretro on your machine

Right, okay. If you’re on Linux:

sudo add-apt-repository ppa:libretro/stable && sudo apt-get update && sudo apt-get install retroarch
sudo apt-get install libretro-nestopia libretro-genesisplusgx libretro-snes9x
sudo apt install libavcodec-dev libavformat-dev libswresample-dev libswscale-dev

On a Mac:

1. Install RetroArch from https://www.retroarch.com/
2. Launch RetroArch, and use the menus to install the following cores: Snes9X, Genesis Plus GX, Nestopia.

On Windows:

It should be totally doable to get retroarch installed in a way that’s compatible with this project on Windows, at least if you’re using mingw or something; but it’s not a setup I’m familar with so I can’t provide a detailed guide here.

Initializing libretro

Let’s dynamically load our newly installed libretro. Below are some helpers that…

1. locate the libretro dynamic library on your platform
2. load it using ffi.load() so you can call functions from it!
3. set all the callbacks to closures that call our own functions, so we can start reacting to events happening in libretro. (just setting self.handle.retro_set_input_state = self._input_state wouldn’t work, because the callback wouldn’t be called with the correct self instance. So we need to capture self with a closure and call the own method with it.)
4. and finally, set up controllers and init the library!

Go ahead and add this code to lua/Emulator.lua:

function os.system(cmd)
    local f = assert(io.popen(cmd, 'r'))
    local s = assert(f:read('*l'))
    f:close()
    return s:match("^%s*(.-)%s*$")
  end

function _loadCore(coreName)
    local searchPaths = {
        "~/.config/retroarch/cores/"..coreName.."_libretro.so", -- apt install path
        "/usr/lib/x86_64-linux-gnu/libretro/"..coreName.."_libretro.so", -- gui install path linux
        "$HOME/Library/Application\\ Support/RetroArch/cores/"..coreName.."_libretro.dylib" -- gui install path mac
    }

    for i, path in ipairs(searchPaths) do
        print("Trying to load core from "..path)
        local corePath = os.system("echo "..path)
        ok, what = pcall(ffi.load, corePath, false)
        if ok then
            print("Success")
            return what
        else
            print("Failed: "..what)
        end
    end
    error("Core "..coreName.." not available anywhere :(")
end

function Emulator:loadCore(coreName)
    if coreName == self.coreName then return end
    self.coreName = coreName
    self.handle = _loadCore(coreName)
    assert(self.handle)

		-- libretro uses this callback to poll us for settings it should use
    self.handle.retro_set_environment(function(cmd, data)
        return self:_environment(cmd, data)
    end)
	  -- libretro calls us asking what the state of the game controllers are
    self.handle.retro_set_input_state(function(port, device, index, id)
        return self:_input_state(port, device, index, id)
    end)
	  -- libretro has some image data for us
    self.handle.retro_set_video_refresh(function(data, width, height, pitch)
        return self:_video_refresh(data, width, height, pitch)
    end)
		-- libretro has some audio data for us
    self.handle.retro_set_audio_sample_batch(function(data, frames)
        return self:_audio_sample_batch(data, frames)
    end)
    self.handle.retro_set_controller_port_device(0, 1); -- controller port 0 is a joypad
    self.handle.retro_set_controller_port_device(1, 1); -- controller port 1 is a joypad
    self.handle.retro_init()
end

So as you can see, we’re able to call C methods like retro_set_controller_port_device(unsigned port, unsigned device) directly from Lua now that we’ve cdef’d its interface. The only bummer is that we can’t use the handy macros like RETRO_DEVICE_JOYPAD, so we have to send in the raw numbers that they map to. But at least we don’t have to manually allocate ffi memory for the arguments! It’s just all handled automatically.

(It’s also blows my mind every time an FFI interface is able to assign a closure to a regular old C function pointer.)

Loading games

While we’re at it, let’s load the game too.

Emulator.coreMap = {
  sfc = "snes9x",
  smc = "snes9x",
  nes = "nestopia",
  smd = "genesis_plus_gx",
}

function Emulator:loadGame(gamePath)
		-- figure out which core to use based on file extension
		local ext = assert(gamePath:match("^.+%.(.+)$"))
		local core = assert(Emulator.coreMap[ext])
		self:loadCore(core)
		
		self.gamePath = gamePath
		
		-- this is how you allocate more complex data types, like structs. 
		self.system = ffi.new("struct retro_system_info")
		-- luajit assumes your reference is a pointer, so we can send it byref to populate it
		self.handle.retro_get_system_info(self.system)
		
		self.info = ffi.new("struct retro_game_info")
		self.info.path = gamePath
		local f = io.open(gamePath, "rb")
		local data = f:read("*a")
		self.info.data = data
		self.info.size = #data
		local ok = self.handle.retro_load_game(self.info)
		assert(ok)
		
		self:fetchGeometry()
end

We use ffi.new to allocate named C structures (which have previously been handed to luajit with the ffi.cdef method). With retro_get_system_info we’re then passing the allocated structure to get it filled in so we can use it later; and with retro_load_game we’re passing in a structure that we have filled with data so that libretro can work with it. The data in question is the full game just read from disk and mashed into RAM. (there are also APIs for streaming game data so that one could play bigger games, but that’s overkill for what we’re doing here).

Constructor

Later on we’re going to want to render the game’s graphics, so we need to know what the size of the screen is going to be (”geometry”), so we set out to fetch that. Let’s also get the class’s constructor in place and finish up all the setup we needed:

function Emulator:_init(app)
    self.app = app

		-- app will need to set speaker to a ui.Speaker and screen to a ui.VideoSurface
		-- to present audio and video. we'll get to that.
    self.speaker = nil
    self.screen = nil
		
    self.sample_capacity = 960*32
    self.audiobuffer = ffi.new("int16_t[?]", self.sample_capacity)
    self.buffered_samples = 0
    self.audiodebug = io.open("debug.pcm", "wb")
    self.soundVolume = 0.5
    self.frameSkip = 2 -- 1=60fps, 2=30fps, etc
    self.onScreenSetup = function (resulution, crop) assert("assign onScreenSetup plz") end
end

function Emulator:fetchGeometry()
    self.av = ffi.new("struct retro_system_av_info")
    self.handle.retro_get_system_av_info(self.av)
    print(
        "Emulator AV info:\n\tBase video dimensions:", 
        self.av.geometry.base_width, "x", self.av.geometry.base_height,
        "\n\tMax video dimensions:",
        self.av.geometry.max_width, "x", self.av.geometry.max_height,
        "\n\tVideo frame rate:", self.av.timing.fps,
        "\n\tAudio sample rate:", self.av.timing.sample_rate
    )

    self.resolution = {self.av.geometry.base_width, self.av.geometry.base_height}
    -- this callback is used to ask the UI layer to create a VideoSurface of the correct dimensions
    self.onScreenSetup({self.av.geometry.base_width, self.av.geometry.base_height}, {self.av.geometry.max_width, self.av.geometry.max_height})
end

Run-fix-repeat: The “environment”

At this point, it’s kind of easier to just try to run it and fix all the runtime errors, than to read documentation and figuring out exactly what needs to be configured to make things work. We’ll be doing run-fix-repeat cycles now until we have a working emulator.

Let’s make the code load up a game. Please legally acquire a NES or SNES rom file for a game you enjoy, and put it in roms. Then, change the bottom of main.lua to read:

emulator:loadGame("roms/rom.sfc") -- or whatever you named your game rom
app:connect()
app:run(emulator:getFps())

So, first try: where do we crash?

$ ./allo/assist run alloplace://sandbox.places.alloverse.com
Trying to load core from ~/.config/retroarch/cores/snes9x_libretro.so
Success
./allo/deps/luajit-bin//bin/linux64/luajit: ./allo/../lua/Emulator.lua:77: attempt to call method '_environment' (a nil value)

Okay. So immediately upon loading a game, libretro is asking us for “environment”, which means it’s asking us for runtime settings. Let’s implement it and check which setting it’s asking for:

$ ./allo/assist run alloplace://sandbox.places.alloverse.com
Emulator is asking for setting #	34

34. What does 34 mean? If we go back to libretro’s API and just search for 34, we’ll find it means RETRO_ENVIRONMENT_SET_SUBSYSTEM_INFO. We don’t care about subsystems, so we can just return false to say “we have no environment value for setting 34, sorry”.

If you’re feeling extra ambitious, you can now go ahead and run-fix-repeat for all the requested environment settings until it works… or you can copy-paste my implementation here 😅

function Emulator:_environment(cmd, data)
    if cmd == 27 then -- RETRO_ENVIRONMENT_GET_LOG_INTERFACE
        local cb = ffi.cast("struct retro_log_callback*", data)
        cb.log = self.helper.core_log
        return true
    elseif cmd == 3 then -- RETRO_ENVIRONMENT_GET_CAN_DUPE
        local out = ffi.cast("bool*", data)
        out[0] = true
        return true
    elseif cmd == 10 then -- RETRO_ENVIRONMENT_SET_PIXEL_FORMAT
        local fmt = ffi.cast("enum retro_pixel_format*", data)
        local fmtIndex = tonumber(fmt[0])
        local indexToFormat = {
            [0]= "rgb1555",
            [1]= "bgra", -- ?? supposed to be xrgb8
            [2]= "rgb565",
        }
        self.videoFormat = indexToFormat[fmtIndex]
        print("Emulator requested video format", fmtIndex, "aka", self.videoFormat)
        return true
    elseif cmd == 9 then -- RETRO_ENVIRONMENT_GET_SYSTEM_DIRECTORY
        local sptr = ffi.cast("const char **", data)
        sptr[0] = "."
        return true
    elseif cmd == 31 then -- RETRO_ENVIRONMENT_GET_SAVE_DIRECTORY
        local sptr = ffi.cast("const char **", data)
        sptr[0] = "."
        return true
    elseif cmd == 32 then -- RETRO_ENVIRONMENT_SET_SYSTEM_AV_INFO
        print("System av info changed")
        return false
    elseif cmd == 37 then -- RETRO_ENVIRONMENT_SET_GEOMETRY
        local geometry = ffi.cast("struct retro_game_geometry*", data)
        print("New geometry")
        self:fetchGeometry()
        return true
    end

    --print("Unhandled env", cmd)
    return false
end

You’ll note that we just return false for all the things we don’t care about. Some notes:

• libretro MUST have a logging callback. And, it must be a vararg function. luajit’s FFI doesn’t support that, so we’re going to have to implement that in C. BUMMER. We’ll get to that later.
• You’ll note that data is a void pointer, so we have to cast it to whatever is appropriate for the given setting, and then set it with pointer dereferencing. Since pointers and single-value-arrays are the same thing in C, we can dereference a pointer by referring to its first array value.
• The rest of the settings should be fairly self-explainatory; if not, ping @nevyn on our Discord and ask him to explain it to you 😅

Run-fix-repeat: The logging helper

Running the code above will crash on field 'helper' is a nil value. So let’s implement the helper. Download libretro.h into c/libretro.h. Implement c/helper.c:

#include "libretro.h"
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>

void core_log(enum retro_log_level level, const char *fmt, ...) {
    char buffer[4096] = {0};
    static const char * levelstr[] = { "dbg ", "info", "warn", "err " };
    va_list va;

    va_start(va, fmt);
    vsnprintf(buffer, sizeof(buffer), fmt, va);
    va_end(va);

    if (level == 0)
        return;

    fprintf(stderr, "[%s] %s", levelstr[level], buffer);
    fflush(stderr);

    if (level == RETRO_LOG_ERROR)
        exit(EXIT_FAILURE);
}

Write a Makefile to build it (in the project root):

.PHONY : clean

CFLAGS += -fPIC -g
LDFLAGS += -shared

SOURCES = $(shell echo c/*.c)
HEADERS = $(shell echo c/*.h)
OBJECTS=$(SOURCES:.c=.o)

TARGET=lua/libhelper.so

all: $(TARGET)

clean:
	rm -f $(OBJECTS) $(TARGET)

$(TARGET) : $(OBJECTS)
	$(CC) $(CFLAGS) $(OBJECTS) -o $@ $(LDFLAGS)

We’ll also have to load this new libhelper.so at runtime. In Emulator.lua, right after self.handle = _loadCore(coreName), add:

self.helper = ffi.load("lua/libhelper.so", false)

Try it out:

$ make all
$ ./allo/assist run alloplace://nevyn.places.alloverse.com
Trying to load core from ~/.config/retroarch/cores/snes9x_libretro.so
Success
Map_HiROMMap
[info] "Street Fighter2 Turbo1" [checksum ok] HiROM, 32Mbits, ROM, NTSC, SRAM:0Kbits, ID:B___, CRC32:D43BC5A3

Woah. It works?! It’s even using our logging callback to print the game name, which means we’re officially emulating a game inside a VR (even though we can’t see it yet).

Hey, you. Well done getting here! You deserve a fika break. Get a coffee and a cinnamon bun. I’m going to do that too, because the above was quite a handfull. Hopefully by the time you’re done, I’ve published part 4, and we can dig into displaying video, playing audio, and receiving controller input.

If you have any questions or feedback, please head over to our Discord and let us know!