Someone recently asked us for a high-level overview of Simula's software stack. Whether you're interested in contributing or simply curious about how our open-source software fits together, this post sketches out how our principal repositories link together:

• SimulaVR/Simula
• SimulaVR/godot-haskell
• SimulaVR/gdwlroots
• SimulaVR/wlroots
• SimulaVR/godot

1 High-Level Overview

At a high level, Simula is a Wayland compositor inserted into a game engine and managed via Haskell glue. A rough formula, in terms of our repositories, is

1. Game Engine. For our rendering system, we use a fork of Godot (SimulaVR/godot), which is an open-source game engine written in C++.
2. Wayland Compositor. For our window management, we use a fork of Drew Devault's wlroots (SimulaVR/wlroots), which provides lots of the Wayland infrastructure we've assembled into our VR compositor. Gdwlroots (SimulaVR/gdwlroots) provides a Godot interface for this infrastructure.
3. OpenXR Runtime. Our program connects to VR hardware via monado, which is an open-source implementation of the OpenXR API.
4. Glue. The integration of everything is handled via Haskell code in our main repository (SimulaVR/Simula). In order to integrate Haskell code with Godot, we've had to generate our own bindings (SimulaVR/godot-haskell).

2 Wayland Compositor

2.1 Wayland Protocol

Wayland is a modern compositor protocol which aims to replace X11.[1] Unlike X11, Wayland programs (or "clients") are 100% responsible for rendering their own window contents. A Wayland "server" (or "compositor") is then responsible for (i) presenting (or "compositing") the resulting window surfaces for display, (ii) providing a window management system to the user (typically involving UX widgets like task switchers, app launchers, and lock screens), and (iii) receiving and arbitrating among different inputs (keyboard presses, mouse clicks, etc).

Wayland clients communicate with a Wayland compositor via the "Wayland protocol". In practice, this means messages are sent back and forth over a UNIX domain socket asynchronously. Here are some example messages a Wayland client might send to a Wayland compositor:

• "I'm a Wayland client, and I just launched!"
• "I just finished rendering a frame, so feel free to draw me now!"
• "I just closed!"

On the other hand, a Wayland server sends messages to clients like

• "Hey client, a mouse cursor just entered your window!"
• "Hey client, the letter "A" was just typed!"

Messages from Wayland clients ultimately bubble up to our high-level Haskell code in the form of event handlers, which in turn trigger useful things to happen:

[1] For more information on the strengths of Wayland over X11 for VR, check out Forrest Reiling's 2014 Masters Thesis: Towards General Purpose 3D User Interfaces: Extending Windowing Systems to Three Dimensions. This paper heavily influenced Simula's early development.

2.2 XWayland

Wayland is cool, but the vast majority of the Linux ecosystem runs on X11. So…how does Simula support these applications? The answer is Xwayland, which is an X11 server that masquerades as a Wayland client:

When X11 apps are launched within Simula, Xwayland intercepts these X11 windows, then helps them act as first-class residents among the other Wayland clients. Thus our Wayland compositor really has two different window management systems working together: a Wayland Window Manager (WWM), and an X11 Window Manager (XWM).

2.3 SimulaVR/wlroots

In practice, managing and responding to the flow of messages among Wayland clients requires lots of infrastructure: useful wayland types, event handlers, Xwayland support, and so forth. Drew Devault's wlroots' handles a lot of this for us. As wlroots' README states:

wlroots provides unopinionated, mostly standalone implementations of many Wayland interfaces, both from wayland.xml and various protocol extensions. We also promote the standardization of portable extensions across many compositors.

...

wlroots implements a huge variety of Wayland compositor features and implements them right, so you can focus on the features that make your compositor unique. By using wlroots, you get high performance, excellent hardware compatibility, broad support for many wayland interfaces, and comfortable development tools - or any subset of these features you like, because all of them work independently of one another and freely compose with anything you want to implement yourself.

This was perfect for Simula's use case. It allowed us to use the parts of a "standard" Wayland compositor that we needed, while discarding those that didn't fit our VR use case. Simula uses a fork of wlroots at SimulaVR/wlroots.[2]

[2] Earlier prototypes of Simula actually used weston as a stripped-down reference compositor. We ended up abandoning those efforts and switched to wlroots.

3 Game Engine

Our Godot fork (SimulaVR/godot) is responsible for handling most of our project's rendering pipeline; it also manages our 3D scene graph, physics interactions, and input mappings, among other things.

Early prototypes of Simula actually had us managing our own rendering loop. As the project grew in complexity we decided to move to a full fledged game engine. We chose to use Godot for the following reasons:

• Linux support. At the time Godot was one of the few game engines with native Linux support.

• Open-source. Godot being open-source allows us to extend the game engine to support proper Linux window management.

• Useful features. Out of the box, the engine provides useful features like 3D physics and scene graph management. Furthermore, it integrates nicely with Linux input devices (keyboards, mice, etc).

• Cross-language. Godot supports cross-language scripting (useful for us, since we like to use Haskell for our glue), and the ability to link to native libraries (which we need for our window management infrastructure).

4 OpenXR

OpenXR is a VR/AR API which provides a standardized protocol for communication between XR hardware and software.

Monado is an open-source implementation of this API. Godot supports a godot-openxr plugin, used by Simula, which allows the game engine to easily interface with an OpenXR runtime (like monado) without much fuss. You can find our godot-openxr assets in ./addons/godot-openxr.

5 Haskell Glue

5.1 SimulaVR/godot-haskell

In order for us to inject Haskell code into Godot, we had to create Haskell bindings for all of the functions in the Godot API. This led to the creation of SimulaVR/godot-haskell, which converts Godot's object-oriented/C++ methods into Haskell style ones.

Let's take the CanvasItem's method draw_texture as an example:

//Draws a texture at a given position.
void draw_texture ( Texture texture, Vector2 position, Color modulate=Color( 1, 1, 1, 1 ), Texture normal_map=null )

Using some nifty Template Haskell, godot-haskell allows us to use this function on a CanvasItem value as follows:

import qualified Godot.Methods as G -- godot-haskell module

G.draw_texture :: GodotCanvasItem -> GodotTexture -> GodotVector2 -> GodotColor -> GodotTexture -> IO ()

Notice the Godot* types in the function signature: GodotCanvasItem, GodotVector2, and so forth. Godot-haskell treats these as "low level" Godot types. But sometimes we want to use idiomatic, "high level" Haskell types. For this, godot-haskell provides toLowLevel and fromLowLevel typeclass methods for all Godot* types. This allows us to easily switch back and forth between the two worlds in our code. For example, with GodotVector2:

import Linear -- Idiomatic Haskell vectors

godotV2 <- toLowLevel (V2 0 0) :: IO GodotVector2 -- Go to GodotVector2 from V2 Float
haskellV2 <- fromLowLevel godotV2 :: IO (V2 Float) -- Go from GodotVector2 to V2 Float

5.1.1 The Haskell Tradeoff

Godot-haskell allows us to manage our Godot program from Haskell's walled garden of type safety. With that said, it's not a magic bullet. At its core, Godot is still an object-oriented system with a C++ API. As such, it still requires us to manually manage memory cleanup when Godot doesn't do it for us. It also forces large portions of our program to be jammed into the IO monad (leading to a "Haskell-C++" coding style).

On the bright side, Haskell does allow us to make these tradeoffs explicit in our code. Overall, we've greatly enjoyed managing our lower level infrastructure using this language, and still think it's an underrated language for ambitious projects.

5.2 SimulaVR/Simula

Our main respository is SimulaVR/Simula, which contains all of our high-level code which glues all of these pieces together. It also serves as the basis of our Godot project.

5.2.1 Godot project files

Since this repo serves as the location from which Simula is launched, it is also structured as a typical Godot project. As such you can find our project.godot config at project root, an .import folder with our default engine assets pre-loaded, and other project resources in the places Godot expects them (./default_env.tres, ./export_presets.cfg, and so forth). If you're not familiar with these things, Godot has some good tutorials on how its 3D projects are set up.

5.2.2 Haskell NativeScript Modules

Our actual Haskell code is located in ./addons/godot-haskell-plugin. Most of the modules found here act as GDNative "NativeScript" modules.[3] NativeScript is a Godot tool which allows you to write script logic in C/C++, similar to how it would be written in a GDScript file. Using godot-haskell, we have brought this capability to Haskell. Thus you can think of our Haskell modules as replicating the functionality of GDScript files, but with typesafe Haskell idioms.

As a simplified example: instead of creating a simple.gd GDScript file like this:

extends RigidBody

func _ready():
    print("Hello, world!")

…we can instead create a Simple.hs Haskell module like this:

module Simple where

data Simple = Simple {
  _simpleObj :: GodotObject
}

instance HasBaseClass Simple where
  type BaseClass Simple = GodotRigidBody
  super (Simple obj)  = GodotRigidBody obj

instance NativeScript Simple where
  className = "Simple"
  classMethods =
    [
      func NoRPC "_ready" (_ready)
    ]

_ready :: Simple -> [GodotVariant] -> IO ()
_ready self _ = do
  putStrLn "Hello World!"

We are glossing over details here; however, this is the basic structure of most of our Haskell modules (found in ./addons/godot-haskell-plugin/src/*).

[3] You can check this out for a reference tutorial on how a normal C GDNative/NativeScript module is implemented.

5.2.3 Putting It All Together

Using the above infrastructure, our Haskell NativeScript modules (located in ./addons/godot-haskell-plugin/src/*) are finally able to glue our compositor, game engine, and OpenXR logic into one cohesive program. Here's a rough sketch of what they do:

1. Engage VR connectivity. First we connect to an OpenXR run-time and add VR headset nodes to the Godot scene graph.
2. Set up compositor. Next we start our Wayland compositor, and an instance of XWayland. Then we sync a bunch of window manager handlers to their proper window management events, so that the desired things happen when windows are launched, mapped, closed, and so forth. In particular, we set up logic so that launched clients are added to the Godot scene graph and rendered in the proper order each frame.
3. Window manipulation. With the windows added to the Godot scene graph, we then provide users with a way to move, resize, and interact with them. This involves routing all Godot input actions (key presses, mouse movements, etc) to the desired wlroots surfaces.
4. Configuration. We have code parsing user configs to allow for keyboard shortcuts and other custom settings. This includes things like launching apps, resizing or grabbing windows, or changing environments.
5. Pancake Mode. We also have some Haskell code which sets up a "pancake view", allowing users to interact with Simula windows outside of VR.
6. Useful tools & debugging support. Finally, we have code which allows users to do things like take in-game picture or video recordings, log data, and other useful things.

The end result is a full-fledged VR compositor running on top of Godot and wlroots infrastructure.