2021-02-17: The Stakker actor runtime: Beyond "Go++"

It's been more than a year since Stakker's first release, and it is now shipping in a commercial product, so it seems like time to announce it and to compare notes on Rust actor systems in general.

Contents:

Rust + async/await + runtime = Go++

Rust's async/await was a big step forward, but now a lot of people seem to think that that's the end of the story, and that's how concurrency should be done in Rust. However Rust can go much much further than that. Async/await plus an associated runtime is effectively "Go++", i.e. you can do almost everything Go can do but in a low-level Rust style.

However the fact that there is a proliferation of actor crates shows that it doesn't cover everyone's requirements. Async/await emphasises sequential execution. Awaiting on something means your coroutine is blocking on one thing and can't receive any other events. Whilst you can choose to await on multiple things, that is not what this model is streamlined for at the source level.

The actor model on the other hand is the opposite. In the purest actor model, nothing blocks. By default everything is asynchronous and events can arrive from any source at any time. Multiple calls can be outstanding on an actor, not just one as in the case of await.

For things that are asynchronous in a big way with events arriving from all directions, or where it's just not going to work out to try to force things down a mostly sequential code path, or where it is much cleaner to reason about things as a state machine plus incoming events (which is essentially what an actor is), then you really need an actor system.

Different interpretations of the actor model

There are different points of view on how to manage actor queues.

Pony-like: My own crate Stakker takes a pure "nothing blocks" approach, something like the Pony language. Once an actor is up and running, nothing blocks — ever. Events arriving at an actor must be handled immediately. If handling an event starts a process that may take some time, then the actor must arrange to be notified when that process is complete. This is very clean and easy to reason about. Events from all different directions multiplex seamlessly at the actor.

Await-blocking: It seems that some actor implementations choose to temporarily block their queue to facilitate interop with async/await. For example an actor may stop processing its own queue whilst it waits for an external process to complete, e.g. an await call on something from the async/await ecosystem. It seems to me that leaving stuff unprocessed on the queue adds latency and maybe could even lead to deadlocks if the coder isn't careful. What happens if something arrives that changes your view on whether you should continue with the awaited operation? It seems like a half-way compromise between the async/await and actor models.

Erlang-like: In Erlang you can scan your queue looking for particular types of messages. I don't know whether anyone implements this in Rust, but it certainly makes the queuing more complicated. Again it's up to the coder to make sure that they don't leave important messages unprocessed on their queue. Given the popularity of Erlang and Elixir, it must work out for those coders, though.

Note that there is an essential impedence mismatch between the async/await model and the actor model. If we have an awaitable object and wish to interface it to an actor system, the cleanest way is to wrap it in an actor and have that actor accept calls and queue them up internally, and then feed them to the awaitable object one by one. Then the rest of the actor system can run without blocking. This is because you can't have two awaits running at the same time on an awaitable object, which is a limitation that actors don't have.

When multi-threaded is slower than single-threaded

It's known, with rayon for example, that if your unit of work is too small, then parallelizing the job will make it slower. So even though you now have 4 or 8 threads working on the problem, it takes longer to execute than with a single thread. This is because the synchronization overheads dominate the execution time. So to parallelize your job effectively, you have to split it into larger units of work.

The same applies to actor systems. Typically the work done when an actor handles a message is very small: update the state, and maybe send a message or two. So spreading the handling of individual actor messages across threads is a bad idea because the unit of work is way too small. So any implementation of an actor system on top of channels will not be very efficient. You need to find a way to break things into larger units of work before sharing the work between threads.

This is one of the reasons why Stakker chooses to be single-threaded at its core (although several independent Stakker runtimes can be run on different threads). This does shift the burden of dividing work between threads back to the coder, a job which the coder would perhaps prefer not to have to think about. However if you care about efficiency, then you have to think about it. And in any case, without a lot of care, throwing more threads at an actor system will just make it slower. So typical actor code will likely run faster single-threaded anyway.

There has been a lot of pressure on coders over the years to move away from single-threaded coding styles and to accept multi-threaded coding. But this has been reduced to "single-threaded bad, multi-threaded good", so people see a multi-threaded solution and think that it must be better. But that's just not true. The whole thing is a lot more nuanced than that. Really it comes down to: How big is your unit of work? i.e. how much work can you get done between synchronization points.

People may say "but single-threaded doesn't scale". Well, multi-threaded doesn't scale either — at least it only lets you scale to the capacity of the machine. Beyond that you have to think about load balancing or sharding or some other mechanism to distribute work between different machines. So in any case adopting multi-threading is only going to solve your problems temporarily, and for actors will possibly even make things worse.

So given that for an actor work-load, the unit of work is often small compared to synchronization overheads, running multiple single-threaded actor runtimes has its advantages. Load-balancing is something you're going to have to solve at a higher level anyway if you really want to scale.

Stakker features

Here are some of the features that hopefully make Stakker stand out:

  • Static checks: Everything possible is statically checked to find bugs at compile-time rather than run-time. Whereas other actor systems may rely on dynamic checks behind the scenes to maintain safety (e.g. RefCell), Stakker does this at compile-time. It uses the qcell crate to extend Rust's borrow-checking into actors. For example, this guarantees and proves at compile-time that no actor can access any other actor's state. So you can have confidence that your code will not unexpectedly panic due to a coding error causing a check to fail. It also means that the checks have no runtime overhead.

  • High efficiency: Message queueing and execution does not require locks or atomics or allocations or match. Stakker does not use structures for messages, so does not need to match on them. Instead a message is a closure that makes a static call to a method on the destination actor struct. When an actor sends a message, it adds a FnOnce to an internal queue that directly calls that method. Rust is free to inline all that code, so handling a message can be reduced to a single branch to a piece of optimised inlined code that directly modifies the actor's state. Rust may even choose to inline the constant values that you're passing within the message, effectively giving you specialization of the handler too. The FnOnce queue is a flat area of memory, so typically no allocations are required to queue or execute a message either.

  • Choice of implementations: Stakker provides a choice of internal implementations controlled by cargo features, all behind the same fixed API. So if you're running just one Stakker instance, internally it can use globals as an optimisation, but if you change your mind, you can just add a feature and it will switch off that optimisation. If you don't want to risk the unsafe code within Stakker, you can turn off unsafe, and it will use safe alternatives (at some cost in memory and time).

  • Rust-native: Stakker is as low-level and Rust-like as possible. Everything that makes Rust what it is has been extended into the actor model. So it is not an emulation of some other actor system on top of Rust, with all the inefficiencies that brings. Rather it aims to be a fully Rust-native actor system. Amongst other things, this means everything possible is borrow-checked and type-checked; it means that the Rust compiler has static knowledge of what you're doing and can inline and optimise; it means that you can count on drop cleaning things up; and it means that everything is safe.

  • Event system independent: It is not tied to any particular underlying I/O or event system. So it can layer on top of anything. Already implemented is mio, but it should be possible to layer it on top of tokio or async_std if required, or even on top of event loops from other languages. (One of the design requirements was that it should integrate well with C++ applications.) It requires just one timer from the external event system to drive the whole Stakker timer queue.

  • Shared state: It's pragmatic about intentionally shared state. Whilst shared state is not allowed in the pure actor model, there is no way in Rust to stop someone passing around an Rc<RefCell<V>>> — at least not without limiting other useful features of Rust. So Stakker has a Share<V> to make this explicit, and also to make it statically-checked and safe from panics. Since shared state is explicit, its use can be monitored in the codebase.

  • Single-threaded: Each Stakker instance runs single-threaded, so there are no locks, no atomics, no memory fences, etc, etc. Your code will run unimpeded on that thread, using the full speed of the core.

  • Solid handling of failure: Actors can be arranged in trees, and if required actors can be set up to automatically fail upwards, destroying each actor and its children as the failure propagates. In addition a Ret will inform the caller if it is dropped, i.e. if the actor it was sent to (or that was storing it) has died, so callers have a way to deal with actor failure too.

  • Virtualization of time: Stakker doesn't know where you got your Instant from, and it doesn't care. So you can make time run faster for tests.

  • Zero overhead: Really zero overhead and as close to the metal as possible, wherever possible.

Resources

History

  • Initial development of qcell happened 2018-2019
  • A simple Rust actor library was developed in Rust in 2019
  • This was redesigned and rewritten as the open-source Stakker crate later in 2019, with first release in Jan-2020
  • As of Feb-2021 Stakker is shipping in a commercial product

Some future possibilities for Stakker

  • Actor coroutines that can be started by the actor itself, and that run alongside the actor with direct safe access to the actor's state. This would allow certain parts of the actor's behaviour to be coded in a sequential way where that suits them. Multiple coroutines could be run for a single actor. Unfortunately it's impossible to implement this on top of async/await, because it needs an 'until_next_yield lifetime. With a planned feature of Rust generators it should be possible.

  • Remote actors, i.e. allow sending calls to actors in other threads or on other machines.

  • Crates to layer Stakker on top of tokio or async_std, and to wrap awaitable objects.

  • Support for offloading CPU-intensive work or I/O work to a threadpool. This would be cleanest if integrated with actor coroutines.

  • Maybe allow Actor<dyn Trait> instead of Actor<Box<dyn Trait>>, if Rust's new union of ManuallyDrop feature turns out to be helpful.