mnemOS

the mnemOS Book

This is the documentation for mnemOS (mnɛːmos), a hobby-grade, experimental operating system for small computers (and bigger ones, too). The mnemOS book includes a summary of mnemOS's design and a tour of the operating system's components, as resources for mnemOS users and contributors, and the rendered documentation for accepted mnemOS RFCs.

The mnemOS Book is an living document, and is updated on a best-effort basis. As mnemOS is in active development, some documentation in the book may not always be up to date with the latest state of the project. Also, some parts of the book have yet to be written.

If you encounter missing or inaccurate information, please let us know by opening a GitHub issue, or feel free to open a pull request to add to the book!

elsewhere

In addition to this documentation, you can find more about mnemOS here:

license

This documentation is dual-licensed under the MIT + Apache 2.0 licenses.

code of conduct

The mnemOS project follows the Contributor Covenant Code of Conduct.

other resources

In addition to this documentation, there are a number of other resources about mnemOS, including talks, blog posts, and API documentation.

talks

podcast episodes

  • Chats With James Episode 011, James Munns and Eliza Weisman (2022-04-12), in which we discuss some ideas that would end up being central to the design of MnemOS
  • Chats With James Episode 012, James Munns and JT Turner (2023-07-17), including a discussion of the design of MnemOS' Forth-based shell environment

development blogs

These blog posts follow the story of MnemOS's development:

generated documentation

what is mnemOS?

mnemOS is a small, general purpose OS. It aims to bridge the gap between project specific bare-metal applications or real-time operating systems (RTOS), and larger more complete general purpose operating systems (like Linux).

It's a hobby project of mine, though more people have started contributing lately! It doesn't have any specific financial backer or particular goals, but I am using it to research areas of "what is possible" on lightweight embedded systems. The project is written entirely in Rust.

It comes from making (or making part of) a bunch of projects that were really too big and complex to be a bare metal project, and I realized I kept building a bunch of fragile, incompatible parts of an operating system for each of the projects. After getting overwhelmed on the last one, I decided to just build an OS I could use.

mnemOS is targeted at supporting both microcontroller and microprocessor systems, at least for now. I tend to do a lot of projects that span the sort of range from a medium sized microcontroller (32-bit, 64-128MHz, 128+KiB SRAM, 256KiB+ Flash), up to a small sized microprocessor (32/64-bit, 500MHz+, 64-512MiB DRAM, 8MiB+ Flash). At least for my hobby projects, there isn't a lot of price difference between the price points of those two classes of chips, though sometimes one will have features (CPU, RAM, Peripherals, existing drivers) that make one choice more appropriate.

As a general purpose OS, it doesn't aim to necessarily be suitable for super time-critical functionality (it may have non-deterministic scheduling or resource usage), and instead is aimed at making other capabilities like networking, file system support, user interface support, and code re-use a higher priority.

why should I (or you) use mnemOS?

I don't have a good answer! There is certanly no commercial or technical reasons you would choose mnemOS over any of its peers in the "hobbyist space" (e.g. Monotron OS, or projects like RC2014), or even choose it over existing commercial or popular open source projects (like FreeRTOS, or even Linux). It's mainly for me to scratch a personal itch, to learn more about implementing software within the realm of an OS, which is relatively "high level" (from the perspective of embedded systems), while also being relatively "low level" (from the perspective of an application developer).

At the moment, it has the benefit of being relatively small (compared to operating system/kernel projects like Linux, or Redox), which makes it easier to change and influence aspects of the OS. I don't think it will ever be anything "serious", but I do plan to use to it to create a number of projects, including a portable text editor, a music player, and maybe even music making/sythesizer components. Some day I'd like to offer hardware kits, to make it easier for more software-minded folks to get started.

For me, it's a blank slate, where I can build things that I intrinsically understand, using tools and techniques that appeal to me and are familiar to me. I'd love to have others come along and contribute to it (I am highly motivated by other people's feedback and excitement!), but I'll keep working on it even if no one else ever shows up. By documenting what I do, I'll gain a better understanding (and an easier route to picking it up if I have to put it down for a while), and that work serves to "keep the lights on" for any kindred spirits interested in building a tiny, simple, PC in Rust.

If that appeals to you, I invite you to try it out. I am more than happy to explain any part of mnemOS. Much like the Rust Programming Language project - I believe that if any part of the OS is not clear, that is a bug (at least in the docs), and should be remedied, regardless of your technical skill level.

where does the name come from/how do I pronounce it?

"mnemOS" is named after Mnemosyne, the greek goddess of memory, and the mother of the 9 muses. Since one of the primary responsibilities of an OS is to manage memory, I figured it made sense.

In IPA/Greek, it would be mnɛːmos. Roughly transcribed, it sounds like "mneh-moss".

To listen to someone pronounce "Mnemosyne", you can listen to this youtube clip, and pretend he isn't saying the back half of the name.

If you pronounce it wrong, I won't be upset.

Parts of mnemOS

mnemOS conceptually is broken up into three main parts:

  • the kernel, which provides resources like an allocator, an async executor/scheduler, and a registry of active/running drivers
  • the drivers, which are async tasks that are responsible for all other hardware and system related functionality
  • the user programs, which use portable interfaces to be able to run on any mnemOS system that provides the drivers it needs.

the kernel

The kernel is fairly limited, both at the moment, and in general by design. It provides a couple of specific abilities:

the allocator

The kernel provides an allocator intended for use by drivers. Rather than RTOS or other more "embedded style" projects, dynamic allocation can be used to spawn driver tasks, and allocate resources, like how large buffers are, how many concurrent requests can be made at once, etc. mnemOS does NOT use the standard Rust allocator API, and provides its own.

Although an allocator is available, it is not intended to be used in the "normal case", such as sending or receiving messages. Instead, buffers should be allocated at setup or configuration time, to reduce allocator impact. For example: setting up the TCP stack, or opening a new port might incur an allocation, but sending or receiving a TCP frame should not. This is not currently enforced, and is a soft goal for limiting memory usage and fragmentation.

the executor/scheduler

As initial versions of mnemOS are intended to run on single core devices, and hard-realtime is not a specific goal, the kernel provides no concepts of threading for concurrency. Instead, all driver tasks are expected to run as cooperative async/await tasks. This allows for efficient use of CPU time in the kernel, while allowing events like hardware interrupts to serve as simple wakers for async tasks.

The executor is based largely on the maitake library. Maitake is a collection of no-std compatible executor building blocks, which are used extensively. This executor serve the purpose of scheduling all driver behaviors, as well as kernel-time scheduling of user applications.

the driver registry

In order to dynamically discover what drivers are running, the kernel provides a driver registry, which uses UUIDs to uniquely identify a "driver service".

By default, drivers are expected to operate in a message-oriented, request-reply fashion. This means to interact with a driver, you send it messages of a certain type (defined by the UUID), and it will send you a response of a certain type (defined by the UUID). This message passing is all async/await, and is type-safe, meaning it is not necessary to do text or binary parsing.

Additionally, drivers may choose whether they also make their services available to user programs as well. This interface will be explained later, when discussing user space programs.

Another day, another chunk of writing.

Parts of mnemOS - continued

In the last post, I covered the kernel. This post moves on to the drivers, which are async tasks that are responsible for all other hardware and system related functionality.

the drivers

As a micro-kernel-ish operating system, most functionality is provided by drivers. Some of the "how" is still being finalized, but this is a look at current/planned capabilities.

drivers as a service

In mnemOS' driver model, drivers are expected to act as services, in a sort of REST or RPC sort of way (drawing parallels to web or microservice style techniques). They take a specific request type (as a Rust data type), and provide a specific response type (also as a Rust data type). For example, a driver that hands out virtual serial ports might have a request type that looks like this:

#![allow(unused)]
fn main() {
pub enum Request {
    /// Register a virtual serial port with a given port ID and
    /// buffer capacity
    RegisterPort { port_id: u16, capacity: usize },
    /* other request types not shown... */
}
}

and a matching response type that looks like this:

#![allow(unused)]
fn main() {
pub enum Response {
    /// A port has been successfully registered
    PortRegistered(PortHandle),
    /* other response types not shown... */
}

/// A PortHandle is the interface received after
/// opening a virtual serial port
pub struct PortHandle {
    port: u16,
    cons: bbq::Consumer,
    outgoing: bbq::MpscProducer,
    max_frame: usize,
}
}

This message passing is done in an async way, so if the message queues are ever full, the sender can await until there is capacity available, and the receiver is transparently notified (and scheduled to run).

Each service is identified using a UUID, such as 54c983fa-736f-4223-b90d-c4360a308647, for the virtual serial port service. This UUID can be registered by any driver implementer that uses the same request and response types, which means that different platform-specific drivers can implement the same interface, when necessary or preferable. This allows drivers services to be "generic" over their implementation, without having complicated type relationships.

This use of UUID borrows heavily from how Bluetooth Low Energy works, with a UUID identifying a Characteristic, or a specific kind of API.

These request and response types and UUID value are specified through a trait called RegisteredDriver, which is explained in the driver registry RFC. The RFC goes into a great bit more detail of how this "type safe service discovery" mechanism actually works under the hood.

two kinds of drivers

In practice, this means that there will end up being two main kinds of drivers:

  • platform specific drivers
  • portable drivers

Platform specific drivers are drivers that are expected to only work on a specific device, or family of devices. Although many microcontrollers and microprocessors have "Serial", "UART", or "USART" ports, the code necessary to configure them, and have them efficiently send and receive bytes, varies incredibly widely. However as we've seen with the embedded-hal traits in bare-metal Rust, it is often very possible to have a portable interface that covers MOST common use cases.

Even though these platform specific drivers are implemented in very different ways, they would be expected to use a common interface at a high level. This consistent lower interface allows for portable drivers to work regardless of the underlying platform specific drivers in use.

In contrast, portable drivers ONLY rely on other driver services, meaning that as long as the services they depend on exist, they will be able to operate regardless of the actual system they are running on. For example, we might have a couple of high level driver services, like:

  • logging and tracing info
  • a command line interface
  • a system status display

These driver services would ONLY rely on the virtual serial port interface, which provides multiple "ports" over a single serial port. In turn, the virtual serial port interface relies on the platform-specific hardware serial port interface.

By implementing ONLY a platform specific driver for a serial port, someone porting mnemOS to a new platform would gain access to use all four of these services (virtual serial port, logging/tracing, CLI, and system status) automatically.

exposing drivers to userspace

note: this is an area that is still under construction. some parts as described already exist in the code, but some do not yet.

In the "kernelspace", where the kernel and drivers exist, we can leverage compile time type safety, because all drivers and the kernel will be compiled together into one binary (at the moment mnemOS does not support "dynamically loading" drivers, they must be statically compiled together).

This is an important distinction because Rust does NOT have a stable ABI, and types and layout can change at any time, even between compilations. In "userspace", where user applications execute, we will not have compiled the applications at the same time as the kernel. They are two completely separate binaries!

To get around this, we can still use an async message-passing interface, but the requests and response will be serialized and then deserialized. Using serde and the postcard wire format, we can be sure that data will be consistently interpreted.

Driver services with request/response types that can be serialized can also make their interfaces available to user applications, though this is not required. The userspace can ask if a certain driver service UUID is registered (and available to userspace), and if it is, it can send serialized messages to the kernel, to be forwarded to the drivers. A full round trip looks something like this:

  • The userspace prepares a request, and then serializes it
  • The userspace sends the serialized request to the kernel
  • The kernel determines which service is being messaged, and if it exists, the message is deserialized and sent to the driver
  • The driver processes the request, and sends a response to the kernel to be returned to userspace
  • The kernel serializes the response, and sends it to userspace
  • The userspace deserializes the response, and processes it

How this actual userspace to kernel messaging works will be covered later, when I talk about the userspace itself works.

mnemOS userspace

Coming soon!

platform support

mnemOS attempts to make the bringup process for supporting a new hardware platform as straightforward as possible. The core cross-platform kernel is implemented as a Rust library crate, which platform implementations depend on. The platform implementation, which builds the final mnemOS kernel binary artifact for that platform, is responsible for performing hardware-specific initialization, initializing drivers for hardware devices provided by that platform, and running the kernel. In addition, the kernel crate provides a set of cross-platform default services, which may be run on any hardware platform.

supported platforms

Currently, mnemOS has in-tree platform implementations for several hardware platforms, with various degrees of completeness.

The following platforms are "fully supported" --- they are capable of running the complete mnemOS operating system, with all core functionality implemented:

Other platform implementations are less complete, and undergoing active development:

mnemOS RFCs

RFC - Drivers and Discovery

Goal

MnemOS should have some way of "automatically discovering" drivers. This would be used by three main "consumers":

  • The kernel itself, which may need to use drivers that are provided at initialization time
  • Drivers, which may depend on other drivers (such as a virtual serial mux driver, which depends on a "physical serial port" driver)
  • Userspace, which will typically interact with drivers through either the stdlib, or other driver crates.

Background

Right now, when spawning drivers, the process goes roughly as follows:

  • The init() code manually constructs drivers, and manually creates any shared resources necessary between drivers
  • The init() code spawns the driver task(s) onto the kernel executor

This is a little manual, and also won't really work for userspace.

There are also some half baked ideas in the abi/mstd crates about having a single "message kind" that is an enum of all possible driver types, but this isn't very extensible

The proposal

This change proposes inntroducing a registry of drivers, maintained by the kernel.

  • This registry would be based on a Key:Value type arrangement:
    • The Key would be a tuple of:
      • A pre-defined UUID, that uniquely defines a "service" that the driver fulfills, e.g. the ability to process a given command type
      • The TypeId of that message
    • The Value would be contain two things:
      • A channel that can be used for sending requests TO a driver
      • If the driver supports serializable/deserializable types, it will also contain a function that can be used to deserialize and send messages via the driver channel.
  • Drivers would be responsible for registering themselves at init() time
  • All (Uuid, TypeId) keys must be unique.

NOTE: Large portions of this proposal were prototyped in the typed-poc repository. Some details were approximated, as this was developed with the standard library instead of MnemOS types for ease.

Details

Some details about why parts of this proposal was chosen:

The RegisteredDriver Trait

In order to collect all of the related types of a service, this RFC introduces a trait called RegisteredDriver. This is primarily a marker trait, and serves to collect types and constants used by a driver. It exists as follows:

#![allow(unused)]
fn main() {
/// A marker trait designating a registerable driver service.
pub trait RegisteredDriver {
    /// This is the type of the request sent TO the driver service
    type Request: 'static;

    /// This is the type of a SUCCESSFUL response sent FROM the driver service
    type Response: 'static;

    /// This is the type of an UNSUCCESSFUL response sent FROM the driver service
    type Error: 'static;

    /// This is the UUID of the driver service
    const UUID: Uuid;

    /// Get the type_id used to make sure that driver instances are correctly typed.
    /// Corresponds to the same type ID as `(Self::Request, Self::Response, Self::Error)`
    fn type_id() -> RegistryType {
        RegistryType {
            tuple_type_id: TypeId::of::<(Self::Request, Self::Response, Self::Error)>(),
        }
    }
}
}

The following types and interfaces will often be generic over the RegisteredDriver type, rather than one or more types, in order to provide a more consistent interface.

The Message Type

All drivers will use a standardized message type which is generic over a RegisteredDriver. This includes the msg, which is the request TO the service, and the reply, which allows to service to reply with a Result<RD::Response, RD::Error> message.

#![allow(unused)]
fn main() {
pub struct Message<RD: RegisteredDriver> {
    pub msg: Envelope<RD::Request>,
    pub reply: ReplyTo<RD>,
}
}

This message type contains a ReplyTo<RD> field, which can be used to reply to the sender of the request. This serves as the "return address" of a request. This enum will look roughly as follows:

#![allow(unused)]
fn main() {
/// A `ReplyTo` is used to allow the CLIENT of a service to choose the
/// way that the driver SERVICE replies to us. Essentially, this acts
/// as a "self addressed stamped envelope" for the SERVICE to use to
/// reply to the CLIENT.
pub enum ReplyTo<RD: RegisteredDriver> {
    // This can be used to reply directly to another kernel entity,
    // without a serialization step
    KChannel(KProducer<Envelope<Result<RD::Response, RD::Error>>>),

    // This can be used to reply directly ONCE to another kernel entity,
    // without a serialization step
    OneShot(Sender<Envelope<Result<RD::Response, RD::Error>>>),

    // This can be used to reply to userspace. Responses are serialized
    // and sent over the bbq::MpscProducer
    Userspace {
        nonce: u32,
        outgoing: bbq::MpscProducer,
    },
}
}

The driver is responsible for definining the Request, Response, and Error request/response types associated with a given UUID.

Using Uuids to identify drivers

There are a lot of different ways to discover drivers used in other operating systems, including using some kind of path, or a type erased file interface.

This proposal takes an approach somewhat similar to Bluetooth's characteristics:

  • A UUID is chosen ahead of time to uniquely identify a service
  • Some "official" UUIDs are reserved for MnemOS and built-in drivers
  • A UUID represents a single service interface
    • Although any driver could implement the UUID, all implementors are expected to have the same interface
    • The UUID has no concept of versioning. If a breaking change is needed, a new UUID should be used

In practice, any unique/opaque identifier could be used, but a UUID is used for familiarity.

When implementing a driver, the driver would be expected to export its UUID as part of the crate. It's likely the MnemOS project would maintain some registry of "known UUIDs", to attempt to avoid collisions. This could be in a text file, or wiki, etc.

Using TypeId

Although all drivers are accessed by a KChannel<T>, which is a heap allocated structure, attempting to hold many channels of different Ts in the same structure wouldn't work.

Instead, the key:value map would need to type-erase the channel, likely just holding some kind of pointer. Theoretically, the UUID should be enough to ensure that the correct T matches the expected type, however the cost of getting this wrong (undefined behavior) due to a developer mistake is too high.

In order to avoid this, the get function ALSO takes a TypeId, which is a unique type identifier provided by the compiler. Since all drivers are statically compiled together, they should all see the same T have the same TypeId.

This WON'T necessarily work for userspace, or if MnemOS ever gets dynamically loaded drivers, which may be compiled with a different version of Rust, or potentially not Rust at all.

For this reason, non-static users of these drivers will need to use the serialization interface instead. Userspace would send a serialized message to the expected UUID. The kernel would then use the associated deserialization function to deserialize the message, and send it to the associated channel

Uniqueness of drivers

This scheme requires that only one version of any kind of driver can be registered.

This simplifies things, but could be potentially annoying for certain drivers are expected to have multiple instances, like keyboards, or storage devices.

This proposal sort of ignores that problem, and suggests that those drivers shouldn't be discoverable. Instead, some kind of "higher level" driver that hands out instances of those drivers should be registered.

(Exclusively) using KChannel<T>

Although a lot of discussion of using "message passing" as a primary interface has been had so far, this would pretty much cement this design choice.

Although there are other communication primitives available ready, such as the various flavors of the bbq based queues, they could be registered and provided through the KChannel as a "side channel". This is how the current (as of 2022-07-18) serial mux interface works:

  • The client who wants to open a virtual port gets the KProducer
  • The client sends a "register port" message
  • On success, the client gets back a bbq channel handle from the driver

Interfaces: Kernel Only

If a type used in the channel does NOT support serialization/deserialization, it is limited to only being usable in the kernel, where this step is not required.

The registry would provide two main functions in this case:

#![allow(unused)]
fn main() {
impl Registry {
    // Register a given KProducer for a (uuid, T, U) pair
    pub fn register_konly<RD: RegisteredDriver>(
        &mut self,
        kch: &KProducer<Message<RD>>,
    ) -> Result<(), RegistrationError> { /* ... */ }

    // Obtain a given KProducer for a (uuid, T, U) pair
    pub fn get<RD: RegisteredDriver>(&self) -> Option<KernelHandle<RD>> { /* ... */ }
}
}

This code would be used as such:

#![allow(unused)]
fn main() {
struct SerialPortReq {
    // ...
}

struct SerialPortResp {
    // ...
}

enum SerialPortError {
    // ...
}

// We have a given "SerialPort" driver
impl RegisteredDriver for SerialPort {
    type Request = SerialPortReq;
    type Response = SerialPortResp;
    type Error = SerialPortError;

    const UUID: Uuid = ...;
}

let serial_port = SerialPort::new();

// In `init()`, the SerialPort is registered
registry.register_konly::<SerialPort::Request, SerialPort::Response>(
    serial_port.producer(),
).unwrap();

// Later, in another driver, we attempt to retrieve this driver's handle
struct SerialUser {
    hdl: KernelHandle<SerialPort>,

    // These are used for the "reply address"
    resp_prod: KProducer<SerialPort::Response>,
    resp_cons: KConsumer<SerialPort::Response>,
}

let user = SerialUser {
    hdl: registry.get_konly(SerialPort::UUID).unwrap(),
};

// Send a message to the driver:
user.hdl.enqueue_async(Message {
    msg: SerialPortReq::get_port(0),
    reply_to: ReplyTo::Kernel(user.resp_prod.clone()),
}).await.unwrap();

// Then get a reply:
let resp = user.resp_cons.dequeue_async().await.unwrap();
}

Interface: Userspace

If a given type DOES implement Serialize/Deserialize, it can also be used for communication with userspace.

The registry provides two additional functions in this case:

#![allow(unused)]
fn main() {
// This is a type that is capable of deserializing and processing
// messages for a given UUID type
impl UserspaceHandler {
    fn process_msg(
        &self,
        user_msg: UserMessage<'_>,
        user_ring: &bbq::MpscProducer,
    ) -> Result<(), ()> { /* ... */ }
}

impl Registry {
    // Register a given KProducer for a (uuid, T, U) pair
    pub fn register<RD>(&mut self, kch: &KProducer<Message<RD>>) -> Result<(), RegistrationError>
    where
        RD: RegisteredDriver,
        RD::Request: Serialize + DeserializeOwned,
        RD::Response: Serialize + DeserializeOwned,
    {
        /* ... */
    }

    // Obtain a userspace handler for the given UUID
    fn get_userspace_handler(
        &self,
        uuid: Uuid,
    ) -> Option<UserspaceHandler>;
}
}

The get_userspace_handler() function does NOT take the T and U types as parameters. If the registry entry for the given uuid was added through the register_konly instead of the register interface, the get_userspace_handler() function will never return a handler for that uuid.

Using the UserspaceHandler

This code would be used as follows:

#![allow(unused)]
fn main() {
#[derive(Serialize, Deserialize)] // Added!
struct SerialPortReq {
    // ...
}

#[derive(Serialize, Deserialize)] // Added!
struct SerialPortResp {
    // ...
}

// We have a given "SerialPort" driver
impl SerialPort {
    type Request = SerialPortReq;
    type Response = Result<SerialPortResp, ()>;
    type Handle = KProducer<Message<Self::Request, Self::Response>>;
    const UUID: Uuid = ...;
}

let serial_port = SerialPort::new();

// In `init()`, the SerialPort is registered, this time with `register` instead
// of `set_only`.
registry.register::<SerialPort::Request, SerialPort::Response>(
    SerialPort::UUID,
    &serial_port.producer(),
).unwrap();

// This is generally the shape of user request messages "on the wire" between
// userspace and kernel space.
struct UserMessage<'a> {
    uuid: Uuid,
    nonce: u32,
    payload: &'a [u8],
}

// Lets say that we have the User Ring worker
impl UserRing {
    async fn get_message<'a>(&'a self) -> UserMessage<'a> { /* ... */ }
    fn producer(&self) -> &bbq::MpscProducer { /* ... */ }
}

let user_ring = UserRing::new();

// okay, first we get a message off the wire
let ser_request = user_ring.get_message().await;

// Now we get the handler for the given uuid on the wire:
let user_handler = registry.get_userspace_handler(ser_request.uuid).unwrap();

// Now we use that handler to process the message:
user_handler.process_msg(ser_request, user_ring.producer()).unwrap();

// Once the user handler processes the message, it will send the serialized
// response directly into the userspace buffer.
}

Implementing the UserspaceHandler

There is one main trick used to generate the UserspaceHandler described above is that the set() function is used to generate a monomorphized free function that can handle the deserialization.

The contents of the UserspaceHandler will look roughly like this:

#![allow(unused)]
fn main() {
// This is the "guts" of a leaked `KProducer`. It has been type erased
//   from: MpMcQueue<Message< T,  U>, sealed::SpiteData<Message< T,  U>>>
//   into: MpMcQueue<Message<(), ()>, sealed::SpiteData<Message<(), ()>>>
type ErasedKProducer = *const MpMcQueue<Message<(), ()>, sealed::SpiteData<Message<(), ()>>>;
type TypedKProducer<T, U> = *const MpMcQueue<Message<T, U>, sealed::SpiteData<Message<T, U>>>;

struct UserspaceHandler {
    req_producer_leaked: ErasedKProducer
    req_deser: unsafe fn(
        UserMessage<'_>,
        ErasedKProducer,
        &bbq::MpscProducer,
    ) -> Result<(), ()>,
}
}

The req_deser function will look something like this:

#![allow(unused)]
fn main() {
type TypedKProducer<T, U> = *const MpMcQueue<Message<T, U>, sealed::SpiteData<Message<T, U>>>;

unsafe fn map_deser<T, U>(
    umsg: UserMessage<'_>,
    req_tx: ErasedKProducer,
    user_resp: &bbq::MpscProducer,
) -> Result<(), ()>
where
    T: Serialize + DeserializeOwned + 'static,
    U: Serialize + DeserializeOwned + 'static,
{
    // Un-type-erase the producer channel
    let req_tx = req_tx.cast::<TypedKProducer<T, U>>();

    // Deserialize the request, if it doesn't have the right contents, deserialization will fail.
    let u_payload: T = postcard::from_bytes(umsg.req_bytes).map_err(drop)?;

    // Create the message type to be sent on the channel
    let msg: Message<T, U> = Message {
        msg: u_payload,
        reply: ReplyTo::Userspace {
            nonce: umsg.nonce,
            outgoing: user_resp.clone(),
        },
    };

    // Send the message, and report any failures
    (*req_tx).enqueue_sync(msg).map_err(drop)
}
}

The fun trick here is that while map_deser is generic, once we turbofish it, it is no longer generic, because we've specified types! Using psuedo syntax:

#![allow(unused)]
fn main() {
// without the turbofish, `map_deser` as a function has the type (not real syntax):
let _: fn<T, U>(UserMessage<'_>, ErasedKProducer, &bbq::MpscProducer) -> Result<(), ()> = map_deser;

// WITH the turbofish, `map_deser` as a function has the type:
let _: fn(UserMessage<'_>, ErasedKProducer, &bbq::MpscProducer) -> Result<(), ()> = map_deser::<T, U>;
}

This means we now have a type erased function handler, that still knows (internally) what types it should use! That means we can put a bunch of them into the same structure.

With this, we can now implement the process_msg() function shown above:

#![allow(unused)]
fn main() {
impl UserspaceHandler {
    fn process_msg(
        &self,
        user_msg: UserMessage<'_>,
        user_ring: &bbq::MpscProducer,
    ) -> Result<(), ()> {
        unsafe {
            self.req_deser(user_msg, self.req_producer_leaked, user_ring)
        }
    }
}
}

RFC - Forth Userspace

Goal

In order to make early prototyping and iterating on the design of mnemos easier, this RFC proposes the addition of a single kind of "userspace" environment, a forth vm that can act as a program or interactive shell interface. This RFC also proposes a minimal set of capabilities and interfaces necessary to make this possible.

Background

mnemos has some design around what it means to launch kernel services, but has no kind of userspace separate from the kernel itself. Designing this "properly" requires a lot of subtle decisions to be made, so this rfc proposes to ignore these problems until we are much smarter people later in the project.

James has been working on an extensible forth vm written in Rust, and recently Eliza has been helping to expand the vm, including the ability to have async host-provided intrinsic functions. While limited, this environment is likely suitable for two main things:

  • "gluing together syscalls" to test out kernel capabilities and bring up hardware support
  • Crossing the "embedded system"/"computer" rubicon, allowing for runtime developed programs

The proposal

  • The OS will spawn a single userspace entity at boot time
    • This entity will be a cooperative async task running an interactive forth vm
    • It will be capable of interacting with the userspace/kernel interface (sending/receiving kernel messages)
  • This "task 0" forth environment will be capable of spawning other userspace tasks
    • It can only spawn other forth vms, running their own separate environments
    • When spawning a new task, a snapshot of the current task's dictionary will be made and shared with the child environment
    • All forth vm tasks will be cooperatively scheduled

Details

Changes to forth3 - the current forth vm

  • We'll need to be able to provide a cow-like, reference counted, linked list of previous dictionary fragments
  • We'll need some way of determining "write-like" behaviors that will cause a deepcopy
    • like obtaining the address of a variable
    • todo: is this all we need it for? what about variable addrs in variables?
  • We'll need some way of allocating + deepcopying dictionary fragments

Note, some of these changes are already in-progress at the time of writing of this RFC:

  • #3 - Adding dictionary deep copy
  • #5 - Multitasking with tokio
  • #6 - Adding copy-on-write dictionary fragments

Changes to mnemos

  • We'll need to add some kind of pattern/interface for managing cooperative userspace tasks
    • We will probably NOT enforce preemptive multitasking for now
    • We will probably NOT enforce memory protection/mapping for now
    • userspace tasks yield when they feel like it, maybe with some "VM fuel" cap to force cooperation
  • We'll need some sort of "trampoline" code to bolt the forth3 vm to whatever userspace tasks look like
    • Start the vm, get it running
    • Alloc dictionary fragments as necessary
    • interface with the OS for certain calls like "spawn" via intrinsic functions
    • handle stdin/stdout
  • We'll need to add some kind of allocation service for bigpages
    • Define a service
    • Needs a way for allocating unshared r/w pages
    • Needs a way to exchange unshared r/w pages for shared r/o pages
    • These would normally be used to add memory slabs to the userspace heap allocator
  • We need some sort of "spawn" service
    • ONLY for spawning forth vms now
    • good practice for later, more general, "spawn" interface
  • Call the "spawn" service with our task-id-0, giving it some reasonable stdin/stdout

Other motivations and thoughts

  • Having something useful to play with NOW is more important than designing something perfect
  • We will learn a lot about what "spawn" needs to do with a relatively restricted interface
    • We own the VM, we can force it to cooperate before we have preemption
    • We can impl memory safety checks (probably not fortified, but good for "oops" checking) before we have MMU support
    • All code is interpreted with the same vm engine, no need to worry about relocations, elf loading, or memory mapping yet

RFC - ESP32 Wifi Buddy Interface Design

Goal

To bring wireless networking to MnemOS Beepy devices, an ESP32-C3 SoM will be integrated to serve as a WiFi coprocessor. This device will run MnemOS-based firmware, which we are referring to as "WiFi Buddy". See #121 for details on the rationale for choosing the ESP32-C3 as the WiFi device, and for discussion of the overall design for ESP32-C3 integration.

The goal of this document is to propose an overall design for the hardware and software interfaces between MnemOS on the CPU and MnemOS on the WiFi Buddy.

Background

Hardware

The WiFi Buddy hardware device is a Seeedstudio XIAO or Adafruit QT Py ESP32-C3 development board. These are development boards for the ESP32-C3 which are pinout compatible, so they can be used interchangeably. The ESP32-C3 board will be treated as a system-on-module (SoM) and integrated with the MnemOS Beepy shield using pin headers (or potentially soldered onto the board).

The following pins are broken out by the WiFi Buddy hardware and can be used for communication between the CPU and WiFi Buddy:

  • One SPI interface (MOSI, MISO, and SCK, and a GPIO as chip select)
  • One I²C interface (SDA and SCL)
  • One UART (TX and RX)
  • Four additional GPIO pins (minus one if used as chip sel)

For development, @jamesmunns has designed a development jig PCB which marries a WiFi Buddy with a XIAO RP2040 board. This board will be used to develop WiFi Buddy firmware with Melpomene replacing the D1 as the "CPU".

With the dev board, we'll be able to access:

  • 4-pin SPI
  • 2-pin UART
  • 2 signal GPIOs (or 4, if we don't use the UART pins for UART)

This limits the number of potential IRQ pins in the Melpomene dev configuration, if the UART is used.

Software

Communication between the CPU and the WiFi Buddy will include both a control path and a data path. Data path communication refers to the actual network frames exchanged between the WiFi Buddy and the CPU, while control path communication consists of messages that configure the operation of the WiFi Buddy.

Potential control path communication may include:

  • The CPU asking the WiFi Buddy to scan for WiFi access points
  • The CPU asking the WiFi Buddy to connect to or disconnect from a WiFi access point
  • The ability for the CPU to reset the WiFi Buddy (?)
  • Information about the network interface, such as signal strength
  • Errors and other diagnostic information reported by the WiFi Buddy
  • trace-proto-formatted logs from the WiFi buddy (?)
  • Potentially, messages related to Bluetooth Low Energy operation (out of scope for this document)

It may also be desirable for the CPU to eventually be able to update the firmware on the WiFI buddy. However, for now, both of the target WiFi Buddy devices can be flashed over USB, so the MnemOS CPU being able to update the WiFi Buddy firmware is not a high priority at this time.

Data path communication will be performed over the SPI bus, as it's the highest-bandwidth link available with this hardware. In #121, we have tentatively concluded that the firmware will use a "dumb WiFi Buddy" design, where TCP or any other higher-level network protocols are implemented on the CPU, and the CPU and WiFi Buddy communicate by exchanging MAC-layer 802.3 Ethernet frames.

Design Questions

Based on the background information discussed above, we can begin to enumerate design questions for the WiFi Buddy interface:

  • Is control path communication in-band or out of band? If data path frames are exchanged between the host CPU and the WiFi Buddy over the SPI bus, do we also exchange control messages on the same bus? Or, are control messages sent using another interface, such as I²C?

    Using a single interface for both control and data path communication is conceptually simpler, and has the advantage of a clear and obvious ordering between data and control messages. It also means that we don't necessarily need to implement drivers for all the interfaces available on the ESP32-C3. Avoiding the need to synchronize between messages on two different communication links is a significant advantage.

    Downsides of in-band control messages include that we must introduce some additional form of tagging of data sent on the SPI bus, in order to indicate what bytes are part of a control message and what bytes are actual network data. This may introduce some additional overhead. On the other hand, if we want to eventually include Bluetooth Low Energy as well as WiFi, we will need the ability to indicate what data is an Ethernet frame and what is BLE, anyway, so we'll already be introducing some overhead for tagging data.

  • Do we want the CPU to be able to receive trace-proto from WiFi Buddy? Both the CPU and WiFi Buddy devices will emit debugging information using mnemos-trace-proto. Do we want trace-proto-formatted logs from the WiFi Buddy to be exposed to the CPU, so that it can proxy them to an attached debug host? Or, is using the WiFi Buddy SoM's onboard USB-serial hardware sufficient for debugging purposes?

    • If we do forward trace-proto to the CPU, is this done over the same SPI link as the modem data path? Or do we use the WiFi Buddy's UART pins?

  • What control messages are necessary? Eventually, we'll need to enumerate what control messages will need to be exchanged in order to implement the required functionality.

  • How does the WiFi Buddy signal that data is ready? Since the CPU is the SPI bus controller, it will always be responsible for initiating communication with the WiFi Buddy. If SPI is the only interface between the CPU and WiFi Buddy, then there is no mechanism for the WiFi Buddy to proactively inform the CPU that data is ready, requiring the CPU to busy-poll for WiFi frames. This is unfortunate, because the overall design of MnemOS is event-driven, and we would like to be able to put the CPU in a low-power state when it is idle. Therefore, the WiFi buddy will need to be able to raise an interrupt to the CPU to signal that data is ready and that an operation has completed.

    • The simplest design is a single interrupt line raised by the WiFi Buddy whenever it wants the CPU's attention. Is this sufficient? Or would there be a benefit in having separate IRQ lines for "RX data is ready" and "TX done"/"operation done"?

    • The WiFi buddy hardware has 4 GPIO pins that could potentially be used for interrupts. How many of the CPU's GPIO pins do we want to use for WiFi Buddy IRQ lines?

  • What additional constraints are necessary if we want to support BLE? A design for Bluetooth Low Energy support in the WiFi Buddy interface is out of scope for this document. However, let's not paint ourselves into a corner, if we can help it. Are there additional constraints imposed on our design if we want to leave space for adding BLE in the future?

RFC - Basic Configuration

This RFC proposes a basic configuration system for use in the kernel.

The goal of this system is to allow for a reduction in target-specific code, including potentially removing separate projects for similar targets, such as the lichee-rv, mq-pro, and beepy projects.

This system aims to address current problems, and get us to the next "order of magnitude" of problems (by ignoring many problems we know will be a "later problem").

In Broad Strokes

The config crate

We would add a new config crate that provides helpers for loading configuration at compile time from a json or toml format.

This crate would also provide helpers for "baking" this data at compile time using the databake crate in a build.rs file, and compiling into the target as code.

The config crate would also provide a MnemosConfig type that is generic over two types:

  • A KernelConfig type, which contains all kernel-defined configuration, provided by the kernel crate
  • A PlatformConfig type, which contains all target- or family-specific configuration, expected to be provided by any of the defined platforms that mnemos supports

This crate uses the generics to avoid circular dependencies between the config crate and kernel/platform crates. It is intended that "user facing" applications of the config crate (primarily in platform crates) is done by providing type aliases to "hide" the generics.

Kernel changes

The kernel will provide a single KernelConfig type, which is a struct that contains all service and daemon configuration values. We will add functionality to the kernel to take a provided KernelConfig type, and automatically launch the configured services.

This capability will allow most platform targets to avoid repeated boilerplate kernel.initialize(...) calls at startup.

Platform crate changes

Platform crates will be expected to provide a PlatformConfig type, containing all drivers or daemons that are specific or applicable to this platform (other than the ones defined in the kernel itself). Similar to the kernel, they should also provide a single call that takes a KernelConfig and PlatformConfig, and automatically launches all kernel and platform services.

One platform may have many targets: For example, the Allwinner D1 platform has three targets currently: the lichee-rv dev board, the mq-pro dev board, and the beepy.

With this system in place, each of these targets would no longer have specific binary targets, and would instead share a single binary target with different configuration files, loaded at compile time.

This configuration file could be set via an environment variable such as MNEMOS_CONFIG=/path/to/beepy.toml, which would be loaded by a build.rs script, and set by just.

In this way, customized targets can be made by creating a new configuration file, rather than an entire cargo project.

Compile Time vs Runtime Configuration

At the moment, most configuration will be done at compile time, with configuration values being "baked into" the binary.

However, we should ensure that all configuration values also implement serde traits, so that platforms that CAN load configuration at runtime (from an SD Card or other external sources) are able to.

Platforms that can load configuration at runtime may need to provide a "two stage" initialization process: using enough of the "baked in" configuration to initialize the SD Card or other external source, then continuing with the initialization process after merging or replacing the baked in data with the loaded data.

Inspiration

This approach is largely inspired by ICU4X's Data Provider model, which aims to "bake in" some common assets, but also allow for loading additional assets upon request.

Other potential inspirations

  • https://crates.io/crates/config
  • https://github.com/rust-osdev/bootloader#usage

"Tomorrows Problems"

This initial approach avoids many complexities by ignoring them completely. Some of these complexities are described below:

Compile time feature enablement

At some point, we may have many drivers, some of which that are not relevant or possible to support on some platforms. Since this system will support runtime configuration, the compiler will still need to compile them into the kernel image, even though there is no chance of them being used.

This approach does not address compile time gating of features to prevent certain capabilities from being compiled in at all.

While this may lead to some "code bloat", we are not at a stage where this is an immediate concern. Once we are, our configuration approach may need to be revised.

Extensibility

Similarly as new features are added that COULD be supported on all platforms, we will need to add them manually to the configuration items of each crate.

For example: if we have SPI drivers in many platforms, and write a bunch of drivers for external components connected over SPI, we will need to add configuration for each driver to all platforms.

This might be mitigated by grouping these into a common "all spi drivers" configuration type that is included "by composition" into all platform crates, but at some point this will cause friction and/or maintenance burden.

We may want to revisit how to support that in the future.

Layering of config

Eventually we will probably want some kind of way to "layer" configs, for example providing a base level of defaults, and adding partial overlays when adding components.

"demand based" configuration

This configuration approach makes no attempt at addressing validation of configuration, particularly looking at the dependencies of each service to determine if the configuration is suitable, nor detecting whether the configured hardware actually exists.

For example, a platform may configure the sermux service, but forget to provide a serial port for the sermux service to use, or a platform might configure a wifi adapter that isn't present on the target board.

At the moment, this RFC just says "don't do that", or "catch that in testing", which will not scale past a certain complexity level

Generating or building a platform config

This RFC makes no suggestion on how platforms should generate a comprehensive config file. We will probably want this eventually.