22-ramdisk.org

Creating a RAMdisk

One of the most important functions of an operating system is keeping track of users’ important (and not so important) data, and providing ways to access and modify it. Usually this is done by organising data into files in a hierarchical directory structure. We’ve made a start on a Virtual File System (VFS) with OPEN, READ and WRITE messages, but not used them for actual files yet. To make sure that the system we end up with isn’t tied to a specific filesystem we’re first going to get something working with data stored in memory. Later we’ll need to worry about block storage devices and all that.

Single directory

To start with we can simplify things by only having a single directory.

Basic shell

fn exec_path(path: &str) -> Result<(), SyscallError> {
    let mut file = File::open(&path)?;

    let mut bin: Vec<u8> = Vec::new();
    file.read_to_end(&mut bin)?;

    // Create a communication handle for the input
    let (exe_input, exe_input2) = syscalls::new_rendezvous()?;

    syscalls::exec(
        &bin,
        0, // Permission flags
        exe_input2,
        syscalls::STDOUT.clone());

    loop {
        // Wait for keyboard input
        match syscalls::receive(&syscalls::STDIN) {
            Ok(syscalls::Message::Short(
                message::CHAR, ch, _)) => {
                // Received a character
                syscalls::send(&exe_input,
                               syscalls::Message::Short(
                                   message::CHAR, ch, 0));
            },
            _ => {
                // Ignore
            }
        }
    }

    Ok(())
}

Now we can run a program (well one program, gopher) but when it exits the shell will be suspended waiting to either send a message to the program or to receive a keyboard input.

When receiving messages we could have a timeout, and when sending messages we need some way to detect whether the process is still running. One way would be to detect if the communication handle had been dropped: If we try to communicate with a Rendezvous with only one handle, or drop a handle with a waiting thread then we could return an error message.

Discovering files and other things

At the moment the ramdisk can store and retrieve files, but there’s no way to find out what files are present. To provide a basic ls command we’ll need to at least be able to list files (and directories). I’d like to make the system discoverable, by enabling any handle can be queried to find out what it is, what messages it can respond to, what inputs it expects etc.

Adding a QUERY message type, which will be responded to with a JSON message (in euralios_std/src/message.rs):

pub const QUERY: u64 = 7;
pub const JSON: u64 = 8;

JSON may not be the best choice but it’s a) human readable for debugging and manipulation, and b) widely supported, in particular by the Rust serde library in no-std environments. Other serialisation formats are available.

In ramdisk/src/main.rs in the handle_directory() function we can add a match for QUERY tagged messages, assemble a JSON string (by hand, for now), returning by copying the string into a memory chunk and sending it back in a message tagged JSON.

In the shell program, entering “ls” opens /ramdisk and queries it:

let mut line_buffer = String::new();
  loop {
      // prompt
      print!("$ ");

      // Read a line of input
      stdin.read_line(&mut line_buffer);
      let input = line_buffer.trim();

      if input == "ls" {
          if let Ok(file) = File::open("/ramdisk") {
              file.query()
          }
      }
      ...

The query method is a EuraliOS-specific extension to fs::File. It sends a QUERY message and for now just prints whatever comes back. In euralios_std/src/fs.rs:

impl File {
  ...
  pub fn query(&self)  {
      match rcall(&self.0,
                  message::QUERY,
                  0.into(), 0.into(), None) {
          Ok((message::JSON,
              MessageData::Value(length),
              MessageData::MemoryHandle(handle))) => {

              let u8_slice = handle.as_slice::<u8>(length as usize);
              if let Ok(s) = str::from_utf8(u8_slice) {
                  println!("[query]: {}", s);
              }
          },
          message => {
              println!("[query] received {:?}", message);
          }
      }
  }
}

The result is in figure fig-ls

Path

Adding some of the path manipulation API from std into a new file euralios_std/src/path.rs. Path is implemented here similar to the standard library, but using str rather than OsString:

pub struct Path {
    inner: str,
}

impl Path {
    /// Directly wraps a string slice as a Path slice.
    ///
    /// This is a cost-free conversion.
    pub fn new<S: AsRef<str> + ?Sized>(s: &S) -> &Path {
        unsafe { &*(s.as_ref() as *const str as *const Path) }
    }

    /// Yields the underlying str slice.
    pub fn as_os_str(&self) -> &str {
        &self.inner
    }
}

And we can copy the implementation of AsRef<Path> from std::path

impl AsRef<Path> for str {
    #[inline]
    fn as_ref(&self) -> &Path {
        Path::new(self)
    }
}

impl AsRef<Path> for String {
    #[inline]
    fn as_ref(&self) -> &Path {
        Path::new(self)
    }
}

so we can change the File::create and open functions to take a Path, str or String as input:

pub fn create<P: AsRef<Path>>(path: P) -> Result<File, SyscallError> {
    let handle = syscalls::open(path.as_ref().as_os_str())?;
    Ok(File(handle))
}

Parsing JSON with serde_json

The serde_json crate can be used without the Rust standard library:

serde_json = { version = "1.0", default-features = false, features = ["alloc"] }

let u8_slice = handle.as_slice::<u8>(length as usize);
if let Ok(s) = str::from_utf8(u8_slice) {
    match serde_json::from_str::<Value>(s)
        Ok(v) => Ok(FileQuery(v)),
    ...

#[derive(Debug)]
pub struct DirEntry {
    name: String
}

impl DirEntry {
    pub fn file_name(&self) -> &str {
        &self.name
    }
}

#[derive(Debug)]
pub struct ReadDir {
    entries: Vec<DirEntry>
}

pub fn read_dir<P: AsRef<Path>>(
    path: P
) -> Result<ReadDir, SyscallError> {
    let f = File::open(path)?;
    let query = f.query()?;

    let entries = match query.0["files"].as_array() {
        Some(vec) => {
            // Transform into a Vec of DirEntry objects
            vec.iter().map(|obj| DirEntry{
                name: String::from(obj["name"].as_str().unwrap_or("_bad_"))
            }).collect()
        }
        _ => Vec::new()
    };

    Ok(ReadDir{
        entries
    })
}

Turning ReadDir into an iterator is just a matter of defining the Item type, and the next function:

impl Iterator for ReadDir {
    type Item = Result<DirEntry, SyscallError>;

    fn next(&mut self) -> Option<Self::Item> {
        if let Some(entry) = self.entries.pop() {
            return Some(Ok(entry));
        }
        None
    }
}

In the shell we can now write:

if input == "ls" {
    if let Ok(rd) = fs::read_dir("/ramdisk") {
        for entry in rd {
            println!("{}", entry.unwrap().file_name());
        }
    }

This now prints the files in the ramdisk, shown in figure fig-listing.

Testing

I’ve basically let things go as far as testing goes, and not followed Phil’s good example. Now that the basics of a system are in place and we’re starting to add utilities to the system standard library, it’s time to make amends and add some tests. A difference is now the tests are going to be in an executable which is run from within EuraliOS.

One way to test the euralios_std library would be a separate user program, but it would be nice to have unit tests, so that the tests could go in the same file as the code. Unfortunately there doesn’t seem to be a way to add build scripts to cargo test, and a build script is needed to change the address of the code and data in the ELF file (see section 2). The easiest solution was to move the kernel to a different memory range, and allow all user programs to occupy the range between 0x20000 (2Mb) and 0x5000_0000 (1.25Gb). Then the only binary which needs a build script is the kernel.

Changing this is a matter of modifying the range of allowed user addresses USER_CODE_START and USER_CODE_END in process.rs. There’s also a check in handle_syscall() (in syscalls.rs) of whether the code being returned to is in user or kernel space. That needed extending so that we check if the instruction pointer is in the user address range.

#![no_std]
#![no_main]
#![feature(custom_test_frameworks)]
#![test_runner(crate::test_runner)]
#![reexport_test_harness_main = "test_main"]

use euralios_std::{print, println};

#[no_mangle]
fn main() {
    println!("EuraliOS system test");
    println!("====================");
    #[cfg(test)]
    test_main();
}

#[cfg(test)]
mod tests {
    #[test_case]
    fn empty_test() {
    }
}

// Custom test framework

pub trait Testable {
    fn run(&self) -> ();
}

// Implement Testable trait for all types with Fn() trait
impl<T> Testable for T
where
    T: Fn(),
{
    fn run(&self) {
        print!("{}...\t", core::any::type_name::<T>());
        self();
        println!("[ok]");
    }
}

pub fn test_runner(tests: &[&dyn Testable]) {
    println!("Running {} tests", tests.len());
    for test in tests {
        test.run();
    }
}

The makefile rule to build system_test is a little complicated:

user/system_test: FORCE
        cargo test --bin system_test --no-run
        @cp $(shell find target/x86_64-euralios/debug/deps/ -maxdepth 1 -name "system_test-*" -executable -print | head -n 1) $@
        @strip $@  # Can't use debugging symbols anyway

That recipe builds the executable with the test feature, using the --no-run= option so that cargo doesn’t try to run it. Unfortunately the output binary is not named consistently so we use find to locate it. We then strip the binary because if we don’t then the binary is 1.9Mb; after stripping the binary is just 119Kb.

Paths

We can see in figure fig-testing that the shell reads the executable from the path /ramdisksystem_test, the operating system matches the first part to /ramdisk and then the ramdisk opens system_test. That probably shouldn’t work this way: the path /ramdisksystem_test should be different from /ramdisk/system_test.

When VFS::open matches mount paths against the requested path it just checks that the requested path starts with the mount path: /ramdisksystem_test starts with /ramdisk so it matches. In addition we need to check that the requested path is either the same as the mount path (so opening /ramdisk matches), or the requested path contains a ‘/’ character after the match (so /ramdisk/system_test matches).

Having fixed the kernel VFS::open function we of course break the shell because /ramdisksystem_test cannot be opened. To fix that we need to extend the Path API and add a path::PathBuf type to the EuraliOS standard library. That will handle joining and splitting paths, making sure the right number of ‘/’ are in the right place.

Opening files

So far we’ve only had one kind of OPEN message, for creating files and for reading them. There are however (at least) four cases we need to consider:

• OPEN_READONLY which fails if the file (or directory) doesn’t exist, and only allows data to be read.
• OPEN_READWRITE which fails if the file or directory doesn’t exist, but allows write operations as well as read.
• OPEN_CREATE which creates a file if it doesn’t exist, and allows read and write operations. If the file already existed then it is the same as OPEN_READWRITE.
• OPEN_OVERWRITE which creates the file if it doesn’t exist, and replaces (truncates) it if it already exists.

As an interface to these different options we can implement stf::fs::OpenOptions

Deleting files

This section is becoming quite long but we’ve made some good progress on the user shell and file system interface. One of the many limitations of the interface is that we can’t yet handle special characters (e.g arrow keys, F-keys) or modifiers: shift and CapsLock work, but Ctrl and Alt don’t. In the next section we’ll move the keyboard handler out of the kernel and add more functionality to it. This will mean finally providing a way for user-space drivers to handle interrupts.