Browser Shell

tl;dr a Linux VM mounting a JS filesystem with a Service Worker web server.

Overview

The following is an experiment in impractical computing. It needs something more than a few tweets or a blog post to explain. So I've made you a demo of a number of interconnected projects I've been working on, all underpinned by the central idea of a browser filesystem. Everything you're about to see and do is going to happen in your browser, in JavaScript, and thanks to lots of features of the modern web platform. And, since a demo is supposed to get straight to the point, I'll pause my explanation here and give you a quick tour. Below that I'll explain how everything works.

If you're on mobile, or a slower machine, I put a short screencast on YouTube that might work better for you.

Terminal to A Linux VM Mounting Shared Filesystem

The first part is a functional terminal to a custom embedded Linux running in the browser. As it boots, it's going to mount a JS filesystem in /mnt—this same filesystem is also accessible outside the VM, both in the window (open your console and you can run node's fs methods) and in a Service Worker, where I'm running a web server (see below).

As far as Linux is concerned, it's just a normal filesystem, and you can do the usual things, for example:

Try it yourself. Once it's booted, click the terminal and try working with /mnt. Clicking away, or hitting the pause button will suspend the VM, hitting play will resume it.

play_circle_outline pause_circle_outline

The filesystem and the VM will survive a reboot (reload of the page) without losing any data. The filesystem lives in indexeddb, and the VM's initial CPU and RAM state are stored in Cache Storage.

Web Server to Shared Filesystem

The second part of this demo is a static web server, hosting the shared filesystem from a Service Worker. Any file or directory you create in Linux under /mnt can be accessed via the web server at the /fs route. For example, if you want to browse to a file called /hello-world.html you need to use the URL /fs/hello-world.html.

To make this a bit more obvious, I've created a simple iframe-based web browser that's serving the /fs/ route. You can try creating/modifying things in Linux, and refreshing the iframe browser to see the changes, and view the files. I've also created a few files to get you started.

arrow_back arrow_forward refresh home

Because a Service Worker doesn't need to be run in a page, you could also navigate to these URLs manually (i.e., you don't need this page, or this iframe). Again, there's no server. It's just JavaScript handling network requests with data from indexeddb. It looks like Apache because that's my how I grew up on the web, and it just felt right to use that style.

Import Files to the Browser Filesystem

If you want to try adding arbitrary files yourself, you can also drag-and-drop them here. NOTE: nothing is uploaded to a server, everything lives in indexeddb. Whatever you add here will show up in /mnt in the VM, and in the / directory in the web server.

folder

Drop files here to be imported...

Discussion

Now that you've got the general idea, let me spend some time talking about how it works, and what I had to build for each of the pieces.

Origins

The entire thing is born out of an idea I'm fascinated with, namely, putting a filesystem in the browser. I've been chasing this idea since 2014 in one way or another. I got started on it when I was porting Adobe's Brackets desktop code editor to the browser for Mozilla's Thimble project. To do that, I needed to trick the app into thinking there was a full filesystem in the browser. A friend told me about a project he'd started to create a JS filesystem running on top of indexeddb called Filer. I started hacking on it with him, and never really stopped. Filer works really well, and we used it in Thimble to handle millions of files in browsers all over the world for many years.

Mozilla Thimble Code Editor

Since then I've continued to improve and extend Filer, building out the necessary node.js fs compatibility, which has grown a lot. While doing this work, one of the things I always wanted was a full command-line shell that I could use to create, view, and modify the files in the filesystem. Having an API you can code against is nice, but as a regular user of filesystems, I wanted a shell. I toyed with the idea of creating one from scratch, but decided it might be faster to just figure out how to get Linux running in the browser, and have it mount the Filer filesystem somehow.

Hardware

As you've just seen, this isn't such a crazy idea (or, it's crazy, but doable!). To make it work, I first needed virtualized hardware running in the browser, something that is relatively easy to come by in 2019. I'm using a version of the amazing v86 project that I've modified for the task. My v86 virtualized hardware is much like any number of old Linux boxes I've used since the mid-90s:

Kernel I/O Support

Next I needed a way to mount my filesystem within Linux. I asked Fabian, the creator of v86, for some guidance, and he suggested that I could probably modify some code that was created as part of the jor1k OpenRISC 1000 emulator, and also included in v86. In both jor1k and v86, this filesystem allowed loading data over XMLHttpRequests from a remote server. In my case, I'd need to rework it to wrap my Filer filesystem API.

The idea is one that you may have encountered in other virtual machines, where a host OS shares a filesystem with a guest VM. Often this is done using virtio, where hardware layers are abstracted. Linux supports a special v9fs filesystem, which allows using a virtio transport to mount a Plan 9 9p remote filesystem.

Plan 9 Space Glenda I think the fact that this project involved me working on something related to Plan 9 is a big part of why I put up with it for so long. I've always loved Plan 9, and its "simple file protocol, 9p", and here was a chance to implement it!

Getting this to work took me quite a while. The 9p docs are terse, but clear if you spend enough time with them. There are lots of 9p implementations in various languages that I read as well. I learned a lot by studying chaos/diod, arm-js, jor1k, and v86.

By far the best thing I found on the web related to implementing the 9p protocol was a series of blog posts from 2015 about writing a 9p server in Go. The protocol parsing post was particularly useful to me. Debugging a failing virtio/9p implementation in JS, as it silently fails deep inside a Linux kernel running in an emulated VM in the browser, is not the most fun I've ever had. But eventually, I got it. If you're at all interested in what the protocol steps are like, reload this page with ?debug on the URL, and open your console while you work with the /mnt directory. I have no doubt that if you press hard enough on the Linux terminal above, you'll likely uncover bugs, which you can file if you like.

Operating System

My next hurdle was to create an operating system that would have all the bits I needed, and nothing I didn't. It also needed to be small, load my virtio kernel module, and include various startup scripts for the serial port setup, filesystem mount, etc.

The v86 project has a bunch of pre-made OSes you can boot. A number of these had aspects I needed, but none of them was exactly what I wanted. I needed to make my own Linux distro for running in the browser.

I wasted a lot of time on this. While I'm happy to spend hours nursing a failing program and deal with ridiculous levels of complexity in my tooling, my feeling about an OS is that it should work invisibly. I don't enjoy configuring or debugging them.

After a bunch of research, I ended up deciding to use Buildroot to create an "embedded" Linux distro. Most people use Buildroot to create a custom Linux kernel and OS config for small, on-board computers (e.g., a BeagleBoard) that get loaded via SD card. In my case, I needed my OS to run in a similarly constrained environment (i.e., v86 and the browser). The parallels were close enough, so I dove in.

Buildroot Menu Config

The Buildroot docs are really well done, but long. They also have great training material, and there are some excellent talks on YouTube, like this one. I found it took me a long time to fill in the gaps in my knowledge that the docs/talks seemed to assume, since many people coming to Buildroot probably already have more extensive Linux and Kernel config knowledge than I did.

Eventually I created my own Buildroot v86 board config and a Docker container to automate the build process. I spent a bunch of time stripping down the Kernel config to get rid of unnecessary drivers and bloat, but I'm sure there's more space saving that could be done. The end result is an ISO file with a custom Linux kernel, Plan 9 filesystem sharing, a root filesystem and shell using BusyBox, and startup scripts to get the serial terminal working and /mnt mounted to the browser filesystem. The browser downloads it and boots it as a CD-ROM.

Web Server

Having the ability to host a filesystem in indexeddb, and work with it via a shell, the next thing I needed was a way to serve and execute these files in the browser. I spent a long time trying to figure out a way to leverage networking, ZMODEM, or some other protocol out of the Linux VM to the browser, so I could interact with what was running within the VM. In the end, this seemed like a dead-end, and I decided it made more sense to simply serve these files in response to network requests. The browser already knows how to parse and render web technologies, so there's no point trying to do better.

I'd actually built a few versions of this for Mozilla's Thimble in the past. My first iteration used Blob URLs and regex to rewrite HTML, CSS, and JS files, swapping relative paths to files in the filesystem for functioning Blob URLs. Once Service Workers were more widely supported in browsers, I rewrote things to use proper network requests for URLs, which got content from the database instead of the network.

For this, my third iteration, I used Workbox to simplify my Service Worker code, and then created some simple routing to watch for URLs using /fs/ and then satisfy them with content from the filesystem. The fact that indexeddb is accessible in both window and worker contexts make sharing data this way really easy.

The only extra thing I did this time was to spend some time reducing the size of the code necessary to maintain the list of MIME types I need in my server. Previously I've used mime-db and mime-types, which are comprehensive, but massive to include in a browser. In the end I wrote some code to build my own, browser-mime.

My solution was to use Puppeteer to process every MIME type in mime-db, one by one, and create empty files of each type to see if the browser would parse/render them, or instead offer to download them. I found that I was able to use the Navigation Timing API to determine whether a page load occurred (i.e., the browser knew how to process this type), or if the stream was simply downloaded to the user's OS (i.e., type not supported by the browser). This info allowed me to come up with a much reduced MIME db that I can bundle with my Service Worker, and which only includes things the browser is likely to parse/render, letting everything else get downloaded as an application/octet-stream.

Conclusion

In the end, this is the solution to a problem you don't have. I get that. And yet, it was so much fun to get working, I had to share it with you. I love making browsers do new tricks. I wanted to put this page together to show what's possible by connecting some of the different things I've been thinking about and working on, specifically:

The techniques I've outlined above are only going to become more realistic as we move forward. WebAssembly is pushing the limits of what's possible in the browser yet again. Running complex code like VMs in the browser is going to get faster. For example, a few weeks ago jor1k got a WASM CPU rewrite.

Being able to use the same page to create, edit, serve, and execute code seems like an idea that could have many interesting uses. It wouldn't be hard to hook up a few more "editor" apps to this site, and have them all share the same filesystem. You could even run them out of the filesystem if that made sense. Combine this with any of the various web p2p protocols for sharing data between browsers, and you could build some really cool and useful applications.

If you're interested in the potential of a browser filesystem, you should get in touch. I'd love to keep doing more with Filer, I think it has so much potential. Right now it's just a side passion, but I'd love for it to be more.

Thanks for checking this out. If you have thoughts you can let me know on GitHub.

David Humphrey (@humphd), April 2019.