[Rust+WASM] Downloading files in runtime instead of include_bytes!()

While developing a raycasting engine, I started to feel limited by the fact that most (not all) projects compiling to WASM were using include_bytes!() macro.

This macro is OK for most things, but it makes the final binary bigger, does not allow you to use a CDN, nor lets the browser do it’s cache magic. The cycle is simple: you change something in your code, compile to wasm, all binaries get embedded inside it, and the client downloads the whole package again.

I didn’t like this. I wanted images and external files to be stored in a CDN. I wanted people to use their local browser cache when reloading the page. All in all, I just wanted something else.

So here came my first headache: Finding projects that didn’t use include_bytes!() AND also we fetching files from inside the Rust code was hard. I kept looking at the source code of many projects and many used JavaScript but I didn’t want to write my code in JS. So I decided that if no one else wanted to do it that way, I had to do it myself, just as I did when I ported minifb to the web.

Lets bring the good news first: I succeeded. The bad news? I had to compromise and use a little JavaScript.

The solutions to downloading files I found often included async code, but I did not want my game loop to be asynchronous and was not willing to change that, and I almost give but, but life… life finds a way.

After a lot of struggle, I came up with this algorithm:

1. start the WASM code normally
2. fire a web worker and set up a response callback
3. send the worker a string with the file you need
4. the worker fetches the file
5. the worker posts a message with the file as u8 array
6. the Rust code handles the response from the callback
7. the Rust code terminates the worker after downloading all required files

Lets go straight to the code, starting with the easier part. The JavaScript worker:

onmessage = function(e) {
console.log(‘Worker: Message received from main script. downloading file…’);
var file_path = e.data;
fetch(file_path)
.then(resp => resp.blob())
.then(workerResult => {
console.log(‘Worker: Working on file ‘ + file_path);
filename_ascii = new Blob([file_path + ‘\0’], {type: ‘text/plain’});
new_blob = new Blob([filename_ascii, workerResult], {type: ‘application/octet-stream’});
return new_blob;
})
.then(complete_blob => complete_blob.arrayBuffer())
.then(array_buffer => {
postMessage(array_buffer);
})
.catch((error) => console.error(error));
}

The worker is in charge of two things:

1. Downloading the file
2. Appending the file name to the reply (more on this below)

The communication between Rust and JavaScript is a little clumsy. You can send and receive messages between the worker and the main thread, but all in all you send a JsValue which limits what kind of reply you can expect. A boolean, a string, an array buffer, you get the idea. Check the documentation if you don’t believe me.

In this case JavaScript will download an image into an u8 array, so we want to keep it as intact as possible. But if JsValue cannot be multi valued, how can we receive extra information from the Worker? If we don’t spin up exactly one worker per file, that means we won’t get a different callback per file, and we need to know which file has just been downloaded! OK, we can’t post a message with an array composed of different variables, and I don’t want to send several messages, but then what happens in the user hit F5 instead of closing the tab? Ha! there will be TWO workers in the background. You didn’t know that detail, did you? Start a worker, hit F5, a couple of times, and send a message to your worker handle… Surprise, you have many workers running! Close the tab and re-open, and now it works as expected. Yeah, that sucks, big time, but that doesn’t mean I can’t cheat the system to do what I want anyway.

My files are all something like “folder/image123.ff”. On my first try I did what any lazy human would. I converted the binary blob inside the worker and posted a String message that Rust would pick up. Since this was a simple base64 string I appended the file name with a pipe as separator and tested my solution. To my surprise, it worked on the first try.

Finally, y second attempt derived in the code you see above. I create two Blobs in JavaScript. One contains the null terminated ASCII filename (NOT utf-16. very important) and the second contains the file we need. I concatenate both into one single Blob, and post the result. Presto! From Rust I just have to take the first bytes until I find the null character and convert them to a string. The rest of the message is the image I was downloading, and I didn’t have to go back and forth with base64.

Now that you understand the motivation behind the JavaScript code, let’s look at the Rust side.

From Rust you need to start the worker, and have a way to store the results from the callback once you get the response. So it’d be something like this

let worker_handle = Rc::new(RefCell::new(Worker::new(“./js/file_download_worker.js”).unwrap()));

Then, on the worker_handle add a callback for the response using set_onmessage(). This code is tedious but repetitive. Check the links below to copy an example.

The important part is that your callback must have a way to communicate with the main thread. That’s an Rc<Refcell<>> somewhere in your code that will store the callback results.

For my project the type was a Rc<Refcell<HashMap<String, Texture>>>.

And be careful if you do the same. Use a String, not a &str, otherwise the compiler can’t determine the lifetime of the &str, which needs to be static, and it fails to compile.

Let’s go back to the code. If you follow the links below you will see that the callbacks for the worker all look the same. Scroll to the part where it declares the Closure for the callback.

The `wasm-bindgen` Guide

This is where the final magic happens:

Leave a reference to a structure inside the callback, and the callback itself can save your files into something like the HashMap I’m using. Then all you have to do it refer to the HashMap from the main thread, and you will finally have access to the downloaded files without having to use async code in the Rust side.

I uploaded my project so you can see the whole solution working, but hopefully this story was enough for you to come up with a better way to do it, or at least the knowledge to do it yourself.

The project is hosted here: https://github.com/dc740/permadi-raycasting-in-rust

And you probably want to start looking at the web part

https://github.com/dc740/permadi-raycasting-in-rust/blob/main/main_app/src/web_setup/mod.rs

Sources:

GitHub - sgasse/wasm_worker_interaction: Examples of using Rust WASM with web workers

The `wasm-bindgen` Guide