PTR Changelog 2017-09-25: WebAssembly support

artch

This post describes changes on the Public Test Realm. The estimated launch date is December 14.

Enabled experimental WebAssembly support. WebAssembly is a binary compiled code format that allows to run C/C++ or Rust code (as well as other supported languages in the future) directly with native performance. Please refer to the WebAssembly documentation for more info.

This is an experimental feature. We need your feedback in order to decide if it worth it. Please share your experience of what you were able to build using WebAssembly.

Example

Here is a short example of how to compile C/C++ code using Emscripten and upload the binary file to Screeps.

Build `.wasm` file

You can skip this step if you use an already compiled .wasm file from the web.

Install Emsripten SDK using these instructions.

Write your native C function and save it as addTwo.c file:

int addTwo(int a, int b) {
    return a + b;
}

Compile it to addTwo.wasm using this command:

emcc -s WASM=1 -s SIDE_MODULE=1 -O3 addTwo.c -o addTwo.wasm

Upload binary module

Using the in-game IDE

Add a new module addTwo with binary type using this switch:

Upload your addTwo.wasm file as binary module contents, so that it looks as follows:

Click the ️ button to commit your modules.

Using grunt-screeps

Switch your grunt-screeps to the binary-upload branch on GitHub:

npm install screeps/grunt-screeps#binary-upload

Now all files with extensions other than .js will be uploaded as binary. Copy your addTwo.wasm file to your upload folder and modify the Gruntfile.js so that it includes .wasm files as well:

module.exports = function(grunt) {

    grunt.loadNpmTasks('grunt-screeps');

    grunt.initConfig({
        screeps: {
            options: {
                email: 'YOUR_EMAIL',
                password: 'YOUR_PASSWORD',
                branch: 'default',
                ptr: true
            },
            dist: {
                files: [
                    {
                        expand: true,
                        cwd: 'dist/',
                        src: ['**/*.{js,wasm}'],  // <----
                        flatten: true
                    }
                ]
            }
        }
    });
}

Run grunt screeps to commit your modules.

Run your native module in Screeps

If you uploaded your binary module correctly, you should see the following in your in-game IDE:

Now you can use the following code to run your imported binary code in main using WebAssembly API:

// This will return an ArrayBuffer with `addTwo.wasm` binary contents
const bytecode = require('addTwo');

const wasmModule = new WebAssembly.Module(bytecode);

const imports = {};

// Some predefined environment for Emscripten
imports.env = {
    memoryBase: 0,
    tableBase: 0,
    memory: new WebAssembly.Memory({ initial: 256 }),
    table: new WebAssembly.Table({ initial: 0, element: 'anyfunc' })    
};

const wasmInstance = new WebAssembly.Instance(wasmModule, imports);

console.log(wasmInstance.exports._addTwo(2,3));

Mototroller

This is incredibly awesome feature! Some questions:

What is it actually aimed at? Is performance of native modules the main reason? Or, is it a next step of virtualization reworking?
How will CPU consuming be counted while using native modules? And what about exception and hard-interruption points in case of CPU-, stack- or heap- overflows?
Are any NATIVE <=> GAME APIs planned? Alternatively, will users be able to use raw WA <=> JS APIs (memory mappings, pointers sharing etc.)?
As I remember, promises (and other async stuff) aren't handled correctly for now by game engine: is it guaranteed that native module loading will be consistent (no resets until new file will be uploaded, for example)?

Being a cpp developer and being faced some time ago with WA projects, I assume that WA introducing is really, REALLY hard and breaking change for run-time environment (new APIs, new bugs, new non-portabilities, suddenly new vulnerabilities -- hello native LLVM!), I just wanna wish good luck and make sure such a useful and powerful feature will be implemented with such a close attention as previous roadmap implemented features.

ags131

I played with WASM a bit when they did the node 8 test, from my experience, we will be able to use full JS <--> WASM APIs, binary support actually makes that easier than before, I had to encode the binary data in base64 then decode it into an ArrayBuffer myself. I'm sure someone in the community will write bindings between the two API sets, I for one look forward to using WASM for all my binary manipulation stuff, such as packing paths and distance transforms.

artch

@mototroller

What is it actually aimed at? Is performance of native modules the main reason? Or, is it a next step of virtualization reworking?

It's an option to write critical code natively to increase your script's performance.

How will CPU consuming be counted while using native modules? And what about exception and hard-interruption points in case of CPU-, stack- or heap- overflows?

Everything remains the same, all WebAssembly manipulations are done within your regular VM environment.

Are any NATIVE <=> GAME APIs planned?

Not officially, but it's an interesting vacant area for community projects.

Alternatively, will users be able to use raw WA <=> JS APIs (memory mappings, pointers sharing etc.)?

All APIs in the WebAssembly namespace are exposed to the user space in full.

As I remember, promises (and other async stuff) aren't handled correctly for now by game engine: is it guaranteed that native module loading will be consistent (no resets until new file will be uploaded, for example)?

Using async API is not mandatory. The example above is already changed to the synchronous instantiation.

Being a cpp developer and being faced some time ago with WA projects, I assume that WA introducing is really, REALLY hard and breaking change for run-time environment (new APIs, new bugs, new non-portabilities, suddenly new vulnerabilities -- hello native LLVM!), I just wanna wish good luck and make sure such a useful and powerful feature will be implemented with such a close attention as previous roadmap implemented features.

Actually, WebAssembly object is exposed by default on Node 8.x, so I don't think it is really that breaking.

Mototroller

Please share your experience of what you were able to build using WebAssembly.

Sorry, missed the request. Some words about WA experience:

Being inspired by Screeps, I was looking for a scalable, high performance and virtualization-friendly solution for Screeps-like AI environment. The candidates were Lua (provides high performance virtual machine and a simple script language) and WA (extremely high performance + built-in virtualization by design). Well, Python should be mentioned here: it's very difficult to build a bulletproof VM for it, and its performance without dirty hacks is really bad. I realized that Lua is not popular enough to be a main language for AI describing and developing, so WA was chosen.

There were several underwater rocks while implementing scalable architecture of shared AI sandboxes:

It was difficult to find a compromise between API flexibility, performance and environment restrictions ("Game map as a shrared mmaped raw memory? Awesome, but memory protection? Map updates? Synchronization?")
For now it's still not very comfortable to perform IPC with WA (message queue layer was written as dirty mix of WA API, JS and native C++ -- this layer and its endpoints seems for me overcomplicated, but all the workarounds are using pretty same API calls, and I've found the performance of portable robust solution is not very nice due to tons of unnecessary memory copying). Memory mapping and sharing (mmap or shmem etc.) are sometimes broken due to unexpected Node or WA restrictions.
Due to environment restrictions, it's very difficult to debug native modules or write correct test suits. Moreover, I'm not sure about C++11 exception support by WA nowadays. So, it's extremely easy to hang or drop native module... or even write a malicious code due to leaky Node + WA API and permissions vulnerabilities -- it still seems not safe enough.

Because of points mentioned above, I've found WA modules are very useful for "dense" computational (some difficult algorithmic code with few inputs and outputs), but not flexible enough to be a full replacement of JS in most egine's cases despite on performance benefits. For now my project is frozen, but I've got tons of experience, and gonna return to it later with new vision and experience (and may be some followers:).

@artch: Thank you for reply!

It's an option to write critical code natively to increase your script's performance.

Nice idea, imho. Endpoints and synchronization problems are still difficult (see above), but it can really increase performance.

All APIs in the WebAssembly namespace are exposed to the user space in full.

It's a disputable point due to safety problems, but still nice for beginning. Offtop: Screeps community is really very nice here to not to abuse or even looking for vulnerabilities, and report them:)

Actually, WebAssembly object is exposed by default on Node 8.x, so I don't think it is really that breaking.

Oh, it's nice, but I'm mostly about safety underwater rocks (and, as you pointed out, about rewriting critical code).

ags131

I've been testing with wasm and so it seems to perform very well, I'm not sure of the memory usage, but overall CPU usage tends to be less than similar JS code, even with the calls back into JS space. My current testing code is on GitHub, it uses a different approach than the guide above and handles much more of the marshaling of types and helps deal with pointers and such. With this, I'm easily able to spawn and random move over 300 creeps with almost no overhead. If this ends up being available, I wouldn't want to switch all my code over to it, I'm not a C++ guy, but I would implement the more computationally intensive tasks in C++ to reduce CPU. Stuff like distance transform, skeletonization, layout planning, path compression, this can all be done in C++ and reduce overall CPU required.

W4rl0ck

I'll give it a try now.

I wonder if webassambly could not only be used in the user side of code ... but maybe also make some of the server code faster ?

UnFleshedOne

Nice, this is definitely something I would use if it ever makes it in. Finally my work experience apply to something that matters!

WebAssembly says it does or will eventually support pthreads, OpenGL and other system libraries. Will those be available in screeps and how would pthreads for example impact CPU usage? (haven't had a chance to actually try it on PTR yet)

Would .wasm file size would count against max script size I assume?

artch

@unfleshedone

WebAssembly says it does or will eventually support pthreads, OpenGL and other system libraries. Will those be available in screeps and how would pthreads for example impact CPU usage? (haven't had a chance to actually try it on PTR yet)

All WebAssemply features should backed by Node.js to be supported in Screeps. OpenGL, obviously, doesn't make any sense. pthreads would require Web Workers AFAIK, and it is not supported in Node.js since they are designed for browsers, and Node.js is single-threaded.

Would .wasm file size would count against max script size I assume?

Yes, they are stored in the same storage in base64 format (approximately 4/3 of the original binary size).

UnFleshedOne

@artch OpenGL might make sense actually -- rendering room map to a texture, running shaders on it (I'm sure many image processing and computer vision algorithms can be done this way) then exposing resulting texture to the main script. Shared tenancy on GPU might be a problem though.

(Not that I'm saying we need openGL support, but that people will try to use it if it is available, and that's another resource that will need to be charged to CPU)

artch

@unfleshedone OpenGL doesn't make sense merely due to the absence of GPU on our runtime servers.

Mototroller

@artch: halp, require('binary') seems to be broken since last PTR patch or smth other update?

const binary = require('bytecode_file') causes following error:

wasm:1
(function __module(module,exports){ [object Object]
                                            ^^^^^^
SyntaxError: Unexpected identifier
    at main:12:17
    at sigintHandlersWrap (vm.js:92:15)
    at ContextifyScript.Script.runInContext (vm.js:50:12)

I just finished lzw native implementation+porting and gonna try it inside screeps, but got an error.

BTW: please, don't disable WASM on PTR at least for now. For example, I'm working on proof-of-concepts and gonna release some API and Emscripten bindings examples and tested production tools (such as native lzw en/decoder) soon this week, so I think it will be easier for community to start working with WASM after giving of some complete examples and utilities.

artch

@mototroller Should be fixed now.

Mototroller

@artch: yep, awesome, thanks!

W4rl0ck

rust has now a direct wasm build target : https://www.hellorust.com/news/native-wasm-target.html

artch

Added binary upload support to the in-game IDE:

This change will be deployed to live servers on December 14 along with upgrade to Node.js 8.

shedletsky

@artch

Yay. Now I need to know two languages to play this game well.

Really, what is the goal here?

All this does is make it incredibly painful to figure out how to write a performant AI using a bunch of undocumented features. Great.

Can we get back to making the actual game better?

artch

@shedletsky You don't need to know other languages, you are able to use them, that's a big difference. JavaScript is performant enough if used properly. But now you have a choice.

Supporting WebAssembly is just a side effect of upgrading to Node.js 8, which is inevitable since 8.x branch is now LTS (Long Term Support). WebAssembly is a new standard that is widely adopted in JavaScript world, it's not strictly related to C++, you can compile Rust or something else to it, and more languages will be supported soon. This way Screeps can (at last!) enable support of different languages besides JavaScript, and it's actually a good thing, it extends possible target audience. But you surely are not obliged to learn all these (potentially dozens of them) supported languages and are free to choose any particular language you like.

I would say that supporting WebAssembly and other languages through it may potentially become the biggest thing "making the actual game better".

tedivm

The webassembly changes are awesome.

I have a feeling not a lot of people are going to be programming their code directly in other languages, but I can see people building libraries for improved speed. @Mototroller has already started an open source webasm project, and I can imagine more are coming.

@artch, I know this probably isn't setup yet but will there be a way to incorporate webassembly in open source bots? I can compile before pushing to NPM so that isn't an issue, but it would be great if we could include these libraries.

artch

@tedivm What do you think would prevent you from doing that? You can include both .wasm files and their sources with instructions how to compile them.

PTR Changelog 2017-09-25: WebAssembly support

Example

Build .wasm file

Upload binary module

Run your native module in Screeps

Build `.wasm` file