Enabled experimental WebAssembly support. WebAssembly is a binary compiled code format that allows to run C/C++ or Rust code (as well as other supported languages in the future) directly with native performance. Please refer to the WebAssembly documentation for more info.

This is an experimental feature. We need your feedback in order to decide if it worth it. Please share your experience of what you were able to build using WebAssembly.

Example

Here is a short example of how to compile C/C++ code using Emscripten and upload the binary file to Screeps.

Build .wasm file

You can skip this step if you use an already compiled .wasm file from the web.

Install Emsripten SDK using these instructions.

Write your native C function and save it as addTwo.c file:

int addTwo(int a, int b) {
    return a + b;
}

Compile it to addTwo.wasm using this command:

emcc -s WASM=1 -s SIDE_MODULE=1 -O3 addTwo.c -o addTwo.wasm

Upload binary module

npm install screeps/grunt-screeps#binary-upload

module.exports = function(grunt) {

    grunt.loadNpmTasks('grunt-screeps');

    grunt.initConfig({
        screeps: {
            options: {
                email: 'YOUR_EMAIL',
                password: 'YOUR_PASSWORD',
                branch: 'default',
                ptr: true
            },
            dist: {
                files: [
                    {
                        expand: true,
                        cwd: 'dist/',
                        src: ['**/*.{js,wasm}'],  // <----
                        flatten: true
                    }
                ]
            }
        }
    });
}

Run your native module in Screeps

If you uploaded your binary module correctly, you should see the following in your in-game IDE:

Now you can use the following code to run your imported binary code in main using WebAssembly API:

// This will return an ArrayBuffer with `addTwo.wasm` binary contents
const bytecode = require('addTwo');

const wasmModule = new WebAssembly.Module(bytecode);

const imports = {};

// Some predefined environment for Emscripten
imports.env = {
    memoryBase: 0,
    tableBase: 0,
    memory: new WebAssembly.Memory({ initial: 256 }),
    table: new WebAssembly.Table({ initial: 0, element: 'anyfunc' })    
};

const wasmInstance = new WebAssembly.Instance(wasmModule, imports);

console.log(wasmInstance.exports._addTwo(2,3));

• What is it actually aimed at? Is performance of native modules the main reason? Or, is it a next step of virtualization reworking?
• How will CPU consuming be counted while using native modules? And what about exception and hard-interruption points in case of CPU-, stack- or heap- overflows?
• Are any NATIVE <=> GAME APIs planned? Alternatively, will users be able to use raw WA <=> JS APIs (memory mappings, pointers sharing etc.)?
• As I remember, promises (and other async stuff) aren't handled correctly for now by game engine: is it guaranteed that native module loading will be consistent (no resets until new file will be uploaded, for example)?

Being a cpp developer and being faced some time ago with WA projects, I assume that WA introducing is really, REALLY hard and breaking change for run-time environment (new APIs, new bugs, new non-portabilities, suddenly new vulnerabilities -- hello native LLVM!), I just wanna wish good luck and make sure such a useful and powerful feature will be implemented with such a close attention as previous roadmap implemented features.

What is it actually aimed at? Is performance of native modules the main reason? Or, is it a next step of virtualization reworking?

It's an option to write critical code natively to increase your script's performance.

How will CPU consuming be counted while using native modules? And what about exception and hard-interruption points in case of CPU-, stack- or heap- overflows?

Everything remains the same, all WebAssembly manipulations are done within your regular VM environment.

Are any NATIVE <=> GAME APIs planned?

Not officially, but it's an interesting vacant area for community projects.

Alternatively, will users be able to use raw WA <=> JS APIs (memory mappings, pointers sharing etc.)?

All APIs in the WebAssembly namespace are exposed to the user space in full.

As I remember, promises (and other async stuff) aren't handled correctly for now by game engine: is it guaranteed that native module loading will be consistent (no resets until new file will be uploaded, for example)?

Using async API is not mandatory. The example above is already changed to the synchronous instantiation.

Being a cpp developer and being faced some time ago with WA projects, I assume that WA introducing is really, REALLY hard and breaking change for run-time environment (new APIs, new bugs, new non-portabilities, suddenly new vulnerabilities -- hello native LLVM!), I just wanna wish good luck and make sure such a useful and powerful feature will be implemented with such a close attention as previous roadmap implemented features.

Actually, WebAssembly object is exposed by default on Node 8.x, so I don't think it is really that breaking.

Please share your experience of what you were able to build using WebAssembly.

Sorry, missed the request. Some words about WA experience:

Being inspired by Screeps, I was looking for a scalable, high performance and virtualization-friendly solution for Screeps-like AI environment. The candidates were Lua (provides high performance virtual machine and a simple script language) and WA (extremely high performance + built-in virtualization by design). Well, Python should be mentioned here: it's very difficult to build a bulletproof VM for it, and its performance without dirty hacks is really bad. I realized that Lua is not popular enough to be a main language for AI describing and developing, so WA was chosen.

There were several underwater rocks while implementing scalable architecture of shared AI sandboxes:

• It was difficult to find a compromise between API flexibility, performance and environment restrictions ("Game map as a shrared mmaped raw memory? Awesome, but memory protection? Map updates? Synchronization?")
• For now it's still not very comfortable to perform IPC with WA (message queue layer was written as dirty mix of WA API, JS and native C++ -- this layer and its endpoints seems for me overcomplicated, but all the workarounds are using pretty same API calls, and I've found the performance of portable robust solution is not very nice due to tons of unnecessary memory copying). Memory mapping and sharing (mmap or shmem etc.) are sometimes broken due to unexpected Node or WA restrictions.
• Due to environment restrictions, it's very difficult to debug native modules or write correct test suits. Moreover, I'm not sure about C++11 exception support by WA nowadays. So, it's extremely easy to hang or drop native module... or even write a malicious code due to leaky Node + WA API and permissions vulnerabilities -- it still seems not safe enough.

Because of points mentioned above, I've found WA modules are very useful for "dense" computational (some difficult algorithmic code with few inputs and outputs), but not flexible enough to be a full replacement of JS in most egine's cases despite on performance benefits. For now my project is frozen, but I've got tons of experience, and gonna return to it later with new vision and experience (and may be some followers:).

@artch: Thank you for reply!

It's an option to write critical code natively to increase your script's performance.

Nice idea, imho. Endpoints and synchronization problems are still difficult (see above), but it can really increase performance.

All APIs in the WebAssembly namespace are exposed to the user space in full.

It's a disputable point due to safety problems, but still nice for beginning. Offtop: Screeps community is really very nice here to not to abuse or even looking for vulnerabilities, and report them:)

Actually, WebAssembly object is exposed by default on Node 8.x, so I don't think it is really that breaking.

Oh, it's nice, but I'm mostly about safety underwater rocks (and, as you pointed out, about rewriting critical code).

I wonder if webassambly could not only be used in the user side of code ... but maybe also make some of the server code faster ?

WebAssembly says it does or will eventually support pthreads, OpenGL and other system libraries. Will those be available in screeps and how would pthreads for example impact CPU usage? (haven't had a chance to actually try it on PTR yet)

Would .wasm file size would count against max script size I assume?

WebAssembly says it does or will eventually support pthreads, OpenGL and other system libraries. Will those be available in screeps and how would pthreads for example impact CPU usage? (haven't had a chance to actually try it on PTR yet)

All WebAssemply features should backed by Node.js to be supported in Screeps. OpenGL, obviously, doesn't make any sense. pthreads would require Web Workers AFAIK, and it is not supported in Node.js since they are designed for browsers, and Node.js is single-threaded.

Would .wasm file size would count against max script size I assume?

Yes, they are stored in the same storage in base64 format (approximately 4/3 of the original binary size).

(Not that I'm saying we need openGL support, but that people will try to use it if it is available, and that's another resource that will need to be charged to CPU)

const binary = require('bytecode_file') causes following error:

wasm:1
(function __module(module,exports){ [object Object]
                                            ^^^^^^
SyntaxError: Unexpected identifier
    at main:12:17
    at sigintHandlersWrap (vm.js:92:15)
    at ContextifyScript.Script.runInContext (vm.js:50:12)

I just finished lzw native implementation+porting and gonna try it inside screeps, but got an error.

BTW: please, don't disable WASM on PTR at least for now. For example, I'm working on proof-of-concepts and gonna release some API and Emscripten bindings examples and tested production tools (such as native lzw en/decoder) soon this week, so I think it will be easier for community to start working with WASM after giving of some complete examples and utilities.

This change will be deployed to live servers on December 14 along with upgrade to Node.js 8.

Yay. Now I need to know two languages to play this game well.

Really, what is the goal here?

All this does is make it incredibly painful to figure out how to write a performant AI using a bunch of undocumented features. Great.

Can we get back to making the actual game better?

Supporting WebAssembly is just a side effect of upgrading to Node.js 8, which is inevitable since 8.x branch is now LTS (Long Term Support). WebAssembly is a new standard that is widely adopted in JavaScript world, it's not strictly related to C++, you can compile Rust or something else to it, and more languages will be supported soon. This way Screeps can (at last!) enable support of different languages besides JavaScript, and it's actually a good thing, it extends possible target audience. But you surely are not obliged to learn all these (potentially dozens of them) supported languages and are free to choose any particular language you like.

I would say that supporting WebAssembly and other languages through it may potentially become the biggest thing "making the actual game better".

I have a feeling not a lot of people are going to be programming their code directly in other languages, but I can see people building libraries for improved speed. @Mototroller has already started an open source webasm project, and I can imagine more are coming.

@artch, I know this probably isn't setup yet but will there be a way to incorporate webassembly in open source bots? I can compile before pushing to NPM so that isn't an issue, but it would be great if we could include these libraries.

I think my confusion was that previous (I thought) they had to be uploaded to a separate branch, but it looks like that's no longer the case.

dist/
  main.js
  addTwo.wasm

This corresponds to Screeps modules:

and thought we were uploading binaries to a separate branch on screeps. What you're saying makes for more sense.

Being inspired by Screeps, I was looking for a scalable, high performance and virtualization-friendly solution for Screeps-like AI environment. The candidates were Lua (provides high performance virtual machine and a simple script language) and WA (extremely high performance + built-in virtualization by design).

I've been in a similar situation recently and when evaluating WA vs Lua I decided for Lua.

WA is definitely a big thing, but in it's current state you are in for a lot of trouble. Even if you ignore security risks - and since this thing is basically still alpha you actually really shouldn't - the issues when trying to get a project done with WA are currently too excessive. Eventually WA will become a reasonable alternative, but that's something far in the future.

The main issue with WA is error handling. Even the best programmer will eventually make a mistake, and at this point you need good debugging and logging to find and fix it. WA makes debugging a nightmare. All tools you are used to from JS debugging will not function, and also all tools you might be used to from C/C++ developing will not function either. You can handle some issues with logging, recompiling and waiting (which is very tedious), but some of the lower-level issues like incorrect memory access, pointer failures, off-by-one errors, all that are a major pain to find and fix when you have zero tools to help you. In the end all you can do is to write extensive test-cases and hope to catch all errors, then spend days debugging those you didn't catch.

And on top of that most JS programmers are not experienced with these kind of bugs, as they simply do not exist (in that way) in JS. Which means that many people will have to learn a completely new language (and all the issues that come with it) first, and in the worst way possible.

Lua was definitely the better choice for my kind of problem. While it is not as fast as native C, it's still much faster than JS and therefore suitable for performance-critical parts. On the other hand debugging and writing code is much simpler, as Lua is much closer to JS than C.