PTR Changelog 2017-09-25: WebAssembly support

[Mototroller]

Please share your experience of what you were able to build using WebAssembly.

Sorry, missed the request. Some words about WA experience:

Being inspired by Screeps, I was looking for a scalable, high performance and virtualization-friendly solution for Screeps-like AI environment. The candidates were Lua (provides high performance virtual machine and a simple script language) and WA (extremely high performance + built-in virtualization by design). Well, Python should be mentioned here: it's very difficult to build a bulletproof VM for it, and its performance without dirty hacks is really bad. I realized that Lua is not popular enough to be a main language for AI describing and developing, so WA was chosen.

There were several underwater rocks while implementing scalable architecture of shared AI sandboxes:

• It was difficult to find a compromise between API flexibility, performance and environment restrictions ("Game map as a shrared mmaped raw memory? Awesome, but memory protection? Map updates? Synchronization?")
• For now it's still not very comfortable to perform IPC with WA (message queue layer was written as dirty mix of WA API, JS and native C++ -- this layer and its endpoints seems for me overcomplicated, but all the workarounds are using pretty same API calls, and I've found the performance of portable robust solution is not very nice due to tons of unnecessary memory copying). Memory mapping and sharing (mmap or shmem etc.) are sometimes broken due to unexpected Node or WA restrictions.
• Due to environment restrictions, it's very difficult to debug native modules or write correct test suits. Moreover, I'm not sure about C++11 exception support by WA nowadays. So, it's extremely easy to hang or drop native module... or even write a malicious code due to leaky Node + WA API and permissions vulnerabilities -- it still seems not safe enough.

Because of points mentioned above, I've found WA modules are very useful for "dense" computational (some difficult algorithmic code with few inputs and outputs), but not flexible enough to be a full replacement of JS in most egine's cases despite on performance benefits. For now my project is frozen, but I've got tons of experience, and gonna return to it later with new vision and experience (and may be some followers:).

---

@artch: Thank you for reply!

It's an option to write critical code natively to increase your script's performance.

Nice idea, imho. Endpoints and synchronization problems are still difficult (see above), but it can really increase performance.

All APIs in the WebAssembly namespace are exposed to the user space in full.

It's a disputable point due to safety problems, but still nice for beginning. Offtop: Screeps community is really very nice here to not to abuse or even looking for vulnerabilities, and report them:)

Actually, WebAssembly object is exposed by default on Node 8.x, so I don't think it is really that breaking.

Oh, it's nice, but I'm mostly about safety underwater rocks (and, as you pointed out, about rewriting critical code).

[ags131]

I've been testing with wasm and so it seems to perform very well, I'm not sure of the memory usage, but overall CPU usage tends to be less than similar JS code, even with the calls back into JS space. My current testing code is on GitHub, it uses a different approach than the guide above and handles much more of the marshaling of types and helps deal with pointers and such. With this, I'm easily able to spawn and random move over 300 creeps with almost no overhead. If this ends up being available, I wouldn't want to switch all my code over to it, I'm not a C++ guy, but I would implement the more computationally intensive tasks in C++ to reduce CPU. Stuff like distance transform, skeletonization, layout planning, path compression, this can all be done in C++ and reduce overall CPU required.

[W4rl0ck]

I'll give it a try now.

I wonder if webassambly could not only be used in the user side of code ... but maybe also make some of the server code faster ?

[UnFleshedOne]

Nice, this is definitely something I would use if it ever makes it in. Finally my work experience apply to something that matters!

WebAssembly says it does or will eventually support pthreads, OpenGL and other system libraries. Will those be available in screeps and how would pthreads for example impact CPU usage? (haven't had a chance to actually try it on PTR yet)

Would .wasm file size would count against max script size I assume?

[artch]

@unfleshedone

WebAssembly says it does or will eventually support pthreads, OpenGL and other system libraries. Will those be available in screeps and how would pthreads for example impact CPU usage? (haven't had a chance to actually try it on PTR yet)

All WebAssemply features should backed by Node.js to be supported in Screeps. OpenGL, obviously, doesn't make any sense. pthreads would require Web Workers AFAIK, and it is not supported in Node.js since they are designed for browsers, and Node.js is single-threaded.

Would .wasm file size would count against max script size I assume?

Yes, they are stored in the same storage in base64 format (approximately 4/3 of the original binary size).

[UnFleshedOne]

@artch OpenGL might make sense actually -- rendering room map to a texture, running shaders on it (I'm sure many image processing and computer vision algorithms can be done this way) then exposing resulting texture to the main script. Shared tenancy on GPU might be a problem though.

(Not that I'm saying we need openGL support, but that people will try to use it if it is available, and that's another resource that will need to be charged to CPU)

[artch]

@unfleshedone OpenGL doesn't make sense merely due to the absence of GPU on our runtime servers.

[Mototroller]

@artch: halp, require('binary') seems to be broken since last PTR patch or smth other update?

const binary = require('bytecode_file') causes following error:

wasm:1
(function __module(module,exports){ [object Object]
                                            ^^^^^^
SyntaxError: Unexpected identifier
    at main:12:17
    at sigintHandlersWrap (vm.js:92:15)
    at ContextifyScript.Script.runInContext (vm.js:50:12)

I just finished lzw native implementation+porting and gonna try it inside screeps, but got an error.

BTW: please, don't disable WASM on PTR at least for now. For example, I'm working on proof-of-concepts and gonna release some API and Emscripten bindings examples and tested production tools (such as native lzw en/decoder) soon this week, so I think it will be easier for community to start working with WASM after giving of some complete examples and utilities.

[artch]

@mototroller Should be fixed now.

[Mototroller]

@artch: yep, awesome, thanks!

[W4rl0ck]

rust has now a direct wasm build target : https://www.hellorust.com/news/native-wasm-target.html

[artch]

Added binary upload support to the in-game IDE:

This change will be deployed to live servers on December 14 along with upgrade to Node.js 8.

[shedletsky]

@artch

Yay. Now I need to know two languages to play this game well.

Really, what is the goal here?

All this does is make it incredibly painful to figure out how to write a performant AI using a bunch of undocumented features. Great.

Can we get back to making the actual game better?

[artch]

@shedletsky You don't need to know other languages, you are able to use them, that's a big difference. JavaScript is performant enough if used properly. But now you have a choice.

Supporting WebAssembly is just a side effect of upgrading to Node.js 8, which is inevitable since 8.x branch is now LTS (Long Term Support). WebAssembly is a new standard that is widely adopted in JavaScript world, it's not strictly related to C++, you can compile Rust or something else to it, and more languages will be supported soon. This way Screeps can (at last!) enable support of different languages besides JavaScript, and it's actually a good thing, it extends possible target audience. But you surely are not obliged to learn all these (potentially dozens of them) supported languages and are free to choose any particular language you like.

I would say that supporting WebAssembly and other languages through it may potentially become the biggest thing "making the actual game better".

[tedivm]

The webassembly changes are awesome.

I have a feeling not a lot of people are going to be programming their code directly in other languages, but I can see people building libraries for improved speed. @Mototroller has already started an open source webasm project, and I can imagine more are coming.

@artch, I know this probably isn't setup yet but will there be a way to incorporate webassembly in open source bots? I can compile before pushing to NPM so that isn't an issue, but it would be great if we could include these libraries.

[artch]

@tedivm What do you think would prevent you from doing that? You can include both .wasm files and their sources with instructions how to compile them.

[tedivm]

@artch but once they're compiled what do we do with them? Can we just leave the compiled binaries alongside the regular code and expect it to work?

I think my confusion was that previous (I thought) they had to be uploaded to a separate branch, but it looks like that's no longer the case.

[artch]

@tedivm I lost you here, why they have to be uploaded to a separate branch? They are plain binary files which are uploaded to binary Screeps modules via either in-game IDE, grunt-screeps or Steam client. Like in the example above, you have:

dist/
  main.js
  addTwo.wasm

This corresponds to Screeps modules:

[tedivm]

@artch thanks for clarifying. I had misread this-

and thought we were uploading binaries to a separate branch on screeps. What you're saying makes for more sense.

[TwoThe]

@mototroller said in PTR Changelog 2017-09-25: WebAssembly support:

Being inspired by Screeps, I was looking for a scalable, high performance and virtualization-friendly solution for Screeps-like AI environment. The candidates were Lua (provides high performance virtual machine and a simple script language) and WA (extremely high performance + built-in virtualization by design).

I've been in a similar situation recently and when evaluating WA vs Lua I decided for Lua.

WA is definitely a big thing, but in it's current state you are in for a lot of trouble. Even if you ignore security risks - and since this thing is basically still alpha you actually really shouldn't - the issues when trying to get a project done with WA are currently too excessive. Eventually WA will become a reasonable alternative, but that's something far in the future.

The main issue with WA is error handling. Even the best programmer will eventually make a mistake, and at this point you need good debugging and logging to find and fix it. WA makes debugging a nightmare. All tools you are used to from JS debugging will not function, and also all tools you might be used to from C/C++ developing will not function either. You can handle some issues with logging, recompiling and waiting (which is very tedious), but some of the lower-level issues like incorrect memory access, pointer failures, off-by-one errors, all that are a major pain to find and fix when you have zero tools to help you. In the end all you can do is to write extensive test-cases and hope to catch all errors, then spend days debugging those you didn't catch.

And on top of that most JS programmers are not experienced with these kind of bugs, as they simply do not exist (in that way) in JS. Which means that many people will have to learn a completely new language (and all the issues that come with it) first, and in the worst way possible.

Lua was definitely the better choice for my kind of problem. While it is not as fast as native C, it's still much faster than JS and therefore suitable for performance-critical parts. On the other hand debugging and writing code is much simpler, as Lua is much closer to JS than C.