PTR Changelog 2018-01-18: isolated VM

artch

Native PathFinder is now enabled in isolated-vm server nodes.

Mototroller

PathFinder seems to be fine, but there are relatively rare errors, see below:

Spoiler:

RangeError: Array buffer allocation failed
    at new ArrayBuffer ()
    at typedArrayConstructByLength ()
    at new Float64Array (native)
    at new Heap (:33344:27)
    // ======= ^ Screeps space
    at Object.findRoute (:33150:13)
    at Object.Creep.moveTo2 (__prototypes:1130:34)
    at Role.work (scout:179:27)
    at legacy_loop (main:1328:60)
    at LEGACY.run (main:1425:9)
    at Kernel:44:77
    // ======= ^ userspace

and

RangeError: Array buffer allocation failed
    at new ArrayBuffer ()
    at typedArrayConstructByLength ()
    at new Uint8Array (native)
    at new  (:39248:22)
    // ======= ^ Screeps space
    at staticRoomCallback (__prototypes:356:21)
    at global.civilRoomCallback (__prototypes:421:21)
    at roomCallback (__prototypes:1151:24)
    at :33018:31
    at Object.exports.search (:33030:19)
    at Object.search (:39286:38)
    // ======= ^ userspace

Moreover, sometimes V8 heap overflow errors appear unexpectedly, cannot track them for now.

artch

@mototroller You seem to exceed your heap memory quota, and game engine is no longer able to allocate some space for its needs.

PostCrafter

Will we get isolated-vm support for private servers as well?

I had the idea of giving bot creators a tool to create a binary of their bot's isolate snapshot that can be shared without publishing the source code. Currently bot publisher are trying to not publish a lot of offensive code, because it's going to be used on the official server, which in turn is hindering the experience of newer players. If this idea were to be implemented they could hold back parts of their code just like they do now, but publish the entire bot including military.

Actually implementing the upload and use of a binary snapshot can be taken care of by a mod, I'm just interested in whether we will get full isolated-vm support.

artch

@postcrafter We'll port it to private server driver once its implementation is finalized enough. isolated-vm library still doesn't have all features enabled on Windows, for example.

Bot snapshots is an interesting idea, but could be a bit tricky, since snapshots are architecture-dependent.

artch

Game.cpu.getUsed method now works with nanoseconds precision like the non-ivm version.

artch

isolated-vm is now available in private servers!

In order to switch your server to this experimental functionality, use beta prerelease tag:

npm install screeps@beta

Please note that some mods may become non-functional since playerSandbox event is not implemented yet.

It is recommended to set runners_cnt to 1 in your .screepsrc file, since otherwise it will create multiple global environments for each player. There is a new option runner_threads instead which should be set to the number of your CPU cores.

W4rl0ck

@artch Game.shard.name is not defined in the current beta

artch

BREAKING CHANGE: v3.0.0-beta.3 now published with playerSandbox event implementation, it has different API than before, see example-mods/player-sandbox.js.

W4rl0ck

The server only runs a few hours for me, then the runner is stuck with 100% cpu not writing any logs or processing ticks anymore. Also it sometimes produces a quite big core file

are you interested in any logs/information ?

artch

@w4rl0ck Try npm install github:laverdet/isolated-vm, it has some bug fixes which are not published to npm yet.

W4rl0ck

hmm.. still the same.. sometimes it runs for a while .. sometimes it locks up after some minutes. it always stop writing to logs and consume 100% cpu from there on. ( before that it uses about 30% when running fine )

those are the last lines from the last run:

---
# Main heap: 1266307072
# ExternalCopy.totalExternalSize: 4505630
# User 2 heap: 20747195
# User 3 heap: 18122793
# User 598e1fcfc0271d001a0722b2 heap: 221088411
# User 5931b0ca0d6e8c000b0303f9 heap: 187032928
# User 5a3eec0a12d7ab001b4c6964 heap: 81734967
# User 59cea4f3bd1554001b7b5912 heap: 38068152
# User 5a4146a582a099001be3fbb0 heap: 29023750
# User 59315ef9807808000bc417a4 heap: 35230562
---
---
# Main heap: 1347788800
# ExternalCopy.totalExternalSize: 4926755

also I added the runner_threads option now... but the only visible change is that it uses more then 300% cpu now when it stops running

artch

@w4rl0ck Could you please send your db.json file so that we can reproduce? By the way, have you tried disabling all installed mods?

W4rl0ck

hmm... I'm running the mongo mod... maybe that's the problem

I ran the runner in shell and it ended with:

node: ../src/isolate/run_with_timeout.h:41: ivm::RunWithTimeout(uint32_t, F&&)::<lambda()> [with F = ivm::RunRunner::Phase2()::<lambda()>]: Assertion `false' failed.

artch

Please try to find a self-contained reproducible case without any mods installed. Otherwise we won't be able to figure out whether it's an issue in engine or in some mods broken due to the playerSandbox API breaking change.

The_General

Hi there this is Marcel the author of the new sandbox environment, isolated-vm.

I think what you're seeing here is a tailcall optimization bug in v8. Essentially what happens here is sometimes v8 will generate infinite loops which cannot be stopped, this causes isolated-vm to panic. You can find the original v8 bug report here v8 issue #6933. The fix will be included in the next nodejs release, v8.10.0, but I don't know when that will be released.

In the meantime I've ported the fix to a branch on github: tailcall-backport. This branch is the nodejs v8.9.4 release but with this commit applied on top. I believe the Screeps guys are running this fix on the PTR as well. So please give that version of nodejs a try and see if it resolves this issue.

W4rl0ck

thanks @The_General. I'll test it.

I have updated my docker image to build node from that branch and include the isolated-vm git version... it's based on an old version of @ags131. If anyone is interested it's available at https://hub.docker.com/r/dafire/screeps-server/~/dockerfile/

It's based on ubuntu but using multistage docker build so it's size is only 87mb

W4rl0ck

The lockups are gone, but now after a while an other problem occurs.

28.2.2018 08:13:56# Main heap: 1968205824
28.2.2018 08:13:56# ExternalCopy.totalExternalSize: 6241895
28.2.2018 08:13:56---
28.2.2018 08:13:58releasing isolate 5931b0ca0d6e8c000b0303f9
28.2.2018 08:13:58isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:13:59creating isolate 598e1fcfc0271d001a0722b2
28.2.2018 08:13:59isolated-vm timeout 598e1fcfc0271d001a0722b2
28.2.2018 08:14:04releasing isolate 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:04isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:06isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:06---
28.2.2018 08:14:06# Main heap: 1967157248
28.2.2018 08:14:06# ExternalCopy.totalExternalSize: 4903808
28.2.2018 08:14:06# User 598e1fcfc0271d001a0722b2 heap: 19169776
28.2.2018 08:14:06---
28.2.2018 08:14:11isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:13releasing isolate 598e1fcfc0271d001a0722b2
28.2.2018 08:14:13isolated-vm timeout 598e1fcfc0271d001a0722b2
28.2.2018 08:14:17---
28.2.2018 08:14:17# Main heap: 1964240896
28.2.2018 08:14:17# ExternalCopy.totalExternalSize: 3565721
28.2.2018 08:14:17---
28.2.2018 08:14:18isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:20isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:25isolated-vm timeout 598e1fcfc0271d001a0722b2
28.2.2018 08:14:27isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:28---

restarting the runner fixes the problem.

I still have to setup the vanilla server without mods ... will do today, probably.

artch

This change will be deployed to the production world on March 5.

artch

v3.0.0-beta.4 has been published with isolated-vm@1.1.0 dependency that fixes some bugs and memory leaks.

It is recommended to launch Nodejs with --harmony_sharedarraybuffer flag in order to activate some memory optimizations.