PTR Changelog 2018-01-18: isolated VM


  • Culture

    PTR seems to be running really slowly right now, which is making testing harder.


  • Dev Team

    Native PathFinder is now enabled in isolated-vm server nodes.



  • PathFinder seems to be fine, but there are relatively rare errors, see below:

    Spoiler:
    RangeError: Array buffer allocation failed
        at new ArrayBuffer ()
        at typedArrayConstructByLength ()
        at new Float64Array (native)
        at new Heap (:33344:27)
        // ======= ^ Screeps space
        at Object.findRoute (:33150:13)
        at Object.Creep.moveTo2 (__prototypes:1130:34)
        at Role.work (scout:179:27)
        at legacy_loop (main:1328:60)
        at LEGACY.run (main:1425:9)
        at Kernel:44:77
        // ======= ^ userspace
    

    and

    RangeError: Array buffer allocation failed
        at new ArrayBuffer ()
        at typedArrayConstructByLength ()
        at new Uint8Array (native)
        at new  (:39248:22)
        // ======= ^ Screeps space
        at staticRoomCallback (__prototypes:356:21)
        at global.civilRoomCallback (__prototypes:421:21)
        at roomCallback (__prototypes:1151:24)
        at :33018:31
        at Object.exports.search (:33030:19)
        at Object.search (:39286:38)
        // ======= ^ userspace
    

    Moreover, sometimes V8 heap overflow errors appear unexpectedly, cannot track them for now.


  • Dev Team

    @mototroller You seem to exceed your heap memory quota, and game engine is no longer able to allocate some space for its needs.



  • Will we get isolated-vm support for private servers as well?

    I had the idea of giving bot creators a tool to create a binary of their bot's isolate snapshot that can be shared without publishing the source code. Currently bot publisher are trying to not publish a lot of offensive code, because it's going to be used on the official server, which in turn is hindering the experience of newer players. If this idea were to be implemented they could hold back parts of their code just like they do now, but publish the entire bot including military.

    Actually implementing the upload and use of a binary snapshot can be taken care of by a mod, I'm just interested in whether we will get full isolated-vm support.


  • Dev Team

    @postcrafter We'll port it to private server driver once its implementation is finalized enough. isolated-vm library still doesn't have all features enabled on Windows, for example.

    Bot snapshots is an interesting idea, but could be a bit tricky, since snapshots are architecture-dependent.

    👍

  • Dev Team

    Game.cpu.getUsed method now works with nanoseconds precision like the non-ivm version.


  • Dev Team

    isolated-vm is now available in private servers!

    In order to switch your server to this experimental functionality, use beta prerelease tag:

    npm install screeps@beta
    

    Please note that some mods may become non-functional since playerSandbox event is not implemented yet.

    It is recommended to set runners_cnt to 1 in your .screepsrc file, since otherwise it will create multiple global environments for each player. There is a new option runner_threads instead which should be set to the number of your CPU cores.

    👍

  • YP

    @artch Game.shard.name is not defined in the current beta


  • Dev Team

    BREAKING CHANGE: v3.0.0-beta.3 now published with playerSandbox event implementation, it has different API than before, see example-mods/player-sandbox.js.

    👍

  • YP

    The server only runs a few hours for me, then the runner is stuck with 100% cpu not writing any logs or processing ticks anymore. Also it sometimes produces a quite big core file 😉

    are you interested in any logs/information ?


  • Dev Team

    @w4rl0ck Try npm install github:laverdet/isolated-vm, it has some bug fixes which are not published to npm yet.


  • YP

    hmm.. still the same.. sometimes it runs for a while .. sometimes it locks up after some minutes. it always stop writing to logs and consume 100% cpu from there on. ( before that it uses about 30% when running fine )

    those are the last lines from the last run:

    ---
    # Main heap: 1266307072
    # ExternalCopy.totalExternalSize: 4505630
    # User 2 heap: 20747195
    # User 3 heap: 18122793
    # User 598e1fcfc0271d001a0722b2 heap: 221088411
    # User 5931b0ca0d6e8c000b0303f9 heap: 187032928
    # User 5a3eec0a12d7ab001b4c6964 heap: 81734967
    # User 59cea4f3bd1554001b7b5912 heap: 38068152
    # User 5a4146a582a099001be3fbb0 heap: 29023750
    # User 59315ef9807808000bc417a4 heap: 35230562
    ---
    ---
    # Main heap: 1347788800
    # ExternalCopy.totalExternalSize: 4926755
    

    also I added the runner_threads option now... but the only visible change is that it uses more then 300% cpu now when it stops running 🙂


  • Dev Team

    @w4rl0ck Could you please send your db.json file so that we can reproduce? By the way, have you tried disabling all installed mods?


  • YP

    hmm... I'm running the mongo mod... maybe that's the problem

    I ran the runner in shell and it ended with:

    node: ../src/isolate/run_with_timeout.h:41: ivm::RunWithTimeout(uint32_t, F&&)::<lambda()> [with F = ivm::RunRunner::Phase2()::<lambda()>]: Assertion `false' failed.


  • Dev Team

    Please try to find a self-contained reproducible case without any mods installed. Otherwise we won't be able to figure out whether it's an issue in engine or in some mods broken due to the playerSandbox API breaking change.



  • Hi there this is Marcel the author of the new sandbox environment, isolated-vm.

    I think what you're seeing here is a tailcall optimization bug in v8. Essentially what happens here is sometimes v8 will generate infinite loops which cannot be stopped, this causes isolated-vm to panic. You can find the original v8 bug report here v8 issue #6933. The fix will be included in the next nodejs release, v8.10.0, but I don't know when that will be released.

    In the meantime I've ported the fix to a branch on github: tailcall-backport. This branch is the nodejs v8.9.4 release but with this commit applied on top. I believe the Screeps guys are running this fix on the PTR as well. So please give that version of nodejs a try and see if it resolves this issue.

    🤟

  • YP

    thanks @The_General. I'll test it.

    I have updated my docker image to build node from that branch and include the isolated-vm git version... it's based on an old version of @ags131. If anyone is interested it's available at https://hub.docker.com/r/dafire/screeps-server/~/dockerfile/

    It's based on ubuntu but using multistage docker build so it's size is only 87mb 🙂


  • YP

    The lockups are gone, but now after a while an other problem occurs.

    28.2.2018 08:13:56# Main heap: 1968205824
    28.2.2018 08:13:56# ExternalCopy.totalExternalSize: 6241895
    28.2.2018 08:13:56---
    28.2.2018 08:13:58releasing isolate 5931b0ca0d6e8c000b0303f9
    28.2.2018 08:13:58isolated-vm timeout 5931b0ca0d6e8c000b0303f9
    28.2.2018 08:13:59creating isolate 598e1fcfc0271d001a0722b2
    28.2.2018 08:13:59isolated-vm timeout 598e1fcfc0271d001a0722b2
    28.2.2018 08:14:04releasing isolate 5931b0ca0d6e8c000b0303f9
    28.2.2018 08:14:04isolated-vm timeout 5931b0ca0d6e8c000b0303f9
    28.2.2018 08:14:06isolated-vm timeout 5931b0ca0d6e8c000b0303f9
    28.2.2018 08:14:06---
    28.2.2018 08:14:06# Main heap: 1967157248
    28.2.2018 08:14:06# ExternalCopy.totalExternalSize: 4903808
    28.2.2018 08:14:06# User 598e1fcfc0271d001a0722b2 heap: 19169776
    28.2.2018 08:14:06---
    28.2.2018 08:14:11isolated-vm timeout 5931b0ca0d6e8c000b0303f9
    28.2.2018 08:14:13releasing isolate 598e1fcfc0271d001a0722b2
    28.2.2018 08:14:13isolated-vm timeout 598e1fcfc0271d001a0722b2
    28.2.2018 08:14:17---
    28.2.2018 08:14:17# Main heap: 1964240896
    28.2.2018 08:14:17# ExternalCopy.totalExternalSize: 3565721
    28.2.2018 08:14:17---
    28.2.2018 08:14:18isolated-vm timeout 5931b0ca0d6e8c000b0303f9
    28.2.2018 08:14:20isolated-vm timeout 5931b0ca0d6e8c000b0303f9
    28.2.2018 08:14:25isolated-vm timeout 598e1fcfc0271d001a0722b2
    28.2.2018 08:14:27isolated-vm timeout 5931b0ca0d6e8c000b0303f9
    28.2.2018 08:14:28---
    

    restarting the runner fixes the problem.

    I still have to setup the vanilla server without mods ... will do today, probably.


  • Dev Team

    This change will be deployed to the production world on March 5.