3. PTR Changelog 2018-01-18: isolated VM

PTR Changelog 2018-01-18: isolated VM

  • YP

    The server only runs a few hours for me, then the runner is stuck with 100% cpu not writing any logs or processing ticks anymore. Also it sometimes produces a quite big core file 😉

    are you interested in any logs/information ?

    1 Reply
  • Dev Team

    @w4rl0ck Try npm install github:laverdet/isolated-vm, it has some bug fixes which are not published to npm yet.

  • YP

    hmm.. still the same.. sometimes it runs for a while .. sometimes it locks up after some minutes. it always stop writing to logs and consume 100% cpu from there on. ( before that it uses about 30% when running fine )

    those are the last lines from the last run:

    ---
# Main heap: 1266307072
# ExternalCopy.totalExternalSize: 4505630
# User 2 heap: 20747195
# User 3 heap: 18122793
# User 598e1fcfc0271d001a0722b2 heap: 221088411
# User 5931b0ca0d6e8c000b0303f9 heap: 187032928
# User 5a3eec0a12d7ab001b4c6964 heap: 81734967
# User 59cea4f3bd1554001b7b5912 heap: 38068152
# User 5a4146a582a099001be3fbb0 heap: 29023750
# User 59315ef9807808000bc417a4 heap: 35230562
---
---
# Main heap: 1347788800
# ExternalCopy.totalExternalSize: 4926755

    also I added the runner_threads option now... but the only visible change is that it uses more then 300% cpu now when it stops running 🙂

    1 Reply
  • Dev Team

    @w4rl0ck Could you please send your db.json file so that we can reproduce? By the way, have you tried disabling all installed mods?

  • YP

    hmm... I'm running the mongo mod... maybe that's the problem

    I ran the runner in shell and it ended with:

    node: ../src/isolate/run_with_timeout.h:41: ivm::RunWithTimeout(uint32_t, F&&)::<lambda()> [with F = ivm::RunRunner::Phase2()::<lambda()>]: Assertion `false' failed.

  • Dev Team

    Please try to find a self-contained reproducible case without any mods installed. Otherwise we won't be able to figure out whether it's an issue in engine or in some mods broken due to the playerSandbox API breaking change.


  • Hi there this is Marcel the author of the new sandbox environment, isolated-vm.

    I think what you're seeing here is a tailcall optimization bug in v8. Essentially what happens here is sometimes v8 will generate infinite loops which cannot be stopped, this causes isolated-vm to panic. You can find the original v8 bug report here v8 issue #6933. The fix will be included in the next nodejs release, v8.10.0, but I don't know when that will be released.

    In the meantime I've ported the fix to a branch on github: tailcall-backport. This branch is the nodejs v8.9.4 release but with this commit applied on top. I believe the Screeps guys are running this fix on the PTR as well. So please give that version of nodejs a try and see if it resolves this issue.

    🤟
  • YP

    thanks @The_General. I'll test it.

    I have updated my docker image to build node from that branch and include the isolated-vm git version... it's based on an old version of @ags131. If anyone is interested it's available at https://hub.docker.com/r/dafire/screeps-server/~/dockerfile/

    It's based on ubuntu but using multistage docker build so it's size is only 87mb 🙂

  • YP

    The lockups are gone, but now after a while an other problem occurs.

    28.2.2018 08:13:56# Main heap: 1968205824
28.2.2018 08:13:56# ExternalCopy.totalExternalSize: 6241895
28.2.2018 08:13:56---
28.2.2018 08:13:58releasing isolate 5931b0ca0d6e8c000b0303f9
28.2.2018 08:13:58isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:13:59creating isolate 598e1fcfc0271d001a0722b2
28.2.2018 08:13:59isolated-vm timeout 598e1fcfc0271d001a0722b2
28.2.2018 08:14:04releasing isolate 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:04isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:06isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:06---
28.2.2018 08:14:06# Main heap: 1967157248
28.2.2018 08:14:06# ExternalCopy.totalExternalSize: 4903808
28.2.2018 08:14:06# User 598e1fcfc0271d001a0722b2 heap: 19169776
28.2.2018 08:14:06---
28.2.2018 08:14:11isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:13releasing isolate 598e1fcfc0271d001a0722b2
28.2.2018 08:14:13isolated-vm timeout 598e1fcfc0271d001a0722b2
28.2.2018 08:14:17---
28.2.2018 08:14:17# Main heap: 1964240896
28.2.2018 08:14:17# ExternalCopy.totalExternalSize: 3565721
28.2.2018 08:14:17---
28.2.2018 08:14:18isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:20isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:25isolated-vm timeout 598e1fcfc0271d001a0722b2
28.2.2018 08:14:27isolated-vm timeout 5931b0ca0d6e8c000b0303f9
28.2.2018 08:14:28---

    restarting the runner fixes the problem.

    I still have to setup the vanilla server without mods ... will do today, probably.

  • Dev Team

    This change will be deployed to the production world on March 5.

  • Dev Team

    v3.0.0-beta.4 has been published with isolated-vm@1.1.0 dependency that fixes some bugs and memory leaks.

    It is recommended to launch Nodejs with --harmony_sharedarraybuffer flag in order to activate some memory optimizations.

  • YP

    Thanks for the update .. I will update my server and see how it goes .)

  • YP

    Looks like in the current beta RoomVisuals are not working for me...