4. Allow access to node's Buffer class

Allow access to node's Buffer class


  • I'm building a heap on top of the RawMemory global to improve script perf, but JavaScript doesn't have great support for converting between byte arrays and strings. The least-bad method I've found is using String.fromCharCode.apply, which starts hitting stack overflows around 115 KB.

    Can we get access to node's Buffer class in the player scripts?

    ☝

  • This would be useful for integrating node-lz4 into my bot.


  • What's special about the Buffer class? As I understand the API memory still needs to be stored as a string in MongoDB so the conversion has to take place somewhere.

    The best approach I've found for stuff like that is to keep a copy in global and drop checkpoints into Memory.

    A concrete example. I store CostMatrix in global but serialize it to a sparse array for storage inMemory.


  • Buffer has convenient support for converting between a string and an array. That's what interested me in it.


  • One problem: this wouldn't work in the simulation, since it can be run in the browser without Node support. Screeps would need a polyfill to handle that. If there's a polyfill available, you can then just drop that into your existing code.

    A quick search online gives me this: https://github.com/feross/buffer

    1 Reply

  • I'd like to add my desire to have this added as well. I'm working on getting Rust working via wasm-bindgen (the "blessed" wasm implementation for Rust) and I've had a number of hoops to jump through in order to get it working. The most recent one is the Rust wasm javascript shim using Buffer.from() and Buffer.byteLength but failing because Buffer is unavailable.


  • @jbyoshi This is a good point about the simulation.

    The polyfill though has awful performance. Converting some arrays of bytes to a string with node's Buffer class takes 242ms, while converting those same arrays of bytes with the polyfill buffer class takes 793ms.

    I believe Node's buffer class uses C underneath, which means it doesn't have to create loads of intermediate string objects and explains why it's so much quicker to convert a string this way.

    The Buffer polyfill could perhaps be provided for the simulator, while the node Buffer class could be provided on the MMO server...

    1 Reply
  • YP

    I don't think that simulator should be the only reason to not allow that .. there could be advanced features that won't work in the simulator... it just should be documented somewhere.. and it should print a warning.

    For example I don't use the simulator at all.


  • Pathfinder is amazing and accelerated in C like Node's Buffer. We assume we have a slower api compatible version for the simulator.


  • More benchnmarking: some code that amongst other things copies some strings into the wasm heap via a Buffer takes 414ms, but the same code that uses the polyfill Buffer takes 1878ms.

    So the polyfill buffer has abysmal performance both at converting javascript strings to arrays, and converting arrays into javascript strings.

    The polyfill Buffer is not something that anybody should willingly use!


  • Like @aphistic, I am also trying to use Rust through wasm-bindgen. Currently, I build with an automated script that run wasm-packet -t nodejs and then use sed to replace the Buffer-dependent methods with their counterparts when targeting browser. This works, and if anyone wants this solution just message me and I'll publish it, but I would love to have the performance benefits of Buffer.