Allow access to node's Buffer class

Reply to Allow access to node's Buffer class on Invalid Date

Esryok — Invalid Date

I'm building a heap on top of the RawMemory global to improve script perf, but JavaScript doesn't have great support for converting between byte arrays and strings. The least-bad method I've found is using String.fromCharCode.apply, which starts hitting stack overflows around 115 KB.

Can we get access to node's Buffer class in the player scripts?

Reply to Allow access to node's Buffer class on Invalid Date

wtfrank — Invalid Date

This would be useful for integrating node-lz4 into my bot.

Reply to Allow access to node's Buffer class on Invalid Date

deft-code — Invalid Date

What's special about the Buffer class? As I understand the API memory still needs to be stored as a string in MongoDB so the conversion has to take place somewhere.

The best approach I've found for stuff like that is to keep a copy in global and drop checkpoints into Memory.

A concrete example. I store CostMatrix in global but serialize it to a sparse array for storage inMemory.

Reply to Allow access to node's Buffer class on Fri, 05 Apr 2019 17:24:23 GMT

wtfrank — Fri, 05 Apr 2019 17:24:23 GMT

Buffer has convenient support for converting between a string and an array. That's what interested me in it.

Reply to Allow access to node's Buffer class on Invalid Date

JBYoshi — Invalid Date

One problem: this wouldn't work in the simulation, since it can be run in the browser without Node support. Screeps would need a polyfill to handle that. If there's a polyfill available, you can then just drop that into your existing code.

A quick search online gives me this: https://github.com/feross/buffer

Reply to Allow access to node's Buffer class on Invalid Date

aphistic — Invalid Date

I'd like to add my desire to have this added as well. I'm working on getting Rust working via wasm-bindgen (the "blessed" wasm implementation for Rust) and I've had a number of hoops to jump through in order to get it working. The most recent one is the Rust wasm javascript shim using Buffer.from() and Buffer.byteLength but failing because Buffer is unavailable.

Reply to Allow access to node's Buffer class on Invalid Date

wtfrank — Invalid Date

@jbyoshi This is a good point about the simulation.

The polyfill though has awful performance. Converting some arrays of bytes to a string with node's Buffer class takes 242ms, while converting those same arrays of bytes with the polyfill buffer class takes 793ms.

I believe Node's buffer class uses C underneath, which means it doesn't have to create loads of intermediate string objects and explains why it's so much quicker to convert a string this way.

The Buffer polyfill could perhaps be provided for the simulator, while the node Buffer class could be provided on the MMO server...

Reply to Allow access to node's Buffer class on Invalid Date

W4rl0ck — Invalid Date

I don't think that simulator should be the only reason to not allow that .. there could be advanced features that won't work in the simulator... it just should be documented somewhere.. and it should print a warning.

For example I don't use the simulator at all.

Reply to Allow access to node's Buffer class on Invalid Date

deft-code — Invalid Date

Pathfinder is amazing and accelerated in C like Node's Buffer. We assume we have a slower api compatible version for the simulator.

Reply to Allow access to node's Buffer class on Tue, 23 Apr 2019 10:45:27 GMT

wtfrank — Tue, 23 Apr 2019 10:45:27 GMT

More benchnmarking: some code that amongst other things copies some strings into the wasm heap via a Buffer takes 414ms, but the same code that uses the polyfill Buffer takes 1878ms.

So the polyfill buffer has abysmal performance both at converting javascript strings to arrays, and converting arrays into javascript strings.

The polyfill Buffer is not something that anybody should willingly use!

Reply to Allow access to node's Buffer class on Invalid Date

Athestus — Invalid Date

Like @aphistic, I am also trying to use Rust through wasm-bindgen. Currently, I build with an automated script that run wasm-packet -t nodejs and then use sed to replace the Buffer-dependent methods with their counterparts when targeting browser. This works, and if anyone wants this solution just message me and I'll publish it, but I would love to have the performance benefits of Buffer.