PTR Changelog 2017-02-17

artch

This post describes changes on the Public Test Realm.

Introduced new memory segments feature:

You can have up to 10 MB of additional memory by requesting asynchronous memory segments.
Each memory segment is a string with an ID from 0 to 99.
Maximum segment data length is 100 KB.
Segments are asynchronous, you should request them using RawMemory.requestSegments. The data will be available on the next tick.
You cannot request more than 10 segments simultaneously on the same tick.
Requested segments are available in RawMemory.segmentsobject and are saved automatically.
Requestng and saving segments have no added CPU cost.

Example:

RawMemory.requestSegments([0,5]);
// on the next tick
RawMemory.segments[0] = 'data1';
RawMemory.segments[5] = 'data2';

You can use this feature to store data which is not needed every tick, for example pathfinding caches, or collected statistics. Offloading data structures from Memory to asynchronous segments will allow to save some CPU spent on Memory parsing.

Tell us what do you think about these upcoming changes in the comments below!

W4rl0ck

Sounds interesting .) Three ideas/questions come into mind:

1. Would it be possible to get a segment automatically loaded when your code runs on new node / a reset occured? that way it would be possible to cache some pathfinding data or a costmatrix or something like that outside of the loop. Requesting a segment would not really work for that because the next tick is probably on another server.

2. HTTP Api to segments ( a very interesting plus would be the posiibilty to give access to a segement using an api key ... so a alliance could update a memberlist or a 3rd party page could access stats )

3. Append Data to segments without loading them

artch

1 . Would it be possible to get a segment automatically loaded when your code runs on new node / a reset occured?

There is no way to reliably detect a reset from runtime code. You may try to request a segment multiple times until it is loaded to the needed node.

2 . HTTP Api to segments ( a very interesting plus would be the posiibilty to give access to a segement using an api key … so a alliance could update a memberlist or a 3rd party page could access stats )

We’ll consider this.

3 . Append Data to segments without loading them

It’s not possible within the current architecture.

W4rl0ck

I guess requesting a segment and having it loaded without processing it is quite cheap?

artch

It's simply an object with strings. Should be free for you. Server-side impact is also very little, it's all about <1MB of network data transfer per tick per user.

W4rl0ck

This really sounds great

So you could request an segment every tick and only process if needed.

Dissi

Nice change!

I’m going to test this later today, I do voice the same concerns as w4rl0ck, I can currently mark an area of memory as dirty the same tick it’s calculated. I also store this memory in global, but the next tick that data might be gone.

Lets assume tick 1:

    var somethingCalculatedInGlobal = "stuff";
    function loop() {
        if(stuff != "somethingWeThoughtWasGood") {
            stuff = "somethingWeThoughtWasGood";
        }
        // We didn't know we needed this RawMemory segment, but we need to update globals of other segments.
    <span class="hljs-comment">// Ideally I'd want to do RawMemory.segments['simenumber'] = stuff; here. </span>
    <span class="hljs-comment">// Mark it as updated so other globals can load it next tick</span>
    RawMemory.requestSegment(<span class="hljs-number">1</span>);
    Memory.updateSegmentsForNextTick = [<span class="hljs-number">1</span>]; 
}</code></pre>

Next tick 2:
GLOBAL RESET HAPPENS 
or worse 
GLOBAL CHANGE HAPPENS ( switched to another server, with different global )



    var somethingCalculatedInGlobal = "stuff";
    function loop() {
        for(var i = 0; i < updateSegmentsForNextTick.length; i++) {
            RawMemory.storeSegment(1, somethingCalculatedInGlobal); // Now we store the wrong thing "stuff" due to a global reset
            Memory.updateSegmentsForNextTick = [1]; 
        }
    }
This is extremely hard to detect, and requires more CPU to handle. I’d plead that we are allowed to store memory segments every tick (max of 10 seems fine).
If it was possible to always do RawMemory.segments['simenumber'] = stuff; it should be easier to manage

Aside from async storing
I’d also recommend letting users set the size of their segments, 100 KB isn’t nearly enough for pathing data, even optimized ones. 
I’m currently using 284.9 KB for my “Promoted Routes segment”. I can split this up, but it would be more fun to have different sizes. I almost never do any pathfinding inter-tick.
 
Thanks for the tip for using https://stackedit.io/editor

artch

I’d plead that we are allowed to store memory segments every tick (max of 10 seems fine).

Yes, you can. If you requested some segments, they will be saved automatically as well. There is no RawMemory.storeSegment method. You can request up to 10 segments each tick, and they will always be saved every time. But you cannot request (and thus save) more than 10 segments at the same time.

I’d also recommend letting users set the size of their segments, 100 KB isn’t nearly enough for pathing data, even optimized ones.
I’m currently using 284.9 KB for my “Promoted Routes segment”. I can split this up, but it would be more fun to have different sizes. I almost never do any pathfinding inter-tick.

You can write your own abstract layer on top of segments, which will concatenate your data on fetching, and split it across multiple segments in the end of the tick.

Dissi

Currently, I can work around this, no question about it.

The problem arises when you got more than 1MB of runtime data to load after a global reset. You might not be able to request data you need during reset storms.

The issue I have can be solved by implementing some abstraction layer with "L0" cache( first 5 segments are always loaded, should only contain data which is critical for your code to run) "L1" 3 optional segments to load next tick, and make actions based on that (load floorplan, next tick check said floorplan etc). and 2 for random use when storing of "dirty" data.

The problem is only with loading more than 1 MB of data after global resets. It might be worthwhile to look into finding out when a new global is created for a certain user, and passing this information to the user script. Even making node ID's (if there are any) we run on known to the script may bring some cool new play styles or ideas to the table..

For storing it just remembered I'm an idiot. I can just store it in Memory for 1 tick and the next tick dump it to RawMemory. I forgot you can use both.

I'm excited to start using this stuff!

artch

The problem arises when you got more than 1MB of runtime data to load after a global reset.

Loading more than 1 MB of segments data is possible only during multiple ticks. This is a design decision, in order not to saturate server internal network.

Dissi

Fair enough!

Dissi

Can we get a fresh copy of prod on PTR?

artch

Can we get a fresh copy of prod on PTR?

Done!

Dissi

Seems like the Game.time was offset, I think my code can recover from the current state the rooms are in though.

Dissi

Can we make it so RawMemory.requestSegments([0,5]); is remembered between ticks? This way we won’t have to re-request memory segments every time. This helps when you get timed-out or when a tick errors out (CPU limit reached).

You could effectively “swap” databanks by calling RawMemory.requestSegments([1,6]);

Dissi

Maybe to alleviate network I/O you'd have to explicitly mark a memory segment as dirty before it's saved to the database.

artch

Can we make it so RawMemory.requestSegments([0,5]); is remembered between ticks? This way we won’t have to re-request memory segments every time. This helps when you get timed-out or when a tick errors out (CPU limit reached).

Good point. It should persist between ticks now.

Dissi

Thanks! Amazing change!

tedivm

It would be amazing if we could put data into segments that aren't "active" yet.

I also want to say that making segments available via HTTP would really help with the various stat collection programs.

Esryok

Also for backups/testing! I like to periodically grab a snapshot of my memory & room objects for porting over to a private test server.

daboross

Thank you for adding this!

This seems like an incredibly useful feature, and one thing would make it even more useful: the ability to save segments that aren't loaded during the tick. Even if it was limited to 10 total segments saved during the tick, this would allow for an expensive calculation that should be merged into one of the "main" segments being saved in an "extra" segment reserved for that specific purpose.

Of course, this could be worked around using values in Memory, but if that can be avoided, that would be ideal for me. Say I'm storing a list of paths that my script should remember & request in segment 2, and on one tick I need to path urgently, when I hadn't expected it (perhaps an invader showed up, and waiting a tick would not be acceptable). The code could create and save the path (and some metadata) to segment 82 (not previously requested), and then request it for the next tick. The next tick, the code now seeing that it has segment 82 and segment 2 loaded, would merge the data from 82 into 2.

That would be just one usage of this feature (the ability to save non-loaded segments), but I could imagine it would be very useful in other cases as well, and would alleviate some of the possibly expensive calculations needed to predict which segments data will need to be stored into!