PTR Changelog 2017-02-17

[W4rl0ck]

I guess requesting a segment and having it loaded without processing it is quite cheap?

[artch]

It's simply an object with strings. Should be free for you. Server-side impact is also very little, it's all about <1MB of network data transfer per tick per user.

[W4rl0ck]

This really sounds great

 

So you could request an segment every tick and only process if needed.

[Dissi]

Nice change!

I’m going to test this later today, I do voice the same concerns as w4rl0ck, I can currently mark an area of memory as dirty the same tick it’s calculated. I also store this memory in global, but the next tick that data might be gone.

Lets assume tick 1:

    var somethingCalculatedInGlobal = "stuff";
    function loop() {
        if(stuff != "somethingWeThoughtWasGood") {
            stuff = "somethingWeThoughtWasGood";
        }
        // We didn't know we needed this RawMemory segment, but we need to update globals of other segments.
    <span class="hljs-comment">// Ideally I'd want to do RawMemory.segments['simenumber'] = stuff; here. </span>
    <span class="hljs-comment">// Mark it as updated so other globals can load it next tick</span>
    RawMemory.requestSegment(<span class="hljs-number">1</span>);
    Memory.updateSegmentsForNextTick = [<span class="hljs-number">1</span>]; 
}</code></pre>

Next tick 2:
GLOBAL RESET HAPPENS 
or worse 
GLOBAL CHANGE HAPPENS ( switched to another server, with different global )

    var somethingCalculatedInGlobal = "stuff";
    function loop() {
        for(var i = 0; i < updateSegmentsForNextTick.length; i++) {
            RawMemory.storeSegment(1, somethingCalculatedInGlobal); // Now we store the wrong thing "stuff" due to a global reset
            Memory.updateSegmentsForNextTick = [1]; 
        }
    }

This is extremely hard to detect, and requires more CPU to handle. I’d plead that we are allowed to store memory segments every tick (max of 10 seems fine).

If it was possible to always do RawMemory.segments['simenumber'] = stuff; it should be easier to manage

---

Aside from async storing

I’d also recommend letting users set the size of their segments, 100 KB isn’t nearly enough for pathing data, even optimized ones.
I’m currently using 284.9 KB for my “Promoted Routes segment”. I can split this up, but it would be more fun to have different sizes. I almost never do any pathfinding inter-tick.

 

Thanks for the tip for using https://stackedit.io/editor

[artch]

I’d plead that we are allowed to store memory segments every tick (max of 10 seems fine).

Yes, you can. If you requested some segments, they will be saved automatically as well. There is no RawMemory.storeSegment method. You can request up to 10 segments each tick, and they will always be saved every time. But you cannot request (and thus save) more than 10 segments at the same time.

I’d also recommend letting users set the size of their segments, 100 KB isn’t nearly enough for pathing data, even optimized ones.
I’m currently using 284.9 KB for my “Promoted Routes segment”. I can split this up, but it would be more fun to have different sizes. I almost never do any pathfinding inter-tick.

You can write your own abstract layer on top of segments, which will concatenate your data on fetching, and split it across multiple segments in the end of the tick.

[Dissi]

Currently, I can work around this, no question about it.

 

The problem arises when you got more than 1MB of runtime data to load after a global reset. You might not be able to request data you need during reset storms.

The issue I have can be solved by implementing some abstraction layer with "L0" cache( first 5 segments are always loaded, should only contain data which is critical for your code to run) "L1" 3 optional segments to load next tick, and make actions based on that (load floorplan, next tick check said floorplan etc). and 2 for random use when storing of "dirty" data.

 

The problem is only with loading more than 1 MB of data after global resets. It might be worthwhile to look into finding out when a new global is created for a certain user, and passing this information to the user script. Even making node ID's (if there are any) we run on known to the script may bring some cool new play styles or ideas to the table..

 

For storing it just remembered I'm an idiot. I can just store it in Memory for 1 tick and the next tick dump it to RawMemory. I forgot you can use both.

I'm excited to start using this stuff!

 

[artch]

The problem arises when you got more than 1MB of runtime data to load after a global reset.

Loading more than 1 MB of segments data is possible only during multiple ticks. This is a design decision, in order not to saturate server internal network.

[Dissi]

Fair enough!

[Dissi]

 

Can we get a fresh copy of prod on PTR? 

[artch]

Can we get a fresh copy of prod on PTR?

Done!

[Dissi]

Seems like the Game.time was offset, I think my code can recover from the current state the rooms are in though.

[Dissi]

Can we make it so RawMemory.requestSegments([0,5]); is remembered between ticks? This way we won’t have to re-request memory segments every time. This helps when you get timed-out or when a tick errors out (CPU limit reached).

You could effectively “swap” databanks by calling RawMemory.requestSegments([1,6]);

[Dissi]

Maybe to alleviate network I/O you'd have to explicitly mark a memory segment as dirty before it's saved to the database.

[artch]

Can we make it so RawMemory.requestSegments([0,5]); is remembered between ticks? This way we won’t have to re-request memory segments every time. This helps when you get timed-out or when a tick errors out (CPU limit reached).

Good point. It should persist between ticks now.

[Dissi]

Thanks! Amazing change!

[tedivm]

It would be amazing if we could put data into segments that aren't "active" yet.

I also want to say that making segments available via HTTP would really help with the various stat collection programs.

[Esryok]

Also for backups/testing! I like to periodically grab a snapshot of my memory & room objects for porting over to a private test server.

[daboross]

Thank you for adding this!

 

This seems like an incredibly useful feature, and one thing would make it even more useful: the ability to save segments that aren't loaded during the tick. Even if it was limited to 10 total segments saved during the tick, this would allow for an expensive calculation that should be merged into one of the "main" segments being saved in an "extra" segment reserved for that specific purpose.

 

Of course, this could be worked around using values in Memory, but if that can be avoided, that would be ideal for me. Say I'm storing a list of paths that my script should remember & request in segment 2, and on one tick I need to path urgently, when I hadn't expected it (perhaps an invader showed up, and waiting a tick would not be acceptable). The code could create and save the path (and some metadata) to segment 82 (not previously requested), and then request it for the next tick. The next tick, the code now seeing that it has segment 82 and segment 2 loaded, would merge the data from 82 into 2.

 

That would be just one usage of this feature (the ability to save non-loaded segments), but I could imagine it would be very useful in other cases as well, and would alleviate some of the possibly expensive calculations needed to predict which segments data will need to be stored into!

[tedivm]

Would it be possible to add an "isSet" style function to see if a segment is in use (even if it's not "active")?

[tedivm]

What happens if we attempt to store data in a segment that isn't available? Will it throw an error?

How will we know which segments are available? `Object.keys(RawMemory.segments)` ?