PTR Changelog 2017-02-17


  • Dev Team

    This post describes changes on the Public Test Realm.

    Introduced new memory segments feature:

    • You can have up to 10 MB of additional memory by requesting asynchronous memory segments.
    • Each memory segment is a string with an ID from 0 to 99.
    • Maximum segment data length is 100 KB.
    • Segments are asynchronous, you should request them using RawMemory.requestSegments. The data will be available on the next tick.
    • You cannot request more than 10 segments simultaneously on the same tick.
    • Requested segments are available in RawMemory.segmentsobject and are saved automatically.
    • Requestng and saving segments have no added CPU cost.

    Example:

    RawMemory.requestSegments([0,5]);
    // on the next tick
    RawMemory.segments[0] = 'data1';
    RawMemory.segments[5] = 'data2';

    You can use this feature to store data which is not needed every tick, for example pathfinding caches, or collected statistics. Offloading data structures from Memory to asynchronous segments will allow to save some CPU spent on Memory parsing.

    Tell us what do you think about these upcoming changes in the comments below!


  • YP

    Sounds interesting .) Three ideas/questions come into mind:

    1. Would it be possible to get a segment automatically loaded when your code runs on new node / a reset occured? that way it would be possible to cache some pathfinding data or a costmatrix or something like that outside of the loop. Requesting a segment would not really work for that because the next tick is probably on another server.

    2. HTTP Api to segments ( a very interesting plus would be the posiibilty to give access to a segement using an api key ... so a alliance could update a memberlist or a 3rd party page could access stats )

    3. Append Data to segments without loading them


  • Dev Team

    1 . Would it be possible to get a segment automatically loaded when your code runs on new node / a reset occured?

    There is no way to reliably detect a reset from runtime code. You may try to request a segment multiple times until it is loaded to the needed node.

    2 . HTTP Api to segments ( a very interesting plus would be the posiibilty to give access to a segement using an api key … so a alliance could update a memberlist or a 3rd party page could access stats )

    We’ll consider this.

    3 . Append Data to segments without loading them

    It’s not possible within the current architecture.


  • YP

    I guess requesting a segment and having it loaded without processing it is quite cheap?


  • Dev Team

    It's simply an object with strings. Should be free for you. Server-side impact is also very little, it's all about <1MB of network data transfer per tick per user.


  • YP

    This really sounds great 😉

     

    So you could request an segment every tick and only process if needed.


  • Culture

    Nice change!

    I’m going to test this later today, I do voice the same concerns as w4rl0ck, I can currently mark an area of memory as dirty the same tick it’s calculated. I also store this memory in global, but the next tick that data might be gone.

    Lets assume tick 1:

        var somethingCalculatedInGlobal = "stuff";
        function loop() {
            if(stuff != "somethingWeThoughtWasGood") {
                stuff = "somethingWeThoughtWasGood";
            }
            // We didn't know we needed this RawMemory segment, but we need to update globals of other segments.
    
        <span class="hljs-comment">// Ideally I'd want to do RawMemory.segments['simenumber'] = stuff; here. </span>
        <span class="hljs-comment">// Mark it as updated so other globals can load it next tick</span>
        RawMemory.requestSegment(<span class="hljs-number">1</span>);
        Memory.updateSegmentsForNextTick = [<span class="hljs-number">1</span>]; 
    }</code></pre>
    

    Next tick 2:

    GLOBAL RESET HAPPENS
    or worse
    GLOBAL CHANGE HAPPENS ( switched to another server, with different global )

        var somethingCalculatedInGlobal = "stuff";
        function loop() {
            for(var i = 0; i < updateSegmentsForNextTick.length; i++) {
                RawMemory.storeSegment(1, somethingCalculatedInGlobal); // Now we store the wrong thing "stuff" due to a global reset
                Memory.updateSegmentsForNextTick = [1]; 
            }
        }

    This is extremely hard to detect, and requires more CPU to handle. I’d plead that we are allowed to store memory segments every tick (max of 10 seems fine).

    If it was possible to always do RawMemory.segments['simenumber'] = stuff; it should be easier to manage


    Aside from async storing

    I’d also recommend letting users set the size of their segments, 100 KB isn’t nearly enough for pathing data, even optimized ones.
    I’m currently using 284.9 KB for my “Promoted Routes segment”. I can split this up, but it would be more fun to have different sizes. I almost never do any pathfinding inter-tick.

     

    Thanks for the tip for using https://stackedit.io/editor


  • Dev Team

    I’d plead that we are allowed to store memory segments every tick (max of 10 seems fine).

    Yes, you can. If you requested some segments, they will be saved automatically as well. There is no RawMemory.storeSegment method. You can request up to 10 segments each tick, and they will always be saved every time. But you cannot request (and thus save) more than 10 segments at the same time.

    I’d also recommend letting users set the size of their segments, 100 KB isn’t nearly enough for pathing data, even optimized ones.
    I’m currently using 284.9 KB for my “Promoted Routes segment”. I can split this up, but it would be more fun to have different sizes. I almost never do any pathfinding inter-tick.

    You can write your own abstract layer on top of segments, which will concatenate your data on fetching, and split it across multiple segments in the end of the tick.


  • Culture

    Currently, I can work around this, no question about it.

     

    The problem arises when you got more than 1MB of runtime data to load after a global reset. You might not be able to request data you need during reset storms.

    The issue I have can be solved by implementing some abstraction layer with "L0" cache( first 5 segments are always loaded, should only contain data which is critical for your code to run) "L1" 3 optional segments to load next tick, and make actions based on that (load floorplan, next tick check said floorplan etc). and 2 for random use when storing of "dirty" data.

     

    The problem is only with loading more than 1 MB of data after global resets. It might be worthwhile to look into finding out when a new global is created for a certain user, and passing this information to the user script. Even making node ID's (if there are any) we run on known to the script may bring some cool new play styles or ideas to the table..

     

    For storing it just remembered I'm an idiot. I can just store it in Memory for 1 tick and the next tick dump it to RawMemory. I forgot you can use both.

    I'm excited to start using this stuff!

     


  • Dev Team

    The problem arises when you got more than 1MB of runtime data to load after a global reset.

    Loading more than 1 MB of segments data is possible only during multiple ticks. This is a design decision, in order not to saturate server internal network.


  • Culture

    Fair enough!


  • Culture

     

    Can we get a fresh copy of prod on PTR? 


  • Dev Team

    Can we get a fresh copy of prod on PTR?

    Done!


  • Culture

    Seems like the Game.time was offset, I think my code can recover from the current state the rooms are in though.


  • Culture

    Can we make it so RawMemory.requestSegments([0,5]); is remembered between ticks? This way we won’t have to re-request memory segments every time. This helps when you get timed-out or when a tick errors out (CPU limit reached).

    You could effectively “swap” databanks by calling RawMemory.requestSegments([1,6]);


  • Culture

    Maybe to alleviate network I/O you'd have to explicitly mark a memory segment as dirty before it's saved to the database.


  • Dev Team

    Can we make it so RawMemory.requestSegments([0,5]); is remembered between ticks? This way we won’t have to re-request memory segments every time. This helps when you get timed-out or when a tick errors out (CPU limit reached).

    Good point. It should persist between ticks now.


  • Culture

    Thanks! Amazing change!


  • Culture

    It would be amazing if we could put data into segments that aren't "active" yet.

    I also want to say that making segments available via HTTP would really help with the various stat collection programs.



  • Also for backups/testing! I like to periodically grab a snapshot of my memory & room objects for porting over to a private test server.



  • Thank you for adding this!

     

    This seems like an incredibly useful feature, and one thing would make it even more useful: the ability to save segments that aren't loaded during the tick. Even if it was limited to 10 total segments saved during the tick, this would allow for an expensive calculation that should be merged into one of the "main" segments being saved in an "extra" segment reserved for that specific purpose.

     

    Of course, this could be worked around using values in Memory, but if that can be avoided, that would be ideal for me. Say I'm storing a list of paths that my script should remember & request in segment 2, and on one tick I need to path urgently, when I hadn't expected it (perhaps an invader showed up, and waiting a tick would not be acceptable). The code could create and save the path (and some metadata) to segment 82 (not previously requested), and then request it for the next tick. The next tick, the code now seeing that it has segment 82 and segment 2 loaded, would merge the data from 82 into 2.

     

    That would be just one usage of this feature (the ability to save non-loaded segments), but I could imagine it would be very useful in other cases as well, and would alleviate some of the possibly expensive calculations needed to predict which segments data will need to be stored into!