Yes that's basically what I've done in the polyfill above, although I haven't done anything with high contention so it's possible that one shard might block the others if it writes every tick.

I've got it checking against a tick number so it doesn't have to compare the data itself.

@wtfrank said in InterShardSegments for each shard rather than a shared one.:

That's absolutely false. Writes to the single asynchronous block are atomic, so it's 100% possible to do locking albeit with an infinite time horizon

There are two points that everyone keeps missing when I discuss this:

1. Atomic writes are not the only requirement for safe locking. You also have to make sure that no-one else can read or write the lock in between time. You can do this in a CPU environment, but not in screeps.
2. Writes are atomic, but you can only write to the entire block. If shardA tries to change a lock flag and shardB tries to write some data, and shardA wins, the data that shardB wrote is gone. At first glance it seems like locking should prevent this situation from occurring at all, but you have to account for slow or glitchy ticks.

Thus, you cannot implement an independent locking system that is 100% safe, because we do not have the guarantees required for this. As previously stated, locking systems which require cooperation between shards are possible, but will block communication if one shard is down (and any attempt to resolve this makes it unsafe again).

I contend that anyone who thinks they have implemented a 100% safe locking system has made one of the following assumptions:

1. That shards all run at the same rate and no shard is ever down (not true)
2. That only one shard is executing at a time (not true, shards are executing simultaneously on different servers)
3. That you can read and write the segment atomically without any other shard reading or writing in between (not guaranteed, since shards are executing simultaneously)
4. That there is a limit to the time between reading the segment and writing it (not true, there are no guarantees here)
5. That you can atomically update one part of the segment (e.g. a flag) without changing the rest of the segment (not true, it's all one block)

Now I'm sure many people have locking systems implemented and working fine 99.9% of the time. The point is that it's not 100% safe, which means you have to think about what happens when it goes wrong. I suspect for most people the answer is that they don't care (so long as their locking system recovers without blocking everything). They'll lose some data, but it either wasn't important or will be re-written again soon enough. This is perfectly valid, but working 99.9% of the time and not caring when it goes wrong is not the same as provably 100% safe.

The interesting thing about this is that any one of the assumptions above would allow us to implement 100% safe locking. It's entirely possible that assumption 4 is actually already true, but we would need guarantees from the devs before we could rely on it.

Of course, the new InterShardMemory scheme is much simpler and solves all these problems.

I can see uses for both approaches. Sometimes you want to send a one-off message to a specific shard or broadcast to all shards. But I think there is a strong case for just keeping some information available that other shards can read when they like.

Given a shared memory system you can implement messaging fairly easily. You can also do it the other way around, but I think this is more of a pain.

The primary issue with TDMA is "large enough time slots". Ticks are several seconds so giving enough time to make that even vaguely safe is completely untenable. Also, it's still not safe even with an arbitrary amount of time.

The key problem is that shards read interShardSegment at the beginning of the tick and write to it at the end, which introduces a substantial delay between reading and writing. Since shards sometimes get very slow, it's entirely possible that one shard could have several ticks during this time. When the slow shard finally catches up, it might write something into the shard memory that was based on a version several ticks ago from the perspective of the other shards.

You can fix this by having a locking system that requires all shards to acknowledge the lock before continuing, or by passing control to the next shard (which has to happen continously, even if no other changes are being made). The issue with these approaches is that you now have to give a list of all the shards you're actually using. And then you have the issue of dealing with shards which are hanging or stuck, which brings us back to the whole problem again, because you have no true guarantee that a shard marked as stuck won't come back and write something really old into your memory.

That said, so far I've been thinking about this in terms of the entire tick length. If the reading/writing happens individually before/after each player's code then that would significantly reduce the problematic delay, although I'm still not sure we can place any guarantees on it.

For anyone else who is interested, I've implemented a polyfill for the new API:

// Polyfill for new InterShardMemory API.
// InterShardMemory.init() must be called at the beginning of the main loop.
// InterShardMemory.update() must be called at the end of the main loop.
let SHARDS = null;

function _load() {
    if (SHARDS) {
        return;
    }
    SHARDS = JSON.parse(RawMemory.interShardSegment) || {};
}

global.InterShardMemory = {
    getLocal: function () {
        let shard = Game.shard.name;
        if (Memory.interShard) {
            return Memory.interShard[1];
        }
        return InterShardMemory.getRemote(shard);
    },

    setLocal: function (data) {
        if (typeof data !== 'string') {
            throw new Error('InterShardMemory must be a string');
        }
        let shard = Game.shard.name;
        Memory.interShard = [Game.time, data];
    },

    getRemote: function (shard) {
        _load();
        if (! SHARDS[shard]) {
            return '';
        }
        return SHARDS[shard][1];
    },

    init: function () {
        // clear the cache each tick
        SHARDS = null;
    },

    update: function () {
        let pending = Memory.interShard;
        if (! pending) {
            return;
        }
        _load();
        let shard = Game.shard.name;
        let entry = SHARDS[shard] || {};
        if (entry[0] != pending[0]) {
            // Try to update the shared interShardSegment
            SHARDS[shard] = pending;
            RawMemory.interShardSegment = JSON.stringify(SHARDS);
        }
    },
};

Example usage:

require('./intershard');
exports.loop = function () {
    InterShardMemory.init();
    // your code
    InterShardMemory.update();
};

NOTE: It is impossible to guarantee that RawMemory.interShardSegment isn't overwritten by another shard with a slow tick rate. No amount of locking can fix this - the existing API is fundamentally broken (it is impossible to implement locking when your only shared memory is a single asynchronous block). As a workaround, each shard's inter-shard memory will also be stored in Memory.interShard, which will be used to update the RawMemory.interShardSegment if it gets overwritten.

I regularly see this in the browser as well, so it's not just the steam client that has the problem.

I think I've found both solutions, and for both of them it's 6 ticks to move both creeps from fully inside room1 to fully inside room2 (e.g. with neither creep on an edge tile). With a lone creep the same movement would take 3 ticks.

For a creep pulling two wagons, I can do it in 9 ticks (a lone creep would take 4 ticks).

If anyone can do it faster I'd love to know!

It was a bug in my code! I've got it working now. Thanks.

I did try this on PTR and I did manage to move the wagon to the next room, but it was pretty fiddly. Is this how it is intended? Maybe there is no better solution? It's a shame because it makes trains with more than one wagon impractical.

Maybe it's an issue with my code but so far I haven't been able to move the wagon off the exit tile in the new room. Despite pull and move both returning 0, the wagon teleports back to the first room instead of moving with the train.