3. PTR Changelog 2017-07-20: world shards

PTR Changelog 2017-07-20: world shards

  • Dev Team

    This post describes changes on the Public Test Realm. The estimated launch date is August 3.

    Introduced the world shards system.

    Now the world on the PTR is divided into two shards: shard0 (the original world) and shard1. You can switch between them using shard controls in the world map UI.

    Every shard is isolated from each other on the following terms:

    • Your script is executed on each shard individually and independently.
    • Your GCL and CPU limit should be allocated to each shard using the special section in the Overview UI (still in development; currently you have full GCL and CPU at every shard).
    • Console contents is broadcasted from every shard with the corresponding note in the console UI.
    • Memory is isolated; you have separate 2 MB of Memory and 10 MB of Memory Segments on each shard.
    • There is special RawMemory.interShardSegment that can be used for passing messages between shards.
    • Markets are separate. However, credits balance and history are shared between shards.
    • Terrain for each shard will be generated individually, but currently shard1 re-uses the original shard0 terrain for testing purposes.

    • Creeps can use special inter-shard portals in highway crossroad rooms to travel between shards in the same way as inter-room portals. These portals are permanent and may be regenerated only when the world is expanded. Every creep loses time to live when using inter-shard portals:

      • 800 ticks for creeps without CLAIM parts;
      • 200 ticks for creeps with CLAIM parts.

      This rule eliminates an abuse of using inter-shard portals to quickly travel in the same shard.

    API changes

  • Culture

    First of all: I'm impressed and super excited that you have a prototype of this up for testing. Great work!

    I like the idea of reducing TTL on creeps that use an inter-shard portal. I am a little worried about how difficult it will be to claim a room in another shard with the 200 tick penalty. But that may just be motivation for me to claim rooms closer to highway crossroads :).

  • Culture

    Currently I'm struggling to get a claimer to the portal with > 250 ticks remaining, meaning its going to die before being able to reach any room, getting a 500 CLAIM creep to another room is hard enough at times 😕 
    Also, interShardSegment doesn't show in the UI, is adding it planned?

  • Culture

    Pathfinding is broken in shard1, looks like the terrainCache needs rebuilt

    EDIT: Specifically findRoute, moveTo seems to be working


  • Is there any chance the terrain layout of shard1 will completely (well, strong enough) differ from shard0's one? It can be extremely interesting, if Otherworld will be more dangerous and will have some other features (smth like more Keepers, other layout generating seeds, other sources and mineral layouts etc.).

    In the late-game (especially late-game) it may introduce new level of diversity, imho 🙂

  • Culture

    • Terrain for each shard will be generated individually, but currently shard1 re-uses the original shard0 terrain for testing purposes.

  • @ags131, got it, I'm about production release, not about PTR testing environment. I haven't found any points about strong layout changing here, so seems like shards are mostly about balancing. Such a pity I missed mentioned discussion and didn't ask anything about shards' layouts there.

  • Culture

    > Your GCL and CPU limit should be allocated to each shard using the special section in the Overview UI (still in development; currently you have full GCL and CPU at every shard).

    Will there eventually be a programmatic way to set this, rather than just the UI? This will be pretty important for automation.

  • Culture

    Since RawMemory.interShardSegment is shared it means there's a race condition- if Shard0 and Shard1 both modify it during the same tick then one set of changes is lost.

    It may be better if each shard had one segment assigned to it for shard sharing, and each other shard could only view that as a read only object. So RawMemory.shardSegments[Game.shard] would be writable, but RawMemory.sharedSegments[Game.shard+1] would be read only (assuming you're on shard0). This would greatly reduce the complexity of intershard communication.

  • Culture

    It seems the novice areas from shard0 are also on the exact same location on shard1:

     

    https://screeps.com/ptr/#!/map/shard0?pos=-86.64,-7.247

    https://screeps.com/ptr/#!/map/shard1?pos=-86.64,-7.247

     

    I don't know if that's a bug or just a remainder of a cloned world.

     

     

     

    • This rule eliminates an abuse of using inter-shard portals to quickly travel in the same shard.

    I think that's part is kind of a preemptive measure. We'll see how that goes though
  • Culture

    @dissi the novice zones are just a result of cloning, controllers also have safemode cooldowns way in the future, as do decay timers on powerbanks and stuff

  • Culture

    On the topic of TTLs, if you don't have a room near a corner or aren't claiming right beside the intersection, the effective 300 TTL  of a CLAIM creep is VERY limiting, I barely had ~20-50TTL left on my claimer going from E3N31 -> E0N30 -> portal -> E0N20 -> E1N21


  • Why not just make it impossible for a creep to pass through the portals twice? As it is you could still do something like send tons of creeps through, renew them on the other side, then send them back somewhere else.
    Give creeps a property shardDamaged = true. Moving from one shard to another causes irreversible damage in a creep and using a shard portal a second time will kill it.

  • Culture

    • Your GCL and CPU limit should be allocated to each shard using the special section in the Overview UI (still in development; currently you have full GCL and CPU at every shard).

     

    I understand if there's some reason for this not to be changeable from within code, but I'm really hoping it's just a temporary limitation that we can't. I really want to be able to write code that intelligently allocates CPU to shards based on how much I need to work on those shards.

     

    I'm also slightly disappointed that I can't use shards as hyperspace, but I guess it makes sense. 🙂

     

    I'm excited for all of this! It looks really fun.

     

  • Dev Team

    Also, interShardSegment doesn’t show in the UI, is adding it planned?

    Yes, some features are still missing.

    Is there any chance the terrain layout of shard1 will completely (well, strong enough) differ from shard0’s one? It can be extremely interesting, if Otherworld will be more dangerous and will have some other features (smth like more Keepers, other layout generating seeds, other sources and mineral layouts etc.).

    In the late-game (especially late-game) it may introduce new level of diversity, imho 🙂

    Shard 1 is going to be an expansion of Shard 0 for newcomers, it doesn’t make sense to make it more dangerous.

    In future though it might be possible.

    Will there eventually be a programmatic way to set this, rather than just the UI? This will be pretty important for automation.

    Not planned currently.

    It may be better if each shard had one segment assigned to it for shard sharing, and each other shard could only view that as a read only object. So RawMemory.shardSegments[Game.shard] would be writable, but RawMemory.sharedSegments[Game.shard+1] would be read only (assuming you’re on shard0). This would greatly reduce the complexity of intershard communication.

    This way every shard should gain knowledge about every other shard in the cluster. Too tight coupling, we want to avoid such architecture. Do you think interShardSegment is too hard to use? What other people here say in this regard?

    I don’t know if that’s a bug or just a remainder of a cloned world.

    Yes, just a remainder. On the live world, shard1 will be generated from the scratch.

    I barely had ~20-50TTL left on my claimer going from E3N31 -> E0N30 -> portal -> E0N20 -> E1N21

    This seems fine. Creating inter-shard colonies are not supposed to be easy.

    Why not just make it impossible for a creep to pass through the portals twice?

    Because we don’t want it to be impossible, but rather complicated to some extent.

  • Culture

    > This way every shard should gain knowledge about every other shard in the cluster. Too tight coupling, we want to avoid such architecture. Do you think interShardSegment is too hard to use? What other people here say in this regard?

    I think it's too hard to use due to the disconnect between Game.time on each server as well as the race conditions that could occur if two shards write to things. It adds a whole lot of extra complexity into the system.

    What if instead of all of the shards having automatic access to each other shard the code can request it (similar to requesting a foreign segment from another player)? This could potentially make it less tightly coupled.

    Another option would be to add another piece of data- perhaps a second segment- that can be used as a control segment. Part of the issue right now is that a proper lock or semaphore system is impossible since you only one source to write to- the simple act of writing a lock could potentially clear data that was added from another shard. 

     

    > Will there eventually be a programmatic way to set this, rather than just the UI? This will be pretty important for automation.

    > Not planned currently.

    I, and pretty much everyone in #automation, is going to have a serious problem with this. I should not have to babysit my computer 24/7 to manage shard CPU, and I should be able to do things like push CPU to a shard where I am under attack from one where I am not. This is yet another situation where people who babysit the game will have a huge advantage over people who do not, and of course it means bots on private servers will not be able to use multiple shards. Please reconsider this, as I think it's a huge mistake that this isn't in the roadmap.

  • Culture

    Personally, I've already worked out a way to handle the interShardSegment limits, its just going to be a pain and slow to sync any data.
    The CLAIM issue means that there are going to be many players who don't have the ability to reach any claimable rooms in the other shards purely because they either 1) don't have a border room or 2) are too far away from a portal that leads to a claimable corner. 
    I've already tried claiming a few other rooms, but have been unable to reach them with CLAIM due to TTL reduction.

  • Culture

    I do think the lack of an in game API for CPU control is more important than the segment issue, for what it's worth.

    Some ideas came up in slack for alternatives- for instance, give all players a minimum of 30cpu on each shard, and then balance the rest of the CPU based on proportion of claimed rooms. However, I'm guessing that'll be too tightly coupled based off of the comments you had about segments.

  • Dev Team

    What if instead of all of the shards having automatic access to each other shard the code can request it (similar to requesting a foreign segment from another player)? This could potentially make it less tightly coupled.

    This is an option, but it introduces more complexity to this mechanic. Let’s see how it goes with interShardSegment on the PTR. I suppose all what you have to implement is some mutual locking mechanism based on real time, not on game tick. Shouldn’t be very hard to come up with.

    bots on private servers will not be able to use multiple shards

    Private servers aren’t going to have shards support at all (at least officially).

    Please reconsider this, as I think it’s a huge mistake that this isn’t in the roadmap.

    Shards resources (not just CPU, but also GCL, Power Levels, Pixels, etc) are entities of higher order than those that your scripts are operating with. Giving an ability to allocate shard resources using the game API is similar to providing an ability of, say, changing your badge or email via the API. It is related to your account and should be set up by user, not by user’s script through an in-game mechanic. The shard itself knows nothing about the multi-shard environment it exists in, similarly to how it knows nothing about other user metadata like emails.

    That being said, you can always use our web endpoints (they will get official Web API keys soon) to act on your behalf if needed.

  • Culture

    > Private servers aren’t going to have shards support at all (at least officially).

    That's disappointing, but I'm pretty sure it'll end up with unofficial support at some point.

    > Shards resources (not just CPU, but also GCL, Power Levels, Pixels, etc) are entities of higher order than those that your scripts are operating with. Giving an ability to allocate shard resources using the game API is similar to providing an ability of, say, changing your badge or email via the API. It is related to your account and should be set up by user, not by user’s script through an in-game mechanic. The shard itself knows nothing about the multi-shard environment it exists in, similarly to how it knows nothing about other user metadata like emails.

    To me this just says it's a challenge, not that it's impossible. Perhaps a special "intent" which literally just makes a call against the API you're building would resolve this. The shard doesn't need to know about the multishard environment in any way that can't be managed with the shared segment (for instance, if there are ten shards and I'm on three of them I should be able to figure that out in my own code without needing assistance from the game engine, which also means I should be able to make the adjustment from the shard with all the info I have).

    `setShardCPU({shard1: 50, shard2:30})` on shard1 triggers an intent for changing data. The engine sees that intent and (for performance reasons) dumps it in a separate queue, and that queue gets processed and literally just makes an HTTP request against the API you're already building. Sure there may be some lag between when the command in game is sent and the CPU gets allocated, but I think that's pretty fair.