PTR Changelog 2017-07-20: world shards

[tedivm]

Since RawMemory.interShardSegment is shared it means there's a race condition- if Shard0 and Shard1 both modify it during the same tick then one set of changes is lost.

It may be better if each shard had one segment assigned to it for shard sharing, and each other shard could only view that as a read only object. So RawMemory.shardSegments[Game.shard] would be writable, but RawMemory.sharedSegments[Game.shard+1] would be read only (assuming you're on shard0). This would greatly reduce the complexity of intershard communication.

[Dissi]

It seems the novice areas from shard0 are also on the exact same location on shard1:

 

https://screeps.com/ptr/#!/map/shard0?pos=-86.64,-7.247

https://screeps.com/ptr/#!/map/shard1?pos=-86.64,-7.247

 

I don't know if that's a bug or just a remainder of a cloned world.

 

 

 

• This rule eliminates an abuse of using inter-shard portals to quickly travel in the same shard.

I think that's part is kind of a preemptive measure. We'll see how that goes though

[ags131]

@dissi the novice zones are just a result of cloning, controllers also have safemode cooldowns way in the future, as do decay timers on powerbanks and stuff

[ags131]

On the topic of TTLs, if you don't have a room near a corner or aren't claiming right beside the intersection, the effective 300 TTL  of a CLAIM creep is VERY limiting, I barely had ~20-50TTL left on my claimer going from E3N31 -> E0N30 -> portal -> E0N20 -> E1N21

[Davaned]

Why not just make it impossible for a creep to pass through the portals twice? As it is you could still do something like send tons of creeps through, renew them on the other side, then send them back somewhere else.
Give creeps a property shardDamaged = true. Moving from one shard to another causes irreversible damage in a creep and using a shard portal a second time will kill it.

[anisoptera]

• Your GCL and CPU limit should be allocated to each shard using the special section in the Overview UI (still in development; currently you have full GCL and CPU at every shard).

 

I understand if there's some reason for this not to be changeable from within code, but I'm really hoping it's just a temporary limitation that we can't. I really want to be able to write code that intelligently allocates CPU to shards based on how much I need to work on those shards.

 

I'm also slightly disappointed that I can't use shards as hyperspace, but I guess it makes sense.

 

I'm excited for all of this! It looks really fun.

 

[artch]

Also, interShardSegment doesn’t show in the UI, is adding it planned?

Yes, some features are still missing.

Is there any chance the terrain layout of shard1 will completely (well, strong enough) differ from shard0’s one? It can be extremely interesting, if Otherworld will be more dangerous and will have some other features (smth like more Keepers, other layout generating seeds, other sources and mineral layouts etc.).

In the late-game (especially late-game) it may introduce new level of diversity, imho

Shard 1 is going to be an expansion of Shard 0 for newcomers, it doesn’t make sense to make it more dangerous.

In future though it might be possible.

Will there eventually be a programmatic way to set this, rather than just the UI? This will be pretty important for automation.

Not planned currently.

It may be better if each shard had one segment assigned to it for shard sharing, and each other shard could only view that as a read only object. So RawMemory.shardSegments[Game.shard] would be writable, but RawMemory.sharedSegments[Game.shard+1] would be read only (assuming you’re on shard0). This would greatly reduce the complexity of intershard communication.

This way every shard should gain knowledge about every other shard in the cluster. Too tight coupling, we want to avoid such architecture. Do you think interShardSegment is too hard to use? What other people here say in this regard?

I don’t know if that’s a bug or just a remainder of a cloned world.

Yes, just a remainder. On the live world, shard1 will be generated from the scratch.

I barely had ~20-50TTL left on my claimer going from E3N31 -> E0N30 -> portal -> E0N20 -> E1N21

This seems fine. Creating inter-shard colonies are not supposed to be easy.

Why not just make it impossible for a creep to pass through the portals twice?

Because we don’t want it to be impossible, but rather complicated to some extent.

[tedivm]

> This way every shard should gain knowledge about every other shard in the cluster. Too tight coupling, we want to avoid such architecture. Do you think interShardSegment is too hard to use? What other people here say in this regard?

I think it's too hard to use due to the disconnect between Game.time on each server as well as the race conditions that could occur if two shards write to things. It adds a whole lot of extra complexity into the system.

What if instead of all of the shards having automatic access to each other shard the code can request it (similar to requesting a foreign segment from another player)? This could potentially make it less tightly coupled.

Another option would be to add another piece of data- perhaps a second segment- that can be used as a control segment. Part of the issue right now is that a proper lock or semaphore system is impossible since you only one source to write to- the simple act of writing a lock could potentially clear data that was added from another shard. 

 

> Will there eventually be a programmatic way to set this, rather than just the UI? This will be pretty important for automation.

> Not planned currently.

I, and pretty much everyone in #automation, is going to have a serious problem with this. I should not have to babysit my computer 24/7 to manage shard CPU, and I should be able to do things like push CPU to a shard where I am under attack from one where I am not. This is yet another situation where people who babysit the game will have a huge advantage over people who do not, and of course it means bots on private servers will not be able to use multiple shards. Please reconsider this, as I think it's a huge mistake that this isn't in the roadmap.

[ags131]

Personally, I've already worked out a way to handle the interShardSegment limits, its just going to be a pain and slow to sync any data.
The CLAIM issue means that there are going to be many players who don't have the ability to reach any claimable rooms in the other shards purely because they either 1) don't have a border room or 2) are too far away from a portal that leads to a claimable corner. 
I've already tried claiming a few other rooms, but have been unable to reach them with CLAIM due to TTL reduction.

[tedivm]

I do think the lack of an in game API for CPU control is more important than the segment issue, for what it's worth.

Some ideas came up in slack for alternatives- for instance, give all players a minimum of 30cpu on each shard, and then balance the rest of the CPU based on proportion of claimed rooms. However, I'm guessing that'll be too tightly coupled based off of the comments you had about segments.

[artch]

What if instead of all of the shards having automatic access to each other shard the code can request it (similar to requesting a foreign segment from another player)? This could potentially make it less tightly coupled.

This is an option, but it introduces more complexity to this mechanic. Let’s see how it goes with interShardSegment on the PTR. I suppose all what you have to implement is some mutual locking mechanism based on real time, not on game tick. Shouldn’t be very hard to come up with.

bots on private servers will not be able to use multiple shards

Private servers aren’t going to have shards support at all (at least officially).

Please reconsider this, as I think it’s a huge mistake that this isn’t in the roadmap.

Shards resources (not just CPU, but also GCL, Power Levels, Pixels, etc) are entities of higher order than those that your scripts are operating with. Giving an ability to allocate shard resources using the game API is similar to providing an ability of, say, changing your badge or email via the API. It is related to your account and should be set up by user, not by user’s script through an in-game mechanic. The shard itself knows nothing about the multi-shard environment it exists in, similarly to how it knows nothing about other user metadata like emails.

That being said, you can always use our web endpoints (they will get official Web API keys soon) to act on your behalf if needed.

[tedivm]

> Private servers aren’t going to have shards support at all (at least officially).

That's disappointing, but I'm pretty sure it'll end up with unofficial support at some point.

> Shards resources (not just CPU, but also GCL, Power Levels, Pixels, etc) are entities of higher order than those that your scripts are operating with. Giving an ability to allocate shard resources using the game API is similar to providing an ability of, say, changing your badge or email via the API. It is related to your account and should be set up by user, not by user’s script through an in-game mechanic. The shard itself knows nothing about the multi-shard environment it exists in, similarly to how it knows nothing about other user metadata like emails.

To me this just says it's a challenge, not that it's impossible. Perhaps a special "intent" which literally just makes a call against the API you're building would resolve this. The shard doesn't need to know about the multishard environment in any way that can't be managed with the shared segment (for instance, if there are ten shards and I'm on three of them I should be able to figure that out in my own code without needing assistance from the game engine, which also means I should be able to make the adjustment from the shard with all the info I have).

`setShardCPU({shard1: 50, shard2:30})` on shard1 triggers an intent for changing data. The engine sees that intent and (for performance reasons) dumps it in a separate queue, and that queue gets processed and literally just makes an HTTP request against the API you're already building. Sure there may be some lag between when the command in game is sent and the CPU gets allocated, but I think that's pretty fair.

[UnFleshedOne]

re: CPU -- it would be nice to be able to set this in script. To avoid excessive synchronization maybe have one shard be a CPU-master (whoever claims that status first) and be able to allocate CPU for own and all other shards. If script wants another shard to be master, current master must relinquish the status first.

re: segment sync -- there absolutely needs to be a way of synchronizing this, currently I don't see a way of writing into it from more than one shard. At the very least we need 2 independent segments as tedivm said.

[artch]

I meant it's more of an ideological restriction than technical. Such things should be changed via Web API, not via Game API.

[tedivm]

> I meant it's more of an ideological restriction than technical. Such things should be changed via Web API, not via Game API.

Why? Why should players who babysit their computers be given an advantage over players who focus on automation- what is the "ideological" reason for that?

[SteeleR]

Shards resources (not just CPU, but also GCL, Power Levels, Pixels, etc) are entities of higher order than those that your scripts are operating with. Giving an ability to allocate shard resources using the game API is similar to providing an ability of, say, changing your badge or email via the API. It is related to your account and should be set up by user, not by user’s script through an in-game mechanic. The shard itself knows nothing about the multi-shard environment it exists in, similarly to how it knows nothing about other user metadata like emails.
That being said, you can always use our web endpoints (they will get official Web API keys soon) to act on your behalf if needed.

 

Is it only me or this answer doesn't make any sense at all - "higher order" or not, your CPU availability matters to how your script can play the game - changing the badge, email address and so on doesn't have any significance to the game itself and can be done manually or not at all. And the "shard" (if it matters at all) knows about the multi-shard environment - it has portals to the other shards - it's not like these are separate servers, but something that, at least i think we are, trying to make work altogether.

So in the end you want us to write a 3rd party script to send data between the shard. If not already, at the near future, that mechanic will be used to do any heavy processing or preprocessing outside of the game thus bypassing the CPU limit and making it uneven for anyone who doesn't do that.

Again, don't know if it's only me, but these 2 points seem really really wrong and totally not in the direction i want the game to be heading 

[ags131]

For the record, I agree with artem on the seperation of user and engine data

@unfleshdone theres a few easy ways to handle interShardSegment, one I'm thinking of is just a hash list of timestamps when each shard updates, then only the lowest timestamp is allowed to write

[tedivm]

ags131 - can you provide actual reasons for that? What about the game play is improved by this distinction? 

[ags131]

@tedivm currently the engine is in 3 very distinct parts:

• storage (db/redis)
• backend (API endpoints + websocket)
• engine (in 3 sets of modules,main(master), runners (usercode), and processors(intents))

While backend and engine share access to storage, they do not have ANY direct ties, and engine only ever writes world state(Objects, memory, etc) within storage, never user state (Such as code, badge, email etc).

This is a good example of seperation of concerns in action, only the backend is allowed to modify userstate, and in limited cases such as for construction, submit intents for the engine to see. The engine never modifies user state and only sends back 'update' events for world state changes the backend subscribes to.

[artch]

I don’t see a way of writing into it from more than one shard. At the very least we need 2 independent segments as tedivm said.

There are a lot of ways of writing into it from any number of shards. It’s a coding challenge of how to implement it effectively. I recommend some reading about mutexes to get more info on the matter.

Why? Why should players who babysit their computers be given an advantage over players who focus on automation- what is the “ideological” reason for that?

Those who babysit their Screeps gameplay will be in advantage in much more serious ways than CPU allocation. It’s how online games work, no matter how much we wanted the opposite.

And the "shard" (if it matters at all) knows about the multi-shard environment - it has portals to the other shards - it's not like these are separate servers, but something that, at least i think we are, trying to make work altogether.

Portals to other shards are just string properties with the shard names. Database for shard1 contains a portal with shard0 string in it. No metadata from the shard0 database is fetched into the shard1 environment.

So in the end you want us to write a 3rd party script to send data between the shard.

You can transfer data between shards without any 3rd party tools. RawMemory.interShardSegment is the only API you would need for that.