3. PTR Changelog 2017-07-20: world shards

PTR Changelog 2017-07-20: world shards

  • Culture

    ags131 - can you provide actual reasons for that? What about the game play is improved by this distinction? 

  • Culture

    @tedivm currently the engine is in 3 very distinct parts:

    • storage (db/redis)
    • backend (API endpoints + websocket)
    • engine (in 3 sets of modules,main(master), runners (usercode), and processors(intents))

    While backend and engine share access to storage, they do not have ANY direct ties, and engine only ever writes world state(Objects, memory, etc) within storage, never user state (Such as code, badge, email etc).

    This is a good example of seperation of concerns in action, only the backend is allowed to modify userstate, and in limited cases such as for construction, submit intents for the engine to see. The engine never modifies user state and only sends back 'update' events for world state changes the backend subscribes to.

  • Dev Team

    I don’t see a way of writing into it from more than one shard. At the very least we need 2 independent segments as tedivm said.

    There are a lot of ways of writing into it from any number of shards. It’s a coding challenge of how to implement it effectively. I recommend some reading about mutexes to get more info on the matter.

    Why? Why should players who babysit their computers be given an advantage over players who focus on automation- what is the “ideological” reason for that?

    Those who babysit their Screeps gameplay will be in advantage in much more serious ways than CPU allocation. It’s how online games work, no matter how much we wanted the opposite.

    And the "shard" (if it matters at all) knows about the multi-shard environment - it has portals to the other shards - it's not like these are separate servers, but something that, at least i think we are, trying to make work altogether.

    Portals to other shards are just string properties with the shard names. Database for shard1 contains a portal with shard0 string in it. No metadata from the shard0 database is fetched into the shard1 environment.

    So in the end you want us to write a 3rd party script to send data between the shard.

    You can transfer data between shards without any 3rd party tools. RawMemory.interShardSegment is the only API you would need for that.

  • Culture

    > Those who babysit their Screeps gameplay will be in advantage in much more serious ways than CPU allocation. It’s how online games work, no matter how much we wanted the opposite.

     

    That doesn't justify making the situation worse. I still don't understand the "idealogical" reason for purposefully crippling players scripts here.


  • Not just that, but the stuff around making power creeps be UI only without programatic controllers just highlights this discrepancy. In a game about coding an independent AI (and thats what makes this game special, not any sort of micro) I feel it should always strive to not require the user's control.

    @artem

    Btw, are you considering having power creeps on a shard before they are brought to the main world? I think that would be a great way of implementing them as an opt-in testing experience. Just make power creeps not able to use shard portals and let people duke it out and find all the bugs there. Also give a limit to the amount of power you can use in the shard and a lot of the concerns about a level playing field fall away.


  • There are a lot of ways of writing into it from any number of shards. It’s a coding challenge of how to implement it effectively. I recommend some reading about mutexes to get more info on the matter.

     

    Ah, I guess the write itself is at least atomic and all shards will be presented with consistent state at all times (which one being undefined), so a lock is easy to write. I was imagining partial writes and other horrors. Nevermind then, sync is not a problem. 🙂

  • Culture

    @ags131 - since Artem already said the reasons were idealogical, not technological, your response doesn't seem relevant to the conversation. I asked what idealogical reasons you agree about here.

    I'm also confused how your separation would work with power creeps. Do you not think that players should be able to programmatically design power creeps? If you do think that should be allowed why are you making an exception for them but not for CPU time?

     

  • Culture

    @tedivm The idealogical reason (at least in my mind) is for seperation of concerns and design. Right now, with shards, there are theoretically infinite engine world states sharing one user state, if every engine decided to start modifying userstate, that breaks the whole concept of them being seperate. 

    I do agree that power creeps should be designable programatically, to me, they should be spawned exactly like a creep, but with a 'level', and point allocation  (Eg: `{ [POWER_KILL]: 4, [POWER_TELEPORT]:1 }` to make them entirely world state. The UI could still work by just submitting the UI customized intent similar to construction sites.

     

  • Culture

    My two cents on being able to control CPU distribution from game code:

    One of the things I would love to do with my AI is implement nomadism; code is always looking for new homes, never keeping a room for too long, exploring and finding new neighbors to interact with, whether the result is friendly trade, unbridled conquest, or crushing defeat. It sure would be nice to be able to have my code opportunistically expand into a shard without having to check on it myself, carefully evaluate the CPU / GCL allocated to each shard I'm currently on, and manually redistribute based on how much I think my code might want there.

  • Culture

    > Shards resources (not just CPU, but also GCL, Power Levels, Pixels, etc) are entities of higher order than those that your scripts are operating with. Giving an ability to allocate shard resources using the game API is similar to providing an ability of, say, changing your badge or email via the API. It is related to your account and should be set up by user, not by user’s script through an in-game mechanic. The shard itself knows nothing about the multi-shard environment it exists in, similarly to how it knows nothing about other user metadata like emails.

    I don't know how to say this less aggressively, so sorry for this, but: how can you be so wrong about your own game?

    Let's start with this:

    > Giving an ability to allocate shard resources using the game API is similar to providing an ability of, say, changing your badge or email via the API.

    No. My code can't see my email or badge at all. They do not affect my code's gameplay in the least. No matter what my badge looks like, or my email is set to, it has no effect whatsoever on the game. Allocating shard resources heavily affects the game.

    >  It is related to your account and should be set up by user, not by user’s script through an in-game mechanic. 

    No, it is related to my AI. My AI already knows about the amount of CPU allocated to it (currently, 100%, of course) and its own GCL.

    > The shard itself knows nothing about the multi-shard environment it exists in, similarly to how it knows nothing about other user metadata like emails.

    The shard itself is the server I'm playing on. It's not my AI. After the shattering, my AI will be a distributed system. It will exist on multiple shards. Crucially, it will be able to communicate between those other instances of itself using the shared segment. From here it is utterly trivial to compile a list of the shards I am running on - all I need is for a list of shards to be in the segment, and every tick my AI reads the list, checks its own shard name, and if it's not on the list, adds that shard to the list. Now I have a list of shards I'm running on. I know quite a bit about the multi-shard environment I exist in.

    I still don't know anything about emails, of course, because emails aren't related to the game in any way.

    I also can learn about new shards that I'm not running on simply by scouting portals. And that brings me to my main point:

     

    If you don't let us dynamically allocate CPU to different shards, then claiming a room on a new shard will always involve user intervention. If my code encounters a portal to a new shard, and it would like to expand across to the new shard, without an API the best my code can do is literally send me an email so I can log into a web UI and alter allocations. This is utterly ridiculous. It's completely against the spirit of this game. You have just caused emails to actually be a gameplay mechanic! My code has no way to get into a new shard without emailing me!

    A similar argument applies to power creeps, of course, but this is even more ridiculous. You give me a single segment to write to and manage mutexes in, claiming that it makes your AI code more interesting - essentially handing me a distsys problem to solve on a platter - but then you decide that I shouldn't be able to manage the distributed system that is my AI programmatically? What?

     

    I hope you reconsider. There is a significant difference between "Of course people who babysit the game will have an advantage, it's just a law of physics" and "We are specifically coding this in such a way that people who babysit the game will have an advantage because we will make it impossible to use this feature any other way."

  • Culture

    > @tedivm The idealogical reason (at least in my mind) is for seperation of concerns and design. Right now, with shards, there are theoretically infinite engine world states sharing one user state, if every engine decided to start modifying userstate, that breaks the whole concept of them being seperate. 

    That sounds like technical concerns, or maybe just your sense of "programming smell", which the admins have already dismissed as not being the reason for their decision.

    Even still I don't understand the concern. Users can modify the "shardSegment", and you want them to be able to create power creeps, so we're already breaking that divide in at least two places. Your method for having power creeps spawn is going to have to know how many power creeps are already spawned, or each shard will be able to spawn up to the user's power level. So the shards are going to have to communicate with the account system, or in some other way, already.

    Further once the shard allocation is set by the user in the UI it's going to have to get pushed to the shards anyways.

    I guess I don't see the concern with having the central account system be a microservice that other shards can talk to. I do agree that there is no reason for changing things like email or badges via the game API, but being able to communicate about power creeps seems like an important requirement and so does CPU management.

  • Culture

    Honestly, imagine if the market had been released in the opposite state as it was. Instead of there being no user-facing UI to place orders, that's the only way that we can place orders. Code can only look at the orders on the market, in order to actually buy from an order you'd have to go to the UI.

    After all, the market doesn't affect gameplay. Credits exist outside of gameplay, they're tied to your account. Having more or less credits doesn't affect your AI in any way, so it can't even see them. All your AI can do is look at the market and send you an email going "Hey, it'd be pretty cool if you bought X from this order", or "Hey, could you put up these excess resources as a sell order?"

     

    We'd have called you insane, and rightly so. That obviously isn't what we want out of a game that's about automation.

     

    That is exactly what is going on here.


  • Since we are getting pretty heated here, I think its probably good to point out that everyone is surprised and impressed that shards are in a testing state so soon. We know there will be plenty of development and tuning involved, but thank you for addressing this in a timely manner. 

     

    That said, I think people are getting concerned about the direction the game is going. Given that this is a game about automation (#factorio) and writing code, specifically limiting that is something that would of course irritate the people that are drawn to this game for that exact reason. 

    As for CPU, I think that it should 100% be controllable by code. 1 min example of what I as a user would want.
    Game.cpu.maxCPLimit = total account CPU
    Game.cpu.shardCPULimit = cpu the shard is allocated
    Game.cpu.setShardCPU = Set your cpu allocation on the shard. 
    Game.cpu.freeCPU = Amount of CPU allocated to your account not currently used in one of the shards.

    In order to increase the amount of cpu a shard has, you have to have unallocated CPU in your account. So to shift from shard to shard, you reduce your cpu limit in that shard, so for a number of ticks you are effectively missing account cpu as it sits unallocated. Once it registers that you have spare cpu you can increase your limit in another shard by up to that cpu amount. 
    That way with the memory segments you can request CPU from your other shards, but it also prevents frequent flipping around as it results in wasted spare cpu.

    I would have new cpu from gcl etc go to whatever shard has the highest existing allocation, unless the user sets a default shard.

  • CoPS

    I think that there should be some ability to control per-shard CPU allocation, otherwise certain designs of a program become limited by trivial manual interaction. Some examples:

    • When someone attacks a room, activities including scanning all rampart health values, recording and predicting the positions and actions of all hostile creeps, generating and updating e.g. Dijkstra maps for defender movement and running tower-attack allocation code all become important. Even more so if your defence involves active components e.g. intercepting incoming hostiles. If a shard has rooms come under attack, the user would need to be emailed about the attack, see the email, log in, and then allocate extra CPU to the shard to be able to defend effectively. On live, code automatically (for many of us) stops unimportant processes to save CPU to allow this to happen (e.g. lab code terminated when invaded to save CPU) - but this paradigm doesn't work if the CPU cap is low.
    • Some cacheable actions, like generating and caching paths or costmatrices for pathing (for combat, or remote mining, or whatever) require high amounts of CPU usage for a very small number of ticks. To allow these scheduled, but uncommon (once per 30,000 ticks or so) updates to occur, I would need to manually reallocate extra CPU to a shard, wait for the new caches to be written, then deallocate it to that shard.
    • An AI will not be able to automatically expand to new rooms (or automatically retaliate against attacks) if it cannot guarantee the CPU required to do so. A player who wants a new room would have to select the shard they want - in advance - and allocate the new CPU to it in order to allow such expansion to occur, before their code can then reliably and safely expand.

    I also have some questions about allocation:

    How will the bucket and 500CPU limit work with shards? On a single shard I can guarantee the order in which actions occur in order to use as much bucket *as required when required*. If I am on two shards and both need to hit the bucket in one tick (say path regeneration occurs) -

    1. Can I guarantee and control the order in which my bucket is used and accesses between shards?
    2. Can I control how much of the (500-usermaxCPU) bucket I have to spend in a tick goes to each shard?
    3. If one shard hard resets (1,000 cpu or whatever) - what happens to the other shard next tick?

      update: I have been informed each may get their own bucket. How would that interact with using a shard with a faster tick rate to do expensive computations that can be sent back to other shards? If I make a small room and allocate 1CPU more than it ever needs on a shard, I would thus have a essentially free bucket to use for computation.
  • Culture

    Something I just thought of.

     

    If our code doesn't know which shards have CPU allocated to them, how can my AI know which portals it should send creeps through? Should I just blindly send CLAIMers through portals hoping that my caretaker has allocated the appropriate amount of CPU to the other side? Am I (as the caretaker) supposed to manually poke values into memory indicating which shards I've allocated memory to?

     

    Am I going to have to literally write an out of game tool that scrapes the web UI for CPU allocation, pokes the values into memory, and then occasionally reads some segment of memory and pushes the desired allocation back to the UI? Because it's impossible to stop that from happening, and sufficiently motivated players will build this tool - and now you've just done two things:

    - Forced me to implement an out of game solution to an in-game problem

    - Put this feature out of reach of people who aren't able or willing to do the same

     

    To be clear, because I've been pretty hostile in this thread, I am blown away that you guys already have a working shard implementation at all. I'm just extremely frustrated that you are making it more difficult to write code that controls this programming game.

  • CoPS

    Anisoptera, while being a tad aggressive... does make me think of an issue

    If I send a creep through a portal into a new shard, do I have to have pre-allocated CPU for the creep to be able to move off the portal?

    With programmatic controls, that wouldn't be too hard, as I can read the destination and give it some minimal CPU amount.

  • Culture

    @artem Quick clarification, `Your GCL and CPU limit should be allocated...` Are we going to be allocating CPU directly? LIke 25 to shard0, 20 to shard1, 45 to shard2, etc, or just redistrubuting GCL points (Thus 1 point towards shard1 = GCL1 on that shard and 20CPU )

  • Culture

    @kotarou from docs: If this is an inter-shard portal, then this property contains an object with shard and room string properties. Exact coordinates are undetermined, the creep will appear at any free spot in the destination room.


  • looking great!

    One suggestion: I'd love it if the shard-portals were way more rare, and if they not only occurred in highway rooms. It would add strategic value to holding certain rooms and zones for alliances. It will also help in making shard populations evolve independently from each other (other players settling the same space). As it is now, new players in shard1 may get squashed often by their upstairs neighbors.

    For clarity, I'd also suggest using different naming, like "conduit" instead of "inter-shard portal", since their mechanics are very different.

  • Culture

    @ags131: what @kotarou was asking is, if I send a creep over to the other shard, do I need CPU allocated there to be able to control it?

    I can't imagine it working any other way. Which is my point. I can't send a creep through a portal I just discovered. I have to wait until my caretaker allocates CPU (or GCL) to that shard. And I don't have any way to know when that has happened (at least, no in-game method - of course my caretaker could have written some values to memory.)