3. PTR Changelog 2016-09-29

PTR Changelog 2016-09-29

  • Dev Team

    We all agree that kilobyte names are abuse. How much of the problem is caused by that? Can we start with that fix and see what happens afterwords?

    Yeah, I’m inclining to agree with that. 50 KB limit is more important than the CPU cost in short term, since it’s causing more problems. We cannot fix it today, as some players rely on this undocumented feature, it will break their code, so they have two weeks.

    But in long term, we have to think out the solution to free flags cost as well. Someone has 100 flags, another has 1000 flags, you have 7000 flags, and potentially someone can create 100000 flags. All of them should face their consequences, it’s just fair.

    And when I say "offer an alternative", I mean that you might be interested in joining this search for solution. If you're not, then criticizing won't help much.

  • Culture

    I have 5k flags, not 7k. I average 625 per "planned" room (claimed room that I will build up).

    Rather than a CPU hit I think a GCL flag limit makes more sense- as long as it's high (but reasonable), and we get enough time to actually work out this change (I'm not kidding when I say you should give us like 6 months notice here). If you're trying to prevent abuse giving people even 1000 flags per GCL should prevent the "make 100k flag" issue from ever occurring.

     

  • Culture

    You seem to be talking about 50KB overall for all flags, the limit we were refering to is 50 characters for flagname. (Which would be anywhere from 1 to 50000 flags at 50KB depending on name length)

    When you say kb sized names, is that refering to per-flag names or the entire serialized rooms.flags entry?

  • Dev Team

    and from the code you showed, I have to pay through a terribly inefficient method that I have no hope of optimizing.

    Not true. Serialization mechanism is like 5% of this performance cost. 95% is Flag objects instantiation.


  • Artem, if the flag instantiation is the cost, can you please review and comment on my suggestion posted earlier for re-using the instances across ticks?  Thank you.

  • Culture

    Also, I still think you should *temporarily* raise the bucket cap until the reset storm issue has resolved. It's a simple way that only allows people to use the CPU they've already banked (so overall shouldn't result in you having more load on systems) to get through the increase in global resets that happen then. 

    Looking over the numbers we've only had 2185 resets in the last 126,134 ticks, but the problem is the clustering of them doesn't give people a time to rebuild the bucket before the next. Raising the bucket seems like a clean solution to that.

  • Culture

    To me it looks like people use the flags for gameplay design that is missing. Some for the visual aspect, debugging (fast access memory), and planning constructions ahead. Maybe some more structureTypes need to be implemented to solve this. Like custom icon structures, blueprints that can turn into construction sites etc..

  • SUN

    Here is middle ground proposal.

    * Truncate the flag name at 50 characters in 2 weeks.
    Even if there are abuse from the consensual pov, nobody is abusing anything as long as they stick to the API.
    * extend the memory by 500 kb or better imho make it scale with gcl with e.g.  500kb +[100 kb per GCL.
    in 2/3 more weeks. (so 4/5 from now)
    But provide us with an API entry point that we can surcharge to optimize the flags implementation.
    (And make it per room, not global so we can factor in the room name :p)

  • Dev Team

    Rather than a CPU hit I think a GCL flag limit makes more sense- as long as it’s high (but reasonable), and we get enough time to actually work out this change (I’m not kidding when I say you should give us like 6 months notice here). If you’re trying to prevent abuse giving people even 1000 flags per GCL should prevent the “make 100k flag” issue from ever occurring.

    What do you think is reasonably high?

    When you say kb sized names, is that refering to per-flag names or the entire serialized rooms.flags entry?

    Per flag, each of them, and thousands of them.

    Artem, if the flag instantiation is the cost, can you please review and comment on my suggestion posted earlier for re-using the instances across ticks? Thank you.

    Oops, I was sure I answered to that, sorry. The hard part in your solution is that all game objects are linked to the execution scope of the current tick. There will be all sorts of weird issues if we try to cache real object instances across ticks. It’s an option, but hard to implement.

    Looking over the numbers we’ve only had 2185 resets in the last 126,134 ticks, but the problem is the clustering of them doesn’t give people a time to rebuild the bucket before the next. Raising the bucket seems like a clean solution to that.

    We actually consider setting all players buckets to 10,000 when a reset is detected, it is why I didn’t answer to your post yet, it’s still under consideration.

  • Culture

    >  We actually consider setting all players buckets to 10,000 when a reset is detected, it is why I didn’t answer to your post yet, it’s still under consideration.

    This would be amazing.

    > What do you think is reasonably high?

    If you set it to 1000 flags per GCL I would literally never use them all, and I think it would prevent abuse (especially when coupled with a nerf to the flag name limit). I would not want it to go below 800. 

    > Per flag, each of them, and thousands of them.

    That's horrifying. You've totally got our support to nerf flagname abuse. To be honest I'm pretty sure most of us just assumed there was a limit in place so we never even tried. If you wouldn't mind truncating to 60 instead of 50 though I'd appreciate it.


  • Artem - if I cache object instances in global scope, they persist across my ticks within the same worker node.  Would you be able to assign them to our own personal global context as a data container, then when we access flags the parsing cost is offset by anything that's already in our global cache on the worker?  I'm not sure if that would be easier than trying to handle it outside of the secure execution context.  Does that make things easier?

  • SUN

    @N00bish  this is a highly abusable mechanism..
    This would be close to unlimited memory for everyone

  • Culture

    > Per flag, each of them, and thousands of them.

    Please name names and we'll solve your performance problems through ingame means. 🙂

    > We actually consider setting all players buckets to 10,000 when a reset is detected, it is why I didn’t answer to your post yet, it’s still under consideration.

    That sounds amazing. Consider this a vote in favor.

    > Not true. Serialization mechanism is like 5% of this performance cost. 95% is Flag objects instantiation.

    Sure, but I can't optimize it in other ways. For example, you're deserializing every flag globally there. I can't override it to just be one room's flags.

    Also, I am sure that creating a billion tiny strings with multiple splits is causing GC pressure.

  • Dev Team

    If you set it to 1000 flags per GCL

    1000 flags per GCL means someone with GCL 30 can create 30,000 flags. 30,000 JavaScript objects is… a lot.

    Artem - if I cache object instances in global scope, they persist across my ticks within the same worker node. Would you be able to assign them to our own personal global context as a data container, then when we access flags the parsing cost is offset by anything that’s already in our global cache on the worker? I’m not sure if that would be easier than trying to handle it outside of the secure execution context. Does that make things easier?

    There are different execution scopes within your single context, all objects are created in the scope with the current runtime data, and it becomes stale when the new data arrives. The global context stays the same, but the scope is changed. You can use the global context and cache something within it, but you can't get access to the previous tick execution scope.


  • Artem - you mean the number of flags?  I'm all in favor of a 1,000 flags per GCL limit, and as far as I know for the people in this thread I have the most flags.  There is a reasonable amount of flags, and name lengths, etc - I think we all agree.  What we are all struggling with is how this was brought to our attention and the proposal seemed to do more harm than good.

    So, in my mind, this is a solid proposal for now:

    NO added parsing cost initially (This gives people more time to prepare and will allow you to collect metrics on how the other changes affect the servers)

    Flag names are limited to 60 (since tedivm was asking for 60) characters

    Flag count is limited to 1000 * gcl (this can be revisited later)

     

    How does that sound to you?  I completely agree that there needs to be some limits in place on flags, they are too cheap as they currently are - but the costs incurred by turning this on will be enormous to players and their code.  If we can try the second two points first, then reevaluate where we are, I think we'd all feel better about discussing options for flag costs while not under the pressure of a deadline.

  • Culture

    > 1000 flags per GCL means someone with GCL 30 can create 30,000 flags. 30,000 JavaScript objects is… a lot.

    Then I think finding a way to allow lazy loading of the flags people actually want, rather than loading them all, or a way to cache them as n00bish suggests is the way to go. I rarely need all of the flags, so I shouldn't have to pay for all of the flags when I need one.

  • Culture

     To re-iterate apparent issues server-side:

    1. Massive flag names 
    2. Massive amounts 
    3. Parsing/creating flags costs

     _____________________________________________________________________________

    Issues on player

    1. Sudden massive increase in CPU, which can't be altered
    2. Sudden change in strategies requiring massive reworks in big player code bases
    3. Some names might be cut off

    _____________________________________________________________________________

     

    Proposals

    1. Add possibility for own implementation to parse flags for the game-engine
    2. Give more CPU to circumvent the change, this doesn't punish players using the flag system
    3. Add a CAP on the max flags per GCL (proposed 800~1000 per GCL) I only need 40 flags per GCL personally
    4. Add separate Memory section for flags, max 50kb on flags, without CPU hits, or 2kb per gcl
    5. Slowly work to remove flags, and make everything memory based, but provide methods of "pinging" on the map.
  • SUN

    "1000 flags per GCL"
    @ All. Also take into consideration that from a scaling perspective we should assume everyone use all that is at its disposal.
    (If not, it will only be an issue later)

    So we are talking about 15 to 30 millions flags to serialize deserialize every tick.
    This looks like a lot...

  • Dev Team

    All. Also take into consideration that from a scaling perspective we should assume everyone use all that is at its disposal.

    Exactly! This is where the CPU cost approach makes more sense.

    Add separate Memory section for flags, max 50kb on flags, without CPU hits, or 2kb per gcl

    What about 10,000 flags static limit without depending on GCL? Like the memory limit which is also static. When you outgrow it, you have to consider other more optimized ways to markup your room, using Memory probably.

  • Culture

    Artem, to get a sense of scale for us, what is the current average of amount of flags per GCL?

    GCL 1 = 20 flags

    ~

    ~

    ~

    GCL 28 = 4000 flags

     

    Something like that.