PTR Changelog 2016-09-29
-
> We actually consider setting all players buckets to 10,000 when a reset is detected, it is why I didn’t answer to your post yet, it’s still under consideration.
This would be amazing.
> What do you think is reasonably high?
If you set it to 1000 flags per GCL I would literally never use them all, and I think it would prevent abuse (especially when coupled with a nerf to the flag name limit). I would not want it to go below 800.
> Per flag, each of them, and thousands of them.
That's horrifying. You've totally got our support to nerf flagname abuse. To be honest I'm pretty sure most of us just assumed there was a limit in place so we never even tried. If you wouldn't mind truncating to 60 instead of 50 though I'd appreciate it.
-
Artem - if I cache object instances in global scope, they persist across my ticks within the same worker node. Would you be able to assign them to our own personal global context as a data container, then when we access flags the parsing cost is offset by anything that's already in our global cache on the worker? I'm not sure if that would be easier than trying to handle it outside of the secure execution context. Does that make things easier?
-
@N00bish this is a highly abusable mechanism..
This would be close to unlimited memory for everyone
-
> Per flag, each of them, and thousands of them.
Please name names and we'll solve your performance problems through ingame means.
> We actually consider setting all players buckets to 10,000 when a reset is detected, it is why I didn’t answer to your post yet, it’s still under consideration.
That sounds amazing. Consider this a vote in favor.
> Not true. Serialization mechanism is like 5% of this performance cost. 95% is
Flagobjects instantiation.Sure, but I can't optimize it in other ways. For example, you're deserializing every flag globally there. I can't override it to just be one room's flags.
Also, I am sure that creating a billion tiny strings with multiple splits is causing GC pressure.
-
If you set it to 1000 flags per GCL
1000 flags per GCL means someone with GCL 30 can create 30,000 flags. 30,000 JavaScript objects is… a lot.
Artem - if I cache object instances in global scope, they persist across my ticks within the same worker node. Would you be able to assign them to our own personal global context as a data container, then when we access flags the parsing cost is offset by anything that’s already in our global cache on the worker? I’m not sure if that would be easier than trying to handle it outside of the secure execution context. Does that make things easier?
There are different execution scopes within your single context, all objects are created in the scope with the current runtime data, and it becomes stale when the new data arrives. The global context stays the same, but the scope is changed. You can use the global context and cache something within it, but you can't get access to the previous tick execution scope.
-
Artem - you mean the number of flags? I'm all in favor of a 1,000 flags per GCL limit, and as far as I know for the people in this thread I have the most flags. There is a reasonable amount of flags, and name lengths, etc - I think we all agree. What we are all struggling with is how this was brought to our attention and the proposal seemed to do more harm than good.
So, in my mind, this is a solid proposal for now:
NO added parsing cost initially (This gives people more time to prepare and will allow you to collect metrics on how the other changes affect the servers)
Flag names are limited to 60 (since tedivm was asking for 60) characters
Flag count is limited to 1000 * gcl (this can be revisited later)
How does that sound to you? I completely agree that there needs to be some limits in place on flags, they are too cheap as they currently are - but the costs incurred by turning this on will be enormous to players and their code. If we can try the second two points first, then reevaluate where we are, I think we'd all feel better about discussing options for flag costs while not under the pressure of a deadline.
-
> 1000 flags per GCL means someone with GCL 30 can create 30,000 flags. 30,000 JavaScript objects is… a lot.
Then I think finding a way to allow lazy loading of the flags people actually want, rather than loading them all, or a way to cache them as n00bish suggests is the way to go. I rarely need all of the flags, so I shouldn't have to pay for all of the flags when I need one.
-
To re-iterate apparent issues server-side:
- Massive flag names
- Massive amounts
- Parsing/creating flags costs
_____________________________________________________________________________
Issues on player
- Sudden massive increase in CPU, which can't be altered
- Sudden change in strategies requiring massive reworks in big player code bases
- Some names might be cut off
_____________________________________________________________________________
Proposals
- Add possibility for own implementation to parse flags for the game-engine
- Give more CPU to circumvent the change, this doesn't punish players using the flag system
- Add a CAP on the max flags per GCL (proposed 800~1000 per GCL) I only need 40 flags per GCL personally
- Add separate Memory section for flags, max 50kb on flags, without CPU hits, or 2kb per gcl
- Slowly work to remove flags, and make everything memory based, but provide methods of "pinging" on the map.
-
"1000 flags per GCL"
@ All. Also take into consideration that from a scaling perspective we should assume everyone use all that is at its disposal.
(If not, it will only be an issue later)
So we are talking about 15 to 30 millions flags to serialize deserialize every tick.
This looks like a lot...
-
All. Also take into consideration that from a scaling perspective we should assume everyone use all that is at its disposal.
Exactly! This is where the CPU cost approach makes more sense.
Add separate Memory section for flags, max 50kb on flags, without CPU hits, or 2kb per gcl
What about 10,000 flags static limit without depending on GCL? Like the memory limit which is also static. When you outgrow it, you have to consider other more optimized ways to markup your room, using Memory probably.
-
Artem, to get a sense of scale for us, what is the current average of amount of flags per GCL?
GCL 1 = 20 flags
~
~
~
GCL 28 = 4000 flags
Something like that.
-
I'm OK with 10.000 flags, purely because it fits.with my code (only 1000 flags)
My statement for:
> max 50kb on flags, without CPU hits, or 2kb per GCL
was that, as a player, makes you think about the impact of flag usage early on, and you can see how much flags use. The maximum should still be reached at GCL 28, just like the CPU.
-
Since the new Flag() part is the problem, using a memoized getter to lazy initiate the flags could help with the CPU issue. Such as below:
serializedFlags.forEach(flagRoomData => {
var data = flagRoomData.data.split("|");
data.forEach(flagData => {
if(!flagData) {
return;
}
var info = flagData.split("~");
var id = 'flag_'+info[0];
Object.defineProperty(register._objects,id,{
get: function(){
delete register._objects[id]
return register._objects[id] = new globals.Flag(info[0], info[1], info[2], flagRoomData.room, info[3], info[4]);
}
})
})
});
-
I still think it's insane that you're creating each flag for each tick. I think exploring the approaches n00bish and others are talking about makes more sense, because most people are not accessing every flag every tick. I think the limits being put in place are trying to solve the wrong problem. Even if limits are put in place it makes sense to optimize this purely from a "lets not waste resources" standpoint. If this has to involve API changes that's okay as long as we can see them coming.
That being said I've now left the "anger" stage and entered the "bargaining" stage. How about 15k flags?
-
Artem, to get a sense of scale for us, what is the current average of amount of flags per GCL?
I don’t have such a calculation at hand, but here is another one:
1 percentile - 613 flags
2 percentile - 312 flags
5 percentile - 140 flags
10 percentile - 54 flags
20 percentile - 15 flags
50 percentile (median) - 8 flagsSince the new Flag() part is the problem, using a memoized getter to lazy initiate the flags could help with the CPU issue.
This is the same CPU-cost-based approach, since the getter will be called in the user CPU space. If you iterate through all of your flags, you will be paying for this instantiation in full every tick.
-
Another thought is that if you are limiting flags you could add another structure, called a Landmark, which is like a flag but is attached to the Room object instead of the game object.
The landmark structures can have the CPU cost you're talking about but with no limit in number. Since we can load them per room we can spread that cost out over multiple ticks ourselves by interacting with those landmarks once every X ticks or something.
This would make a limit on Flags much more palatable as it would still give people who do automated layouts a way to do so. A 10k limit with the promise of landmarks on the roadmap would work. People will naturally migrate to landmarks as they get closer to the flag limit.
-
For the record I never iterate over all the flags.
-
Yes, it would be the same cost IF the user iterated over them all, but, if the user say, called Game.flags.xyz, it should only initiate xyz and not all of them. It would only be in cases like lookAt or iterations that would trigger them all.
I am still all for the 50-60 char name limits, I'm just trying to help out on the related core issue
If I'm not mistaken, this would also end up user side, but, would give the user much more controll over how much it costs.
-
Another thought is that if you are limiting flags you could add another structure, called a Landmark, which is like a flag but is attached to the Room object instead of the game object.
The landmark structures can have the CPU cost you’re talking about but with no limit in number. Since we can load them per room we can spread that cost out over multiple ticks ourselves by interacting with those landmarks once every X ticks or something.
This would make a limit on Flags much more palatable as it would still give people who do automated layouts a way to do so. A 10k limit with the promise of landmarks on the roadmap would work. People will naturally migrate to landmarks as they get closer to the flag limit.
Well, now you are offering an alternative, and I like it! We have to think about it a bit more though, but it might be the path that leads to the solution we’re looking for.
-
It would be great if we had a way of iterating flags without causing the objects to instantiate. That would still disallow a few ways people tend to use them now, but would at least open up another area.
IE: I only use a handful of flags dynamically. Most other flags are objects that really don't need to be checked every tick. Now I have to stop using flags for everything but the periodic checks because I have to pay the entire deserialization cost by touching one of them. (I don't have many flags; just planning ahead).
IE: an iterator or array containing flag _names_, separate from a function like Game.getObjectById() that gives you the whole flag object back.
I understand that some people are checking properties of flags on every tick and there isn't much to be done there. Caching or object reuse would help, but it seems like our instances aren't very sticky so I'm not sure how that would play out in reality.
As an aside: It sounds like you're having a lot of systems issues in recent months. Is there any way at least one of us who does this professionally could help with a few hours of labor? I'm well familiar with being on the other side of something with performance issues and having people armchair-narrate how to fix them without a full understanding of the problem.
However I've also been pulled in many dozens of times by random (very large) companies and made pretty large impacts by dealing with easy low hanging fruit. If it helps smooth things out I'm sure at least someone would be open to it (I could throw a few hours or days at it).
Thanks
