@W4rl0ck: It's very easy to limit API access to clients only if you control the clients. Hell, it's a browser based game, with a great community. Set a cookie, or better yet a header, have the web server look for it and tell the player base to back off.  It doesn't need to be super secure, hell, if we were told to stop using it outside the clients most everyone would comply. 

"we do know how to fix the problem - the world sharding change is the solution"

That's kind of my point. Tweaking isn't going to do it. A big change is needed.

As for:

"Regarding the visibility thing - don't forget about observers,"

I would think that it's rare compared to the number of "in room" read and writes

"globally synced operations like market and terminals,"

That can't be taking up that much DB time. Limit it to one market call per tick, Terminals are already limited in such a way with intents. (essentially)

" external APIs fetching game state (we're going to introduce API keys in the future)."

Turn it off for now. I know that makes our pretty graphs go away, but that's a fair trade for better tick times/stability. It can be brought back when you flesh out the API key stuff. Lots of games and other companies do that when a secondary part like the "Unofficial" API gets to be to burdensome. If API really is the problem then ditch it. We all know it is "Unofficial" anyway. 

"Getting rid of the persistent database will complicate things to nearly unmanageable state."

Maybe, but maybe not. I can't see your mongoDB database engine, but if it's like the opensource one, then you could come up with a way to just not write to the DB. Yes it would add some work, but I'm not sure that it would add that much work. A Pure in memory MongoDB that flushes to disk every 100 ticks or so could be an easy start. 

"You basically propose to develop our own distributed database management system, I don't think such a task can be managed by a team of 2 developers." 

IDK, two developers can do quite a bit  But yes, that is kind of my point. It seems like your going "what can we do in budget (meaning time and money not just money), but what I am saying is that you might not be able to solve the problem "in budget". It may be much bigger then that. To me (not knowing anything internal about the team) It's a big enough problem that all future dev stops on any thing that isn't this problem/solution. For example Stability, Tick rates, sharding is a "Must Have" while GUI, new clients, API keys, etc etc. all become "Nice to Haves".

But as I said in my last post, it doesn't matter. The fact that you guys are going down a path, regardless of the path, is light at the end of a tunnel. It may be a long, twisty, narrow, tunnel, but look there's hope, and I think that's the important take away. "It's being worked on"

I hate to say it but it sounds more like you don't know how to fix the problem so your goring to throw solutions at it till it sticks. In other words that the entire situation is fundamentally flawed and perhaps a rewrite is needed and not just a "muck with DB settings" approach.  It seems that "sharding" is your rewrite. 

However I worry that you have not solved the fundamental problem. You hinted at it in one of your last posts. X reads/writes = suck. Sharding may make that less common (cause your doing less read/writes per shard) but the limit still exists. Why not focus on fixing "X".

Actions don't need to be written to the database all that frequently. They can be stored in memory then flushed to the database. Maybe once every 1000 ticks you can flush to the database. Yes that means a crash is an auto roll back, but just don't crash (gotta love that one). 

As to world interactions, what does it matter. Rarely in all those read writes are players interaction with more then 1-2 other players. Some maybe 10-20 players. But some kind of status propagation based on visibility could fix that. I mean what really needs to be passed, not much changes per tick "normally". When an attack happens (or when two players creeps are in the same room) more information needs to be passed around, but that's not "often" compared to a creep in it's own rooms.

In other words I see

• State info stored in memory most ticks
• That state shared to people that have visibility to that room and otherwise ignored
• state is flushed to database  infrequently.

Now this bounds your hardware and database and what not to number of rooms and not number of players, or what those players are doing. It also removes the "slow part" from the "fast part". If processing isn't a problem and just database-ing is a problem then just don't database.

ALL THAT SAID

I am very happy that you guys are doing something. At the current rate, you won't have a game in 6 months. These changes are the first glimpse of light at the end of that tunnel. Tick speeds per shard are going to be a huge issue. They need to be "equal" somehow. The screeps world needs to be "one world" somehow. But the fact that your making progress in some direction, is a great thing. It needs to happen. It may cause a "burp" while the player base adjusts, but don't do "something" and you won't have a player base. If your not growing your shrinking. Staying still isn't really an option.

Well, I'd suppose you'd have to write one, but two people that know (or a large group that knows) to look at a certain segment could use that as a place to store "messages" that other people that know where to look, could read and process.

One thing is for sure, I don't need yet another source for spam. So no Game.notify("coteyr", "my message here") or anything like that.

you can "in game message" now with public memory segments.

Last time I used mongoDB (ditched for the same reason) it was really good as storing text gibberish, and rubbish for anything else. Maybe mongo is good because of the serialized memory and the code storage. But it was really quite slow on normal stuff back when I used it last.

This could be very exciting. I may even go back into, "Here just take my money" mode. If the new VM changes actually give me control over my own CPU usage and the tick rate becomes "normal" again.

@starwar15432 really?

https://www.youtube.com/watch?v=N_dUmDBfp6k

What do they teach you in school these days?

@Atavus It's a minor gripe, but I just don't like alliances. I don't have an alternative. The largest thing I don't like about them is they create a "red v blue" situation. If I go to poke a bear, I first make sure it's smaller then me, then I poke. Alliances mean the bear you just poked suddenly gets 500 friends. That great for the pokie, but the only "counter" to that is to bring my 500 friends. Which leads to a EVE like red v blue situation. Specially with the timers and cool-downs and tick lengths. 

It's been years sense I played EVE but all I see is a TCU and several POSes every time the "safe mode" kicks in.  Alliances just aren't my thing, but, as I said, minor gripe, and totally on my end. What would be nice is an in game representation of an alliance. At least that way I could make sure I was poking a small lonely bear and not 500 unknown bears.

As to supportive, I try to be where I can, but IMHO the game play is just purely broken. Not in the teenager, "match making sucks how come this noob is on my team broken", but fundamentally flawed. It's not that you have to account for CPU that part of the game, it's that you CAN NOT account for CPU. For no reason at all  that you the player can control, you not only use more CPU this tick but are also blocked for 5 ticks.  That's a big deal in a fight. It's even a big deal in a game where the SCOREBOARD's margins come down to 0.00001% differences in resource collection. 

Now again, it's not that I care, like a couple of days ago when the servers are down. That happens. Everyone sucks the same. But with the CPU issue it seems to be isolated to "nodes". Every once in a while your code will get stuck on a "bad" "node" and then no matter what you do, your screwed. 

Having to "work around" the issue is not fun, mostly because the work around doesn't exist. You simply have to accept that though no fault of your own, your code just vomits this tick, and you get punished for it.

Currently there is no way to "win", there may be a lot of objectives  to aim for, but you simply can not reach them. You have the score board and it's metrics, except when you get punished DAILY for something outside of your control and the top spots are all very close in terms of efficiency, your not going to be doing things there. Mostly, because the "punishment" is not handed out to everyone equally, it's "random".

There are PVP aspects, which you may or may not be able to win, but in a fight when your actually using 80-90% of your CPU your can be damn sure your going to get locked out. How will that effect the fight. Your opponent get's 5 free hits? Well if your HEAL tanking a tower shot there goes your expensive, possibly boosted creep. 

Maybe you just want to strive for efficiency, except there again,  looking at the "top players" a whacking or two with the ban hammer is all the difference that exists between the top 10.

So once again you have this, thing (CPU Usage) that is core to your game play (most of the game is actually managing CPU v.s. your goals) and you get no control over it. It would be like me saying Tedvm and ags131 enter into a race from Florida to London. But I will dish out the gas. You can pick out anything else you want, but you must use a car, boat, or plane, and I will give you the fuel. They ask how much fuel, I will give you 1 gallon of fuel every 30 mins. They start the race, but this I give ags131 1 gallon of gas and tedvm .8 gallons. Then for some unknown reason I give Tedvm 1 gammon of gas but tell ags131 he can have any for 2.5 hours. Later I give Tedvm .5 gallons and tell ags131 he gets no gas again for 2.5 hours.  When asked why, I just shrug and say it's the way gas works, and hand them both an empty gas can, but I continue to dish out gas at seeming random amounts.

I hope that makes sense.

And before we go down the path of "it's not that bad" YES IT IS!!!! That's my complaint. Sometimes things do exactly like they should and that code uses a tiny amount of CPU, but other times, for no reason that I have any control over it doesn't and it "uses" a  ton of CPU.