The scenario @Tigga described ^ is exactly what I see. Here's a graph of my CPU (top) and heap + Global Age (bottom) for the last week or so. You can see that my CPU usage drops steadily the longer my global has been alive. I believe this is because some small leak slowly changes the heap characteristics such that GC transitions from almost always happening during my user code (incurring massive CPU cost) to almost never happening during my code. I know it's a difficult problem, and I certainly don't have a solution, but I dream of not having to deal with the volatility of GC. I'm confident that the immortal Game object changes will help immensely.

I'm not sure about the new-space old-space thing, as I mentioned in my comment, I just read a quick post about it. Seems like something that should be investigated if it hasn't already been.

My current experience is that I'm getting 5-10 CPU time limit reached messages an hour, and they are very clearly occurring when GC hits in the middle of my tick. For a period of about a week ending a couple days ago, it seemed that the 500 CPU/tick limit was relaxed. During that time, I was never getting errors, but was frequently seeing ticks that would take 700-800CPU. Given that I short circuit my code at 350CPU and average ~215CPU, this means that GC is taking 400-500CPU when it hits my code mid-tick.

I suspect these heap/GC issues have always been around and now they're just more obvious with IVM and the addition of get_heap_statistics(). I also suspect that if we're able to figure out what's going on, the servers as a whole will really benefit. Similar to @demawi, I see my used heap climb ~70MB ever tick before my code even runs. It only increases 3-5MB while my code is running.

So it turns out I had 4027 open orders on shard0, which seems to have been the primary contributor to slow getAllOrders performance. I've deleted them all and things seem to have returned to normal. There must've been a period of about 600 ticks where Game.market.orders was empty for both my code and the server, which both prompted and allowed me to create so many orders.

Code in my main loop:

module.exports.loop = function () {
console.log(JSON.stringify(Game.getObjectById(undefined)));
return;

Output on the console:

{"room":{"name":"E7S64","mode":"world","energyAvailable":0,"energyCapacityAvailable":0,"visual":{"roomName":"E7S64"}},"pos":{"x":23,"y":46,"roomName":"E7S64"},"id":"undefined","ticksToDecay":828,"hits":4700,"hitsMax":5000,"structureType":"road"}

Now that I'm typing this up, I've gone and looked at that room and there's actually two roads at that spot, one with an ID and one without.

So, I suppose it's possible Game.getObjectById() is functioning properly, and the real problem is that there are structures without IDs in various places in my rooms.

I spent awhile trying to track down an issue, and it turned out to be that Game.getObjectById(undefined) is returning a seemingly random object in one of my rooms (it changes), but with the id field undefined.

I have many calls in my code that will call Game.getObjectById(undefined) and expect to get null back, and that has never caused problems on the main server. No idea if this issue is related in any way to having a different DB.