Posts made by Xenofix

Bug in Replay here

First Bug: In this replay you can see a room under attack of an invader and the first tower is just finished the next tick. But instead the tower is invisible. However, the invisible tower can be clicked and inspected.

Second Bug: After death of the invader after t=16691971, clicking the back button tick by tick, the invader does not reappear in time.

Third Bug: The creep RangeHarvesterS24_6 who will be refilling the tower is missing in the replay until t=16692000, going backwards from there, you can see the tower and that he was refilling the tower after t=16691968

Maybe market system changes should be discussed in a separate threat. I'm also not a friend of order fees and I'm more a friend of transaction fees, but it would be too offtopic to deepen this topic now.

The current changes show what we'll expect but there is still something missing:

• The regional commodities need different resources, so their production cost varies a lot between different regions. What will be the market prices for these commodities?
• How much ops will OPERATE_FACTORY need for which effect duration? ( I know proposed is 100 ops for 1000 ticks, but they are "still considering other timings too" )

Maybe a non-general-purpose and multiple-resource structure needs capacityFor of e.g. {G:5000,energy:250000}.

capacity, getUsedCapacity() and getFreeCapacity() just makes sense for general-purpose (Storage, ...) and single-resource structures (Tower, Extension, ...). For all structures with mixed resources (powerSpawn, lab, nuke, ...) there needs to be more methods. Maybe getCapacityFor(resourceType), getUsedCapacityFor(resourceType), getFreeCapacityFor(resourceType). These can be implemented for general-purpose Store, too. So that the API for both types is the same.

I also welcome the API change but without deprecation of energy. If energy must be deprecated, then please make sure they are configurable to deactivate any warnings.

I think professional players already know how to overwrite prototypes and have aliases and getters for their needs already. Everyone can already model a unified store, and properties for energy and whatever. Keeping that in mind, the API change is mainly for newbies and those who cannot change prototypes using typescript or have found ugly ways around the inconsistent API which they like to simplify.

As a newbie I did often access creep.energy directly instead of creep.carry.energy. The same for storages, it was just so annoying that I always have to access either carry.energy or store.energy or energy when at first all I care about is just simple energy. The tower has no store, the creep has no energy and no store but carry instead; just not consistent. But I can make it consistent with prototypes. So yes, this change would not really help me, but it will help other newbies. One store for all kind of objects is a great change for the API.

About the database change: Please just choose the most efficient model.

I think every high-level player at 300 cpu cap and those who are looking forward to be one agrees with me that performance is more important than convenience. The database is hidden from us, so I prefer an efficient database model over a simple engine code. I have no idea if your proposed database change will have a positive or negative impact on overall performance, but I just fear that it's a negative impact because it looks more complex than before. In my eyes, the API and the database change are two different changes.

In the end these proposed convenience changes should meet the following requirements:

1. one store for every relevant object to allow a unified way to access energy and resources for convenience.
2. It shall not increase overall tick rate, better decrease it.
3. It shall not significantly decrease the free memory we can use
4. It shall not increase our cpu consumption, better decrease it.

If that's all met, then I believe every single player in the community will be happy about this change.

To me, carry is inconsistent. I would want to have it be named store instead. And a Store-object for general purpose stores is also very welcomed.

But for every other non-general-purpose structure this is only a little nice-to-have feature to simplify withdraw/transfer. If dealing with structure-specific code, I would prefer to access energy and other fields directly. If it's general purpose refiller/robbery code, then I would go for store.

Important to be is that such a change does not have a negative performance impact.

@artch said in Draft: Store prototype API:

• Two separate stores energyStore and ghodiumStore each with its own capacity.

With any other property than store it's not a simple unified API anymore. It has to be one store or no store, but we don't win anything if we wrap number fields in xxxStore-objects.

For developing I must disagree with making the energy and energyCapacity deprecated. If you know the structure you are using, then these fields are the preferred way to access these values. The function of store is just to simplify some code for generic transfer/withdraw.

I like the idea with store-properties returning 0. I think what @TuN9aN0 pointed out is an implementation detail here. I would be against Proxy. Store-objects for Extension or Tower could be using a Prototype which is just a view to the Structure without own memory, which has a property defined in the prototype to be const 0 for each resourceType and energy is a getter to the (hopefully not deprecated) energy field of structure.

I like the idea of a unified Store-object, but only for Structure, Terminal, Creep and PowerCreep. But for 'other structures' it would be nice to have a unified way to access resources.

Almost every structure depends on energy, so it's good to be never undefined. energy===0 is an ugly exception in store, but it's not ugly in 'other structures' like extensions, towers, spawns, etc.

But every player can change the prototypes to get the desired behaviour, so it's not a big deal.

I'm worried that a preinitialized store for 'other structures' just increases memory consumption a lot more, which is already an issue for some players.

@tun9an0 You get me wrong. It will affect me, so I am supporting to get more memory but I'm also against restricting memory more than necessary.

Your argument seems to be: You hit the memory limit with many rooms, so everyone else should also hit the memory limit with less rooms.

I'm against that for obvious reasons.

The memory limit is there because of physical restrictions. That has nothing to do with fairness. Everyone can go intershard to prevent memory problems. That is fair. While I agree that it would be optimal to just have enough cheap memory, after all nowadays 8 GB is nothing...., I assume that we won't get it because of economic reasons.

A new architecture can also solve these problems. But I am still and again very much against a stricter memory cap. The original topic was that memory is not enough for 50+ rooms, it was not that other people's memory is too much.

I think it's not documented that the memory in the UI shows you the sum of every Memory from each shard. But the limit is just shown per shard. But you can have 2 MB per shard. Maybe you still have some old unused Memory in shard 0. CPU is a little different. Maybe shard0 is busy with parsing and seriallization of its Memory, but if you respawned then the script shouldn't run in shard 0, so there should be no cpu cost for the Memory. Except something keeps the script in shard 0 active. Is maybe a creep or a power creep alive there?

No, that's the very wrong direction. Memory should not be another limiting factor. We are not writing code in C++ here where we have full control of the memory. If it's just limited because of limitations of the node servers then that's pity for you but that's not a reason to make memory a limiting factor for all the other players!

Btw. intersharding is more complex than staying in one shard, it also adds more overhead for each shard. The extra memory can be seen as an incentive for intersharding. Maybe you take the challenge? What about this idea: Every player gets 20+GCL*10 cpu without a cap, but can only assign max. 300 cpu to one shard. Would that be incentive enough to go intershard? Then players wouldn't hit the memory limit as they strive for intersharding empires to receive full cpu.

One last thing I would like to know from you Totalschaden: How much code can be executed per tick? Is the first line of your main.js executed? Is the first line in your loop executed? How much memory is used before you initialize any modules at the very first lines of your main.js and what's the memory consumption there? That first line should just be executed once per vm reset and the memory consumption there, before you initialize anything shows how much the engine really takes.

Summary of some memory optimizations:

1. Replace spatial and findCache by Map instead of {}. Likely increases performance, impact on memory consumption is unknown. Easy to implement.
2. Replace elements of spatial: new Array(2500) by Map. Maybe increases performance, impact on memory consumption is unknown. Easy to implement.
3. Lazy creation of spatial and subtracting the cpu cost from user cpu time. Can decrease engine cpu comsumption for bots which don't depend on look. Decreases memory consumption.
4. Compress terrain data from 2500 bytes down to 625 bytes. Decreases memory consumption, may increase cpu cost a little, breaking change for Room.Terrain.getRawBuffer, but that's already documented.
5. Replace pos of all RoomObjects by a property which wraps the new integer field __packedPos. into a RoomPosition. Just like the previous change of RoomPosition this decreases memory consumption a little and adds a little more cpu cost for accessing pos.
6. Write about memory problems with large empires in the documentation, so that players are prepared that intersharding is a 'must have' for large empires.

Every of these points can be done independently. @o4kapuk Which changes would you support?

That documentation says JavaScript arrays are not guaranteed to be dense; this depends on how the programmer chooses to use them.

I've tried it in screeps: {let x = new Array(1000000 * 4000);x[1000000 * 4000 - 1]=123;console.log(x.length);} works, but I shouldn't be able to reserve 32 GB of ram and use some of it. So it can only be a sparse array. I've tried the same in my Chrome console and it also works. It just fails if I enter new Array(big_number); because then the array is stringified and that string exhausts my memory.

new Array(2500) does not create a dense array with 2500 prefilled slots. new Array(2500) just creates a sparse array with a fixed length. But that doesn't mean a dense array is used internally. Maybe that's an implementation detail which differs with different engines, but in V8 it is a sparse array. It will be changed into a dense array if it has no holes, but spatial always has holes. So you can change it to [], but I believe it wouldn't help, because the array is already sparse. spatial and the findCache could be replaced by Map, that might help a little. The elements of findCache can also be replaced by Map, but I'm not sure if it helps to replace the elements of spatial with Map, but we could have a try.

You can't create the indices for look and find with zero overhead. Game is not created per room, it's created per player. The loop iterates over each object once and fills all the caches for all rooms at once. If you create them lazily, then you must iterate over all objects for each requested room. This is better in case of very rare uses of look, but it's worse if you use look in several rooms. So this problem must be fixed. One possible way to fix it is, to group the objects by room and use these grouped objects to create spatial lazily, maybe register.objectsByRoom is already such a structure. If look is used much, then this would need just a little more cpu, but if rarely used, then overall memory and engine cpu time is reduced.

One question still is (and I didn't find it in code, yet), if Game.cpu.getUsed() === 0 starts after Game is created or before Game is created. I suspect it's after Game is created. Because my bot needs at minimum only 0,03 ms to the beginning of loop. It's very unlikely that Game is created in just 0,03 ms with JavaScript for 23 rooms. That means, if parts of the cache will be created lazily, then they will drain the users cpu time instead of the engines cpu time. So that change would increase cpu consumption of all players who use any kind of look and they will start complaining about this change again. In other words: Such a change helps few and punishes many. Keeping that in mind, it would be very necessary to count how much cpu is needed to initialize the spatial cache lazily and subtract that time from the users cpu time, so that it comes with almost zero cost for the user again, just like now with pre-initialized spatial which is initialized outside the users cpu time.

And don't forget the other options which I mentioned before. Compressing terrain data can save memory at the cost of little cpu, compressing positions of RoomObjects can save memory at the cost of little cpu and allowing much temporary memory for each bot could even be a long term fix for the problem with no cpu cost. Maybe it's an option for huge players to switch back to the legacy model, so that they can access more memory.

Ok, after looking up some details. new Array(2500) would also create a sparse array. So it is no difference. Array.apply(null, Array(2500)) would create an efficient dense array. However, I think a sparse array was wanted here, because this spatial is used for look and you already said, it's usually far from full. I've read about sparse arrays and tested it: new Array(0x7FFFFFFF) should use up all your heap, but it doesn't. That means, I don't expect a positive impact by changing it to [].

I've spend some time now looking at the engines code for that lazy look/find-idea: What I see in code looks like look and lookFor are fully precached, it's recreated at every request and uses that spatial array, but it doesn't need to search for anything thanks to that sparse array. See addObjectToRegister. But look for area is a different story, it uses a different cache. find is cached and only for exitTiles lazily evaluated. The findCache is initialized when the Game-Object is constructed. See addObjectToFindCache. So calling find just reads directly from the prefilled cache, it's cached for maximum performance. But I'm sure it's not needed to precompute it for every type of find. Maybe a Map helps for the findCache.... which is already an object. In fact, every object that has roomNames as keys could use Map instead. But I don't know if Map uses less memory than object, I just know that it performs better. But I'm not sure if Map also performs better than a sparse array, after all a sparse array can be optimized for integer keys. However, conclusion: creating spatial on demand might be less performant if it's used on every room, but it's a win for those who rarely look. However, there is at least one thing which can only be seen by look and not by find, like spawning hostile creeps, but also here, look would only be used in some hostile rooms.

I think a big impact could have a compressed terrain data, which is in runtimeData.staticTerrainData. Terrain data is very big and the usage looks like it's an staticTerrainData[id][y*50+x] with values 0,1,2,3 each. I've found staticTerrainData in the driver. It's an Uint8Array. If every tile is compressed to 2 bits in that Uint8Array, then the size for terrain decreases from 2500 bytes + overhead to 625 bytes + overhead per room. That's something. Compressing terrain data will come with a performance cost.

However, I'm really not sure if such optimizations (as well as the packed RoomPosition fix) should be done without profiling how much memory is really used and where optimizations can be done.

Talking about packed RoomPositions... to pack pos of RoomObject could also save a little memory, likely even more than the last change of RoomPosition. An access of the property pos would unpack the RoomPosition... but since RoomPosition itself is already packed, it would effectively just wrap the integer in an instance of RoomPosition.

One last note: deft-code mentioned that lazy initialization of look could also decrease CPU. I think that depends if the cpu time for the creation Game counts to the users cpu or not. Means, if the user doesn't pay cpu for the creation of Game right now, then it's obvious what happens on lazy creation of spatial.

Edit: What about giving bots more memory? Obviously the problem here are temporary objects which exist only during one tick. The memory of ivm is persistent, but it doesn't need to be persistent for these temporary objects. So my idea is, to allow all bots to use more memory during his execution time, but at the end of the bots loop, free up the space used by temporary engine objects, garbage collect and if then the memory still exceeds the current ivm memory limit, then reset the ivm. This way we introduce a hard limit, which is what the ivm can persist, and a soft limit which is what the ivm can use while active.

I looked up the isolated-vm from laverdet. If more than 100% of memory is used, then that's already possible for a very short amount of time. So you would install the soft limit and hard limit here, so that it's just checked for the soft limit during execution and there is one explicit check for the hard limit after the temporary objects are released. Maybe that's also possible without changing the code of ivm, if the memory limit can be changed on the fly. Then you can set the higher soft limit before Game is initialized and the lower hard limit after Game is destructed.

Good idea. Instead of zoom I would wish width and height in tiles, so that it's guaranteed that your area of interest is completely visible on every display / window size.

Every bot should just get more memory instead of optimizing for less memory consumption at the cost of cpu cycles.

I agree, Respawn zones doesn't need any changes here. I would go further for newbie rooms: they should be reset as if they are new. The internal energyHarvested should also be reset to delay the first invader attack for newbies. Also partially harvested minerals should be replenished.

Ok, I agree that boosting by spawns would make the game more simple. I just wanted to take away the "breaking change" argument against MrFauls idea.

The memory hack is different. If something is changed there, then it's ok to break code, because RawMemory._parsed is not documented.

But RoomPosition was documented and everyone works with it. It was expected to be an intermediate class to work with, with simple fields, not checked properties. Such an object should not be packed, cpu is more important here. So this change was very unexpected.

The point is, this was just an example that breaking changes seem to be acceptable if not all players will be affected. And again, if labs are still working, then MrFauls idea wouldn't be a breaking change.

The argument of Smokeman was, that logistics is difficult and boosting would be too simple with this change. And while I kind of agree with this, I'm not sure if boosting should be so difficult. Yes, everyone who masters logistics doesn't want to loose their advantage, but the difficulty for new players is already raising with power creeps and regional resources. Maybe we should consider some ideas to decrease this difficulty.

Nontheless, we've had a breaking change with RoomPosition before. So a nonbreaking change would be, that a spawn can boost creeps and labs also can boost creeps.