4. Possible solution for multi language support?

Possible solution for multi language support?


  • Screeps and multiple programming languages

    I know this is a gripe for a lot of people that currently only JavaScript is available and that this is in the road map.
    Some people don't mind but I think this is something that should have priority above all.

    All the talking about the new isolated VM got me thinking, why not "ditch" the entire JavaScript engine and change the model to stin and stout streams.
    Sure that would mean that the current API would need a complete rewrite so that they offer interfaces that handle the parsing and serialization to the streams but on the bright side this allows the use of "any" executable as screeps script regardless in which language it was written...
    With the stin/out API any language can be used even if no native API is provided, this takes a huge burden off the devs. They can focus on the JavaScript interface API as provided default.

    There need to be serious precautions so, like a proper sandbox and I think the executables should be even compiled in-house with a safe compiler.

    But once you have a signed executable it should be even possible to make them available for dedicated servers as bots 🙂

    There would be the need for some architectural changes to the engine but I see the potential for a significant performance boost as well.

    So your thoughts on that?


  • @mrfaul said in Possible solution for multi language support?:

    stin/out

    I am assuming you are suggesting that we should use stdin / stdout for communication rather than the current API.

    This would require literally everyone to rework their code to use the new stdin / stdout structure. This is simply not going to happen, players would quit in their hundreds. Furthermore it would increase the barrier to entry for new players, they would need to write something to parse stdin and format stdout. I guess you could use JSON, but it creates an extra overhead and more potential bugs and some languages have better JSON support than others.

    The wasm functionality already allows you to use a wide range of languages (any language?) albeit with some limitations.

    It would also mean that those using relatively slower languages such as python would be disadvantaged over those using fast languages like c.

    There are other options that javascript, you can use truescript and I have heard of some people writing their code in python and using a transpiler.


  • Some (most?) of the public bot codebases are using TypeScript already, and there's a lot of folks using WASM transpiled stuff.

    JavaScript as a target is becoming more and more common due to it's use on browsers, so I just don't think this is an issue really versus the entrenched playerbase that a titanic shift in language support would throw out the window whole-cloth.


  • That change would be made in the background and not disturb the current scripts.
    Hence the conversation of the current API to a interface. Which means everything should work as expected besides the scripts with highly modified environments which really get to the guts.

    And no this doesn't add a barrier. CodingGame.com uses this system and it works great.
    Stin/out is basic functionality which every coder/scripter should know how to use.
    WASM on the other hand has some really nasty particularities which I would describe as barrier and you can't really call it finalized either.

    I admit that any interpreted language with slow interpreters would have a disadvantage if they are not cached. But there are tons of upsides for stand alone executables like proper thread control of a script.

    This game needs a bigger audience but there are a lot of potential "professional" players that are reluctant to try it because they don't like or can't be bothered with JavaScript...


  • @artch could please move it back?
    Under no angle is this a feature request.

  • Dev Team

    You're offering a major refactor in order to introduce some new capabilities, hence this is a feature request, I believe.

    As for the idea itself, it's very, very technically complicated matter. You probably don't see the entire picture of implementing separate properly isolated virtual machines in different languages working in one server-side cluster simultaneously and synchronously. CodingGame can't be compared, since their architecture is totally different, they don't have the persistent world aspect. As much as we want it, not going to happen at least in 2018, sorry.


  • I'm not talking about VMs in different languages, I'm talking about isolated threads.(which share a lot of similarly, but aren't a VM)
    And yes right now there are more pressing matters for you. I see this as discussion to achieve a optimal and sustainable solution for this and not like a feature request but that is a matter of interpretation 😉

    About the whole picture, yes I see it and the biggest bottle neck for screeps right now in my understanding is the database connection and its load.

    With the decupling from the scripts of the engine (which is essentially what the isolated VMs currently in beta are doing) you gain a huge advantage.
    This allows proper metrics with that load balancing becomes a breeze.
    Changing to the stin/out API would even further that making both engine and scripts complete separate entities which could in theory run in different universes as long they are able to communicate.
    Would offer a immense flexibility.

  • Dev Team

    How do you suggest running userland code in different languages on servers?

    isolated-vm has nothing to do with this matter by the way, it's just a replacement for Nodejs vm module.


  • Oh, crap didn't see your reply.
    And yes I'm talking about userland code but thinking about it why don't you offer automated WASM compilation for some of the most popular languages. This could help a lot.
    I really like screeps I just don't want to see it die of a lack of non-paying users.
    (Yes, I'm currently one them but my current "duck-taped script" isn't worth the money -_-)

  • Culture

    I've actually had a system like this working at one point, except it was using TCP rather than stdio, it would be a massive project to properly sandbox it, likely requiring containers for each user, which would be difficult to manage automatically. As @artch mentioned, CodingGaming is a completely different setup, they do a onetime container for each run, and unlike screeps, its not time sensitive so doesn't have to respond as quickly.

  • YP

    I thought about it some time ago and my idea was using redis as the interface to other runtimes. Maybe that's possible, but as @ags131 said permormance is key.. and the screeps dev are currently investing lots of time in optimizing the runtime ... I don't think new runtimes that might cause slower tick times are currently the right direction.

    1 Reply
  • Dev Team

    @w4rl0ck Exactly. Implementing a protocol between runtimes in different languages and processors is trivial, it can be done in a few days easily. Discussing its design here (whether to do it via stdin/stdout, Redis, TCP, etc) isn't worthwhile, it's not a complicated engineering task in any sense, it's a basic one. It is implementing proper runtimes for each language that is complicated, it would require months of R&D for each language, and we're really not ready to start this journey right now.


  • I don´t know if I was understood correctly I´m talking about to skip the "runtime environment" completely and to run standalone executables.
    They don´t even need necessarily a VM to do so, just super strict right management and bare bone libs.

    If you compile those "on side" it should be fairly easy to scan for malicious intends.
    There should be a number of such sandbox compiler already out there.

    But this doesn't help with interpreted languages like Lua those would still be in need of a proper runtime environment.

    1 Reply
  • Dev Team

    @mrfaul In order for your idea to work we'd need to run a separate container for every user to manage their resources usage, with instant startup, in an environment of thousands synchronized users. So my point still stands - the complexity of such a system makes it not even a major refactor, but a huge R&D project with uncertain results.