[MOD Concept] Improving gaming performance by offloading CPU-intensive processing scripts

[waadaa85]

Hello Community,

Motivation
==========
With the base game, most X4 players have experienced at some point, and at various levels, performance issues when a game is consuming most of the resources available on their rig. At the other end, brilliant modders have pushed the game boundaries to new heights, introducing new features, opening new perspectives, new game-plays and tactics, improving AI capabilities, etc. Even total conversion mods.

Frustration
===========
Alas, my current game has reached a plateau. Since my first game start, I aimed to reach a level where I could think of building an empire. I've been able to eradicate Xenon (it was not easy but it's done) and I currently own all their sectors. Same for the Kaaks, I bottled them in such a way they are now confined in their usual spawning spots and reduced to dust the minute a installation is spawning by one of my local Asgard Battle groups that permanently patrol each of these sectors. Actually I have so many of these battle groups (One Asgard with 6 Osaka configured as "FlakPz" escort) that I don't know how much exactly I have... with 5-6 "Station-buster" specialized battle groups with again one Asgard accompanied by 6 Sym Destroyers (all-ARG plasma turrets). Money is not a problem.

So peace at last in all sectors of the galaxy. I'm at > +25 relations with all factions (+30 with VIG, which I thought it would give me the option of buying another Astrid, but no). Only ALI is stuck at +18.

No, most concerning is what I'm experiencing now (besides low performance when using left or right menus and filters) is the trading levels are (from other factions I means) anemic at best. Whole sectors, once flourishing are now literally regressing... Even Argon Prime now have only 5 stations left... (from >15 they had initially). Their stations are not destroyed by some belligerent faction sneaking while I'm not looking, no. They are simply withering... And, after some time, disappears completely. Even some TEL stations with which I had constant trade exchanges to fuel my local terraforming efforts had gone "out of business" and disappeared.

Hopes
=====
I perused existing economy+finances mods from modders like mysterial, kungfury, ilimuno and others, seeking a way to improve the "I" in the AI without killing my gaming PC's few remaining resources with yet more lengthy XML scripts. In hope to have an open and more dynamic economic model that would be able to generate realistic trading activities, yet aligned with the current situation and chain of events happening in the universe. Given the fact, X4 AI rules are mostly XML scripts (MD and cues) fed to a single-threaded process tied to a 5.1 lua engine, improving the AI by extra-feeding the engine with just bigger, more elaborated scripts, can just translate into a reduced processing capability... That is not good...

Inspiration
=========
Besides X4, I'm a long time fan of Arma2 and Arma3 Series. This franchise introduced with Arma2 (circa 2008) a clever way to reduce resources contentions on a gaming PC (graphics, CPU, disks, memory) through a mechanism called "Headless client". It is still in operation with Arma3 and will most likely (I hope) be refactored to work along with the upcoming new flagship Arma4 (currently sandboxed as "Arma Reforger") as resources contentions still exists and will continue to exists in the future... Even with Intel i9 ddr5 memory, SSD drive...). "How it relates with X4" you may say? Please read along.

The main objective of such "Headless Client" is to offload AI calculations away from the graphics rendering device ie. the gaming PC hosting the game. Which currently have to renders all the 3D graphics, and manage the game's world/universe dynamics. So in essence, the gaming PC keeps the intensive hard work of graphics rendering and delegate chunks of AI-related calculations and evolving universe' dynamics processing to 1..n headless client instance(s). Those headless clients are running on resources provided by other interconnected device(s), such as laptops, family PC, LAN server, locally hosted VMs, etc.). In short, an "headless client" is meshed with the central game PC as a slave processing node.

Multiple individual processing nodes can be tied to a game instance for additional power and provide scalability for multi-player games. MP not being a use-case in X4, but extra processing would indeed be very welcome to support late game phases, or address extensive wars that develop at some point between factions (ARG Vs TER, or Paranid civil war(s), etc.) Some rules are enforced though, in order to preserve the game instance's integrity and reduce potential risks of cheating. An headless client is bounded to one and only one game instance, which doesn't allow arbitrary connections from headless clients once a game is started/running.

Concept
=======
From an technical infrastructure PoV, the idea I'm currently looking at is to deploy (on one LAN server connected with the main gaming PC) a semi-autonomous lua container able to mimic the lua engine running on the game PC. Which could process stuff instead of letting the central engine doing the processing all on its own. Things that could be offloaded and processed asynchronously from the central engine, could be :
- additional/improved MD and cue scripts providing better AI trading choices and rules;
- Better economic ask-bid/availability-scarcity models and calculations for trading items;
- Better AI ships captains routes determinations and waypoints calculations;
- Better Trade items management by Stations managers and resolve once and for all problem of having two different type of solids (or liquids) harvested by miners on one station...
- AI factions station managers could have better economical stances and situational awareness over geopolitical situations among factions to do better choices in sector development and stations improvements (Can't get out of my head Mysterial mods series to fulfill that requirement );
- comportment and social interactions, (insert 1..n kuertee mods you like here as examples )
- non-ui QoL scripts improvements (kind of a long shot this one...) but anyway lets think having an externalized dynamic filter managed at the sector-level instead of being at the universe level as currently. Recalculate and regenerate dynamically any given sector's short-list of ships for a specific zoom level can hold some gain in performance from the ui.
- etc. --> your suggestions are very welcome

If these objectives are too ambitious, my prime objective is actually to offload only some specific parts of already existing cues and MDs scripts from the base game to another node. Such as good number of the trading scripts or some other well-scoped/defined set of scripts. even that would be beneficial for the gaming PC to use the spared resources to process something else. Sounds nice so far? keep on digging...

Communications between a remote lua container and the base game
===========================================================
First, I want to say that I'm far of being a modder. I'm only starting to begin to comprehend how the game is working internally. but I'm a IT guy specialized in infrastructures and application/components & software integration. So while perusing those very interesting mods and all, I saw a opportunity I wish to share. So better, more knowledgeable people are welcome to pitch in, provide guidance and make suggestion. And if after some discussions it is clear it is not feasible, we simply close the thread and move along. No hard feelings! And will all probably wait for X5, with the hope that future X instalment architecture will be modularised/containerised enough to enable asynch, mesh-enabled, inter-processes collaboration and inter-networking.

Any half-decent PC nowadays can run docker-desktop. So crafting a container as a computing node with the very same lua 5.1 engine provided with the game should be somewhat simple (given I know the intricacies and dependencies between the lua engine, the game objects and how they are managed. Now the question: How communications between the game PC and a given processing node can be handled?

I found two potential answers to that one that I want to share with the community and have your take. Every people in X4 universe that ever took a look at X4 mods and modding has most likely already came across over the truly amazing work of SirNuke python pipe server. I also found an alternate communication handling interesting approach with Alias5's REST API & server POC (both are on github). Unfortunately though, I never had the time to fully dig into that work so far. If someone have that perspective and able to share, I'll happy to read your thoughts in the comments.

I started experimenting with the Python pipe server, which currently uses local pipes. I looked into the code 3-4 months ago and imho only some minor adjustments to the existing code should be required to support named pipes over a network. The capability to read a simple config file is already there, so we can simply add necessary network parameters to the python server prior launch. Parameters like the IP address of the gaming PC running the game, additional IP(s)- one for each headless client(s) members of the mesh network to communicate with, node prioritization, etc. The intent is to package that meshed version python pipe server in the same container as the lua engine, which shouldn't be too difficult either. And because we don't want additional python pipe server(s) to cannibalize already scarce resources on the gaming PC. It would defeat the whole purpose.

BUUUTT!
=======
The hard part is beginning here: According to SirNuke's extensive documentation (kudos to this guy really), the way the communication between the lua engine residing on the gaming PC and the outside world is through an existing dll (debug-ui.dll if my memory is correct). So what I understood so far (please correct me if I'm wrong) is that hack used between the engine and the dll has significant limitations and prone to contentions. So asynchronous communications through that dll toward the Lua Engine wont work with multiple channels/processing nodes. It can still work from a single source, so it is still manageable (under certain conditions).

The best option would be for the dev team to provide a dedicated open dll (and its code) to bridge communications in and out with the central engine. I doubt they have the time and resources to do that with the upcoming sortie of KE DLC. So moving on.

An alternative would be to put a local "master" python server on the gaming PC, which would act as an orchestration tier and funnel both egress and ingress communications with the various processing nodes. And be "The One" to the lua engine able to talk through the dll bridge. That way, it is possible to keep contention risks to a minimum while the python master server handle multiples asynchronous ingress communications, queuing them (if necessary) into an XML stream in an orderly fashion. Because the core of the Pipe server is written in Python, extending the original server to support these new tasks and functions should also fairly easy to do. The drawback is : to offload chunks of AI works and calculations to somewhere else, we have to run another process locally. So for that local master server, due care in its coding will be paramount to keep its resources consumption at its lowest. it would be preferable not to containerized that master server, to avoid the installation of additional stuff, like docker-desktop or any supporting runtime(s).

I'm currently in the experimentation phase to assess the technical feasibility of that aspect. SirNuke if you're reading this, I have some questions and I think only you can shed a light over some implementation intricacies and also identify potential pitfalls that you may see. Thanks in advance

Way ahead
=========
Imho Egress communications are simpler to handle (from the game engine to an external processing node), as the various functions are triggered by cues and MD scripts. I have another concept, this time within the game, to ties such exchanges with game's events, Universe dynamics and trigger in a timely manner these egress communications. Luckily, the aspects of the game I flagged as "candidates" for offloading are, 1) indeed process-intensive (of course); but also, 2) asynchronous in nature. So the game universe can perfectly live without having to wait for an answer from a processing node. That topic will be the subject of my next post.

thanks all,

[SharpEars]

There is a lot in the above post. Fundamentally, X4 needs to be enhanced, if it at all possible, to leverage more (physical) CPU cores when such cores are available on a user's gaming host. Writing multi-threaded code is a difficult endeavor and not everything can be parallelized, but in its present state, X4 simply does not appear to leverage (well) the CPU resources available on a host.

Those of us who are fortunate enough to own PC's with >=16 physical cores and able to properly profile the game's usage of said cores (i.e., not via Task Manager or any of the other built-in nonsense on Windows - properly!) can see just how few cores actually get used. This is true even when the game completely bogs down. The (relatively) small number of busy threads indicates very little parallelization taking place and subsequent utilization of (all or even most of) the CPU resources that are available. This may be an engine limitation or just a "simpler" implementation of the game mechanics on top of the engine. It is impossible to say, since we are not privy to the inner working of the engine, its limitations, its reliance on and connection with Lua, and just how multi-threaded and/or concurrency aware the engine itself fundamentally is.

Speaking somewhat abstractly (i.e., high level), It is clear to me that the best way the game could be improved (i.e., perhaps only starting with X5 or beyond, if the present (parallelization) state of the game is due to engine limitations, etc...) is through far better utilization of (all of) the CPU cores that are available on the host running the game.

Since gaming PCs and their CPUs are only moving further and further towards (way too) many core architectures, games like X4 (and beyond) have to be able to better utilize the CPU resources that are available to as great an extent as is possible, even though this would obviously increase the complexity of the engine (and game) code in many cases and introduce the additional possibility of race-condition type bugs that are presently being avoided through minimal and/or absent parallelization.

[magitsu]

X4 is more CPU reliant than GPU. So it might not be first priority to offload something out of GPU.

Honestly recently many non-Egosoft people have come to the forums wanting to improve performance and end up nowhere. Because the devs have likely gone through those steps years ago. "Just make the game multithreaded" is a very common complaint with many games and it's usually shot down by the devs as "already done" with their first reply.
To me when you started talking about Sirnukes pipes, I'd rather see experiments in visualizations / UI work. That kind of prototyping with public testing could prove valuable.
But of course if you are just interested in optimizing...

I agree that trading dying seems like far more significant problem (if it reliably happens) than performance. Did you look through DeadAir Ware? It's probably the most common mod that alters it.

[SirNukes]

The hard part is beginning here: According to SirNuke's extensive documentation (kudos to this guy really), the way the communication between the lua engine residing on the gaming PC and the outside world is through an existing dll (debug-ui.dll if my memory is correct). So what I understood so far (please correct me if I'm wrong) is that hack used between the engine and the dll has significant limitations and prone to contentions. So asynchronous communications through that dll toward the Lua Engine wont work with multiple channels/processing nodes. It can still work from a single source, so it is still manageable (under certain conditions).

The custom dll just provides a lua wrapper around windows file open/read/write/close operations, for accessing the pipe. An arbitrary number of pipes can be open at once. The game interface with the pipe is run during the lua/ui phase of a frame (sending queued outgoing messages and gathering incoming messages), and is synchronized with the game inherently.

So, I think most of your worries aren't actually problems. The challenge would just be writing up all of the details on what information to export/import, developing whatever external logic (doesnt have to be python, but probably needs some sort of orchestration), and modifying the x4 scripts to use those imports instead of whatever internal logic is being superseded.

[waadaa85]

Thanks for this info. It is good to know it is feasible and the interaction with the game engine through that dll should not interfere (too much).

I'm currently looking at putting together (in python) the various necessary functions to insure proper comms between the local master pipe server and one headless client.
You already gave a serious help on how you structured the existing pipe server code and documented it. I'll continue on that path though. For the python coding of network comms I'm ok (I guess). A bit rusted tbh but it will come back.

My concern is indeed over the XML mechanics and interacting with the game. I'm totally out of my depth on that matter.
I think the best for me would be to build a simple test use-case based on one trivial in-game orders and/or actions, and then decompose + document + map the various necessary interactions to complete the use-case.
That use-case should start with a trigger event/action from an game actor (like a ship captain, or a station manager), which build a series of XML entries, call a "round-trip" from the game-engine to an headless client and back, then see how the game will ingest the resulting feed from the headless client and renders it in the game.

Cheers,

[waadaa85]

@Magistu: Interesting thought regarding the GPU. Actually, I wasn't thinking about that kind of "transfer of power" (from CPU to GPU).
Because, the GPU is itself very very solicited already by the game too. It is like to undress someone to dress someone else.

And I totally agree with you on the fact the dev team has already optimized every nook and cranny of the engine to squeeze out every drop of performance that could have been left somehow. With the inherent complexity of this game and its richness, yet still able to run respectably on middle-tier PC, lots of efforts have been put into it. No doubt. And it is way beyond the point of "simply turning XYZ option "on" like some threads may assume.

My line of thoughts is more on "How to have a gigantic XML "piano music sheet", currently playing on one CPU able to interact with another process running on another CPU. Not to play the same tune, no, because only one (the game engine) is (and must stay) the "system of records" for the whole universe. But to convey additional tasks in XML that other thread running on an other CPU host. Once completed, the result(s) is returned to the game engine. I don't know yet how to return something back to the engine though.

Prime rule being - not messing with the way the engine works. So I'm still trying to understand how the engine interact through XML. Second rule: keep things asynchronous and independent, so the game engine won't wait for something that could never come. I'm still reading a lot from X4 modding and community, so far I"m juggling with 3 ways of have two engines interact with each others in a "Client-server" /Master & Servants way.

1) As an XML Patch to a predesignated MD "stub" (following the rules described in theqmann XML Patch Guide), the game engine would have specially-tagged action triggered in the game by a game actor (ship captain, station manager, etc.), which would call an headless client for a series of lengthy and complex tasks calculations and/or extensive enhanced AI rules conditions. The result(s) would come back under the form of an XML file (complete MD or cue? or a XML-snippet ? I don't know yet) saved on a repository (where ? X4/libraries ?), from which the game engine could continue its play by reading the resulting MD/cue file/XML snippet the headless has returned;

2) Have a set of XML scripts - MDs and related Cues located on an headless client. Even whole specialty mods. The point is: Headless clients have a specialized role, dedicated on running specific tasks, MDs / cues, aiming to improve something or the AI on certain aspects where (again) complex + lengthy AI calculations and/or conditions would be processed and the result being returned to the master game;

3) (not yet explored - because I never actually been able to try the "venture Experience") I'm curious how the venture mechanism works. Because, technically how a ship goes out to a venture (ie. on another player's game instance running on another host somewhere on the internet) do things like buying and selling, interacts with others game objects in another game, and then get back, with probably less/more goods and money that will be added to the local game. To me, sounds like two independent game engines capable of interacting asynchronously with each others

"Interesting" like the Boron would say.

[magitsu]

My concern is indeed over the XML mechanics and interacting with the game. I'm totally out of my depth on that matter.

Here's some historical info about how the XML has been utilized throughout X series development.
viewtopic.php?p=4473344#p4473344 (from 2014, but X4 was based on XR VR engine - seems like a great thread overall for understanding the devs point of view)

viewtopic.php?p=4875180#p4875180 (dev priorities)

viewtopic.php?p=4930526#p4930526 (here's something about live state and its limits to serialization)

viewtopic.php?p=4875213#p4875213 (state info storing)

viewtopic.php?p=4473450#p4473450 (interesting 2014 dev reply about parallax mapping "lower on the bang-for-the-buck list" -> parallax occlusion mapping with X TECH 5 is now 8.5 years later coming in Boron DLC)

Probably worthwhile searching more before diving very deep solo.

I agree that headless should be interesting considering the future in case the revamped ventures takes off (pure sp possibly becoming co-op at some further entry to the series)

[waadaa85]

Thanks Majitsu for the info
Sorry for the delay in my reply got caught with RL events and responsibilities.
I'll take a close look on it for sure. To have an historical perspective will certainly help me better understand the game engine and not goes over solution paths that have already been explored.

Indeed, the current coop mechanism Egosoft dev team is currently putting in place could be a possible avenue to implement the headless client concept. The venture stations modules are probably something like a "game stub" where others instances from other players' universes can interact with each others (?)

For the headless client, I was thinking of something similar, a station module that would represent "in-game" anchor point for a RL pipe server container hosting a replicate of the Lua game engine. So additional processing provided by an headless client would be through that module.

Given my focus + intent to get such additional processing power is primarily to improve the whole economic model of the game universe, my module concept function would be something like a sector stock market. I'm currently modelling the underlying principles and basic rules (including a extended role for the faction representative and maybe the introduction of a new manager type to trigger specific DC scripts or cues that could be send to the headless client asynchronously for processing).
But before moving forward, I need to better understand the game engine inner working and see if the concept is able to fly before committing effort into it.

Right now it is the whole XML scripting and how the DC scripts and cues are interacting with each others and the game universe "master universe" XML that is still my big unknown to understand.

Cheers,

[Valhalla_Awaits]

Interesting, I had actually briefly considered this as well a year or so ago but waved it off and likely impossible from a modding perspective, requiring the developers to implement something that would likely require an engine overhaul to make dual PC/CPU load sharing possible. (Or effectively using a second PC as a sort of server to offload the insanely cpu intensive tasks.)

It's a fantastic idea, and reminds me of the game ARK's major crippling issues in singleplayer that required you to use a second PC as a server to share the burden of handling all the stuff going on in the world to get acceptable performance and fix resources respawning. But I think this dream may be too large to become a reality, and it's equally unlikely the developers would care enough to spend considerable development time to make this a reality.

Sad truth is, we'll likely just have to wait another 8 years of CPU development to make X4 playable at an "endgame/empire" level. For now, the developers expect you to just quit and restart the game once the performance tanks to the point it's unplayable.