Curious as to why X4 is not configured for high end gaming computers.

[Scoob]

Even on high, I don't get the game to use more than 7gb, and I've got 64gb to use.

How do you get it to use 16gb?

Anyone know if you can do the same thing as that bat file does, inside the Steam start, like you can add the debug statement?

I only really started monitoring RAM usage closely when I had issues, and those were with a modified game. I really cannot recall - if I even checked - what my mature vanilla games used. I'll see if I have a backup somewhere of a vanilla save and I'll check...

I don't know about debug, never used it. Is it just another -blah option that could be added to the .bat? Or is that not what you mean?

[linolafett]

The entire game is like 21gb installed, it would be unreasonable to expect to fill up your 64gb of ram just because you got it.
I personally see no advantage in doing that.

[apricotslice]

Even on high, I don't get the game to use more than 7gb, and I've got 64gb to use.

How do you get it to use 16gb?

Anyone know if you can do the same thing as that bat file does, inside the Steam start, like you can add the debug statement?

I only really started monitoring RAM usage closely when I had issues, and those were with a modified game. I really cannot recall - if I even checked - what my mature vanilla games used. I'll see if I have a backup somewhere of a vanilla save and I'll check...

I don't know about debug, never used it. Is it just another -blah option that could be added to the .bat? Or is that not what you mean?

I've never run a vanilla X4 game. I think I lasted about 10 minutes after buying the game before I cancelled the window and went looking for mods.

-debug, yes. There's a place you put it in the steam loader, so the game generates a debug file every time it starts. Essential for mod making when things don't work.

The entire game is like 21gb installed, it would be unreasonable to expect to fill up your 64gb of ram just because you got it.
I personally see no advantage in doing that.

If it sped up the transition between menu items, it would be worth it.

At the moment, I get significant pauses when going from a build to the map to find a builder, and back again after, as well as between build and station operations. It's a significant game stops period, and a lot of that is storing all the modules in the build, or the number of modules in the finished station you're in sector with.

I'm pretty sure Egosoft has never tested the game with a station build with over 3000 modules before, limited to only 7gb of RAM.

Yes, I'm pushing the limits, but that's where the game is for me currently. Building large closed loop stations.

And the game lag between menu items is killing the enjoyment.

Not to mention what was said about FPS in sector with large build stations. That's hopeless at times.

I had to abandon putting weapons on my stations, because being in sector while the guns were firing was a major slideshow. It was totally unworkable.

[Rei Ayanami]

I'm not sure if it's even possible to force Windows (or any other modern OS) to keep the entire game and all of its assets in memory, as it's up to the OS, not the apps, to decide how to handle memory allocations and what to keep in RAM and what to cache out on disk.
Also, why force X4 to use more memory than it thinks it needs? If X4 really needs more than 7 GB at once, it will request more and will most likely get more from the OS. At times I've seen it go up to 12 GB on my PC, which has 16 GB RAM. That doesn't mean that it necessarily ran faster, though.

If it sped up the transition between menu items, it would be worth it.

At the moment, I get significant pauses when going from a build to the map to find a builder, and back again after, as well as between build and station operations. It's a significant game stops period, and a lot of that is storing all the modules in the build, or the number of modules in the finished station you're in sector with.

[...]

And the game lag between menu items is killing the enjoyment.

If I had to guess then the most likely reason why there is a detectable pause between different screens is that map data, station build data, etc are probably not updated while that particular menu/screen is not active, so when switching to a menu/screen it has to poll game data and build the menu/map from scratch. Even if everything was in RAM, it'd still take some CPU time to poll everything and build the menus. Of course, the more complex an entity, such as your 3000 module station, the longer it takes to fully poll all data for it.
Unless you want to have all screens always updated, even when they are not active, (which would be a CPU time waste), the only three alternatives would be a) having a small pause for menu entries to load (as it is now) or b) opening a blank screen and streaming in all information over time (which would look rather bad), or c) only displaying the menu after everything has fully loaded, which means considerable input lag because after pressing the button you have to wait for the menu to be fully ready (also bad). Pick your poison.

There are some screens which COULD be preloaded in a speculative manner in the background (station selected -> preload station build menu and station logistics just in case player wants to see them), but that too has downsides, like selecting one station on the map -> begin preloading that station for all of its possible menus/screens, in case player wants to select it, but then player select another one -> abort preloading previous station, begin preloading new station's menus/screens would also be kind of a CPU waste and would also most likely have some form of lag (have to wait for assets to get streamed in).

Not to mention what was said about FPS in sector with large build stations. That's hopeless at times.
I had to abandon putting weapons on my stations, because being in sector while the guns were firing was a major slideshow. It was totally unworkable.

Well, that's the limit of software. It's the same with real time strategy games without unit limits : It runs fine as long as you only build in expected amounts, but if you build 1000+ units it''s reasonable to expect that there will be some kind of slowdown. If you have thousands of players on an MMO server at the same location, chances are there will be lag. And your 3000 modules are not only an extreme version when it comes to calculating production, but also rendering time (I assume one or more draw calls per module) and possibly collision time (ships near station doing pathfinding, turrets checking for line-of-sight, etc).

And no, I'm sure Egosoft didn't test with a station with 3000 modules, because that's not what's expected to happen in 99.9% of all played games, just like Microsoft probably doesn't test for Windows having 10000 apps open and active at the same time. Of course, you can try it, but I don't think criticizing Microsoft for not having an OS able to have 10000 apps active at the same time while keeping acceptable performance would be fair.

I'm not saying that there is nothing the X4 engine could improved in, but If you test out the limits of software and perform unexpected actions in extreme ways, you have to expect to bump into limits in return (low performance, high memory usage, etc).

[Scoob]

If you have 64GB, pop the game into a RAMDrive. I used to do that, but with "only" 32GB the game is too big now.

Edit: Btw: try creating a new instance and applying the Star Wars Interworlds mod to it. This is likely on the more-demanding end when it comes to mods, and I certainly see 16GB+ used by the game process during play, my game is a few days old now though.

[BitByte]

With 16GB of RAM you'll need have pagefile / swap enabled (atleast 4GB). With 32GB+ pf can be disabled or set permanent small (128MB).
If you put game to background then Windows doesn't tell the full truth.
From Windows task manager open details-leaf and make sure you see column called "Commit size". This value is the one you might be interested.

For example in my case as game in now background task manager shows little below 7GB of mem use for X4 process. But "Commit size" value is over 10GB.

[apricotslice]

If I had to guess then the most likely reason why there is a detectable pause between different screens is that map data, station build data, etc are probably not updated while that particular menu/screen is not active, so when switching to a menu/screen it has to poll game data and build the menu/map from scratch. Even if everything was in RAM, it'd still take some CPU time to poll everything and build the menus. Of course, the more complex an entity, such as your 3000 module station, the longer it takes to fully poll all data for it.
Unless you want to have all screens always updated, even when they are not active, (which would be a CPU time waste), the only three alternatives would be a) having a small pause for menu entries to load (as it is now) or b) opening a blank screen and streaming in all information over time (which would look rather bad), or c) only displaying the menu after everything has fully loaded, which means considerable input lag because after pressing the button you have to wait for the menu to be fully ready (also bad). Pick your poison.

Well yes. I would like to have each of the major menu screens open at the same time in a different window, and dynamically updated in real time.

In terms of open processes, that would mean 6 to 8 of them all open at once, instead of 1 now.

But the thing I don't understand is why the GPU isn't being used for rendering the station build screen, and other places where there is so much going on that the lag is happening? I've got a 3090 which as far as I can see is being ignored.

So this seems to be a 2 part problem. Everything trying to happen in 1 process window when it shouldn't be, and the GPU not being used when it should be.

I'm not saying that there is nothing the X4 engine could improved in, but If you test out the limits of software and perform unexpected actions in extreme ways, you have to expect to bump into limits in return (low performance, high memory usage, etc).

Sure, and I'm pointing them out.

Egosoft used to push the hardware limits with their games.

To play them you needed to keep up to date.

Now I'm finding they've stopped doing that. Or maybe the hardware is pulling away from them faster than they're aware of, or maybe they've stopped pushing the envelope.

For someone like me who's always been prepared to upgrade the computer hardware as a new X game comes out, I'm disappointed to find that going to the latest hardware now is no longer required, because the game cannot use it.

What stops me playing any X game is ultimately the frustration factor hitting the limitations of the game.

Hence this thread.

My frustrations this time around are based on the fact the game isn't using the available hardware, and is instead creating a series of choke points, and forcing you to endlessly go back and forth through the same screens, with lag between them, on a computer which is effectively under utilized.

All I'm saying is, Egosoft please think about where the hardware has gone, and push that envelope out for those who already went there.

And as I suggested, release a high performance game option as an add-on and make me pay for it, and then those of us who can run it, will. And be happy doing it.

[KextV8]

Your own possessions are clouding your judgement for what the average computer looks like. Easily accessible sources of data like the steam hardware and software survey quickly demonstrates that.

The most popular setups still only have 6 cpus, with roughly 75% of the surveyed having 6 cpus or less. The most popular RAM configuration is 16gb, with more than 80% of the survey have 16gb or less. The most common GPU is the GTX 1060, with the xx60 line in general dominating the charts by a wide margin, though the xx50 line definitely makes a good showing. The most common display resolution is 1920x1080, with a little over 70% having that resolution or lower on their primary display. The most common multi-monitor setup is Dual 1080p monitors with over 90% of the survey having a combined total resolution of dual 1080p or lower.

You dramatically overestimate the typical computer out there playing today's games based on your own experience. That's fine, it would be nice to get a little more performance out of the game. But is it worth the development time and cost, especially once you consider the opportunity cost of development, when it won't benefit most of the game's audience?

Egosoft used to push the hardware limits with their games.

To play them you needed to keep up to date.

Now I'm finding they've stopped doing that. Or maybe the hardware is pulling away from them faster than they're aware of, or maybe they've stopped pushing the envelope.

As for this, my personal perspective is that hardware limits, especially regarding raw CPU processing power simply have not improved at a really impressive rate compared to the gains that were happening 15 years ago. GPU's have gotten a lot better, but CPU's have really kind of stagnated. Like... great you shoved more cores onto a single chip, but that doesn't really help much on single thread processing, and doesn't address thermal issues very well.

TL:DR - Why isn't the game configured for high end gaming computers? Because high end gaming computers are a small minority of the product audience.

[apricotslice]

TL:DR - Why isn't the game configured for high end gaming computers? Because high end gaming computers are a small minority of the product audience.

Well, I have to admit to being surprised with that.

Here's me considering upgrading my CPU and going to a 4090 because my computer was falling behind, and apparently I'm a light year ahead of normal anyway.

Didn't see that coming.

But it does make me wonder where all the gamers went, because in gamer expectation terms, I thought I was falling behind.

I guess I'm just a dinosaur now. I no longer have the normal computer for games, I refuse to use the videogame style device's which apparently most people use now (I read the survey results), and I stick with a real joystick.

The game industry went a different direction on me when I wasn't looking.

But you know, I won't apologize for my expectations of a game these days.

Companies like Egosoft made me push the hardware forward with each new game for decades, in fact, and now I still expect them to even when apparently they stopped.

I'm their creation.

And while you can't see where I'm coming from, they should. But I guess I'm not representative of the 'market' anymore. Who knew?

[Okay, that was a bit sarcastic, but I feel quite let down now.]

[linolafett]

Pushing for stations with 3000+ modules is not what we designed the game for. This is something you can do, but are not required to do so.

You can put 14 people in a smart car, but it wont be an enjoyable ride, same here.
It should be clear that using the vehicle in that way is not what it was designed to do and it is alien to me to ask for it to be redesigned to work in this configuration.

[Imperial Good]

But the thing I don't understand is why the GPU isn't being used for rendering the station build screen, and other places where there is so much going on that the lag is happening? I've got a 3090 which as far as I can see is being ignored.

The GPU is being used to render those screens. The GPU is used for all rendering. Unless for some reason you installed a third party Vulkan software device and for some reason the game decided to use that as the main render device.

Due to how powerful the RTX 3090 is and how well optimised the renderer for X4 is it is quite possible that even a "3000 module station" at high resolution does not tax the GPU much. The GPU is still rendering it all, just what is being rendered is so simple for the GPU that it is nowhere near bottlenecking performance.

So this seems to be a 2 part problem. Everything trying to happen in 1 process window when it shouldn't be, and the GPU not being used when it should be.

The use of only 1 window is likely due to the low uptake of multi-monitor setups. Even of those with multi-monitor a lot will not want to use the other monitors for X4 and instead have them display other things for user multitasking.

Most of the time the GPU is being used when it makes sense for it to be used. There are likely a few extreme cases, such as GPU texture decompression and direct I/O, where the GPU may not currently see use but these are cutting edge technologies which last I checked lack support for Vulkan and have as good as no adoption in the non-console gaming industry.

Now I'm finding they've stopped doing that. Or maybe the hardware is pulling away from them faster than they're aware of, or maybe they've stopped pushing the envelope.

The issue is more that there is no real logical progression path for consumer hardware anymore. CPU, memory and GPU capabilities and performance are all over the place and ultimately you have to design a game that targets a wide variety of hardware if you ever want to see significant sales.

In the old days there were single cores, dual cores and then quad cores, which then remained the status quo for over a decade with the main difference between tiers being clock speed. This resulted in the progression path of 1 -> 2 -> 4 threads where you could expect every few years significant IPC improvements per thread. Sure, there were outliers with more cores but these were usually lower clocked so no one would expect to perform better when gaming on them. However recently things have all changed... Now we have 6, 8, 12, and even 16 core processors in consumer space thanks to AMD. Well we do not.. since even more recently we have those and 4 to 8 additional E cores thanks to 12th gen Intel. Well we do not... Now we have all those plus an additional 8 to 16 E cores thanks to 13th generation Intel. So within the consumer space performance cores are anywhere from 6 to 16 and E cores from 0 to 16 depending on the product tier with high end parts having between 16 and 24 cores. It only makes sense to build your game to be fully playable by the lowest core configuration to have access to the largest player base which means that X4 must be highly playable on a 4-6 core processor, which it is. Since it is highly playable on a 6 core processor it is very difficult to find optional uses for additional processing cores, especially when there are large diminishing returns with some tasks scaling with core count. It is just not reasonable to expect modern high-end processors to be fully loaded by games like X4, or even games in general. I do not know of a single game client that would fully load a modern high end consumer processor.

Memory is a similar issue as you have people with anything from 16 GB all the way up to 128 GB or more. The game has to be highly playable on the lower end (most common) configurations of that. Since it is already highly playable without using a lot of memory, it is hard to find optional uses for that memory to try and improve performance. Especially considering memory bandwidth does not scale linearly with memory amount. Sure, they could cache more and more assets of X4 in that extra memory, but X4 is not even that big (will fit inside 128 GB of memory completely) and if installed to a high speed NVMe drive this might not even give a noticeable increase in performance over reloading the assets from storage.

Modern GPUs not being highly utilised is largely due to CPU bottlenecks and the overall environment of X4. Space, like in real life, is quite empty so quite a bit of the time is much easier to render than a city, or forest, or even a crowd of people. Of course, they could heavily enhance the quality of assets, adding more fine geometry and higher resolution textures, but assets cost a lot of money to create and are generally considered a huge development bottleneck in the gaming industry. Raytracing could be an option, but this would require significant development time for a feature that most people probably will not use and might not even improve visuals significantly in most scenes. Even then if you are flying to the middle of nowhere there might be little for your GPU to do.

My frustrations this time around are based on the fact the game isn't using the available hardware, and is instead creating a series of choke points, and forcing you to endlessly go back and forth through the same screens, with lag between them, on a computer which is effectively under utilized.

Might be a good idea to post the not modified save along with recreation instructions to experience this "lag". The developers might be able to profile it to look for potential optimisations in those cases for the future.

As for this, my personal perspective is that hardware limits, especially regarding raw CPU processing power simply have not improved at a really impressive rate compared to the gains that were happening 15 years ago. GPU's have gotten a lot better, but CPU's have really kind of stagnated. Like... great you shoved more cores onto a single chip, but that doesn't really help much on single thread processing, and doesn't address thermal issues very well.

Single thread performance has improved significantly over the last few years. It might not be improving as fast as it was during the 1990s but something like a Core i9 12900k does have a lot higher single thread performance than a Core i9 9900K from a few years ago. There are still significant performance improvements expected over the coming generations that are not all from throwing more power at the problem.

But it does make me wonder where all the gamers went, because in gamer expectation terms, I thought I was falling behind.

Most PC gamers were always chasing the best bang for the buck. Like quad core i5 processors that could be overclocked to perform like much more expensive i7s with only a moderate cooling solution. This used to include some high-end GPUs, but back before they cost anywhere close to a thousand dollars each. As it stands no one currently recommends a RTX 4090 or for gaming, that GPU just does not make sense in that use case given its price and capabilities.

[apricotslice]

@ Imperial Good

What is a 'not modified save'?

I don't have one, since I've never run the vanilla game long enough to want to save it.

And what I'm talking about, while it can be built in game, I made a work around to get it at the beginning of the game to provide a game challenge. It just has some side effects on game play.

I did some further checking, and the game is using 3gb of GPU RAM. I just wasn't seeing it before. One of the other previously hidden things was the game also uses 100+ Threads.

Aside from the mechanics of it all, the single biggest issue I have is the constant going back and forth between the same screens.

And since I spend 95% plus of game time NOT in a ship, all I do all the time is go back and forth across those same screens, sometimes changing them every 30 seconds or so. That's a huge frustration factor for me.

I never stop playing X games because I run out of things to do.

I always stop playing X games because the frustration factor rises too high.

[KextV8]

Single thread performance has improved significantly over the last few years. It might not be improving as fast as it was during the 1990s but something like a Core i9 12900k does have a lot higher single thread performance than a Core i9 9900K from a few years ago. There are still significant performance improvements expected over the coming generations that are not all from throwing more power at the problem.

Significant depends on perspective. It has been roughly a 20% IPC gain over 4 generations. At this rate it will take 20 generations to have doubled the IPC. To me, that is not significant improvement.

[Imperial Good]

What is a 'not modified save'?

A save file that has only been played by X4, and any of its its DLCs, without any third party mods being installed and that has not been tampered with at any stage. In other word an X4 save file that is what the developers expect it to be.

Non-budgeted starts probably will be accepted for technical and performance issues despite being marked as modified, as long as the save file itself has not been tampered with or corrupted.

[apricotslice]

What is a 'not modified save'?

A save file that has only been played by X4, and any of its its DLCs, without any third party mods being installed and that has not been tampered with at any stage. In other word an X4 save file that is what the developers expect it to be.

Non-budgeted starts probably will be accepted for technical and performance issues despite being marked as modified, as long as the save file itself has not been tampered with or corrupted.

Umm, no, I wasn't being serious.

I thought the next line made it quite clear I knew what it was, but didn't have one. Apparently not.

I've been playing with a modified tag since X4 day 1.

I can't think of any reason why I'd even start the game now without mods in place.

Part of this is having been playing with modded games since X2, and being well over the whole starting from scratch thing back in X3TC. I never do now.

I did try with X4, having not played XR at all, but I deemed the original build as unplayable within an hour. I've never had a reason to retest that since.