The game uses CPU resources inefficiently

[af_2017]

I bet even most of programmers do not know how to write parallel code.

The problem is slightly deeper: quite often programmer who believe they can write good multi-threaded code actually cannot.

[Panos]

c) Consider trying the game on Linux. I have switched since 2019 and the game runs faster, expanding on all 12 (3900X) and 16 (5950X currently) cores, providing much better perf than on Windows.

Could you please prove it? Screenshot of the system monitor should be enough. Because you know changing OS to see the same would be frustrating. Thank you!

Why not pop in the Linux discussion?
You can run Linux without replacing windows shall you wish to do so.

[af_2017]

Why not pop in the Linux discussion?
You can run Linux without replacing windows shall you wish to do so.

The thing is I'm on linux and can see the games uses 2 cores (amd 7950: 16 equal cores/32 threads) during game and only one core during loading/saving.
I did notice that X:Rebirth performs smoother on linux.
That's why I asked for proof that x4 uses all cores on your system.

Thanks!

You can run Linux without replacing windows shall you wish to do so.

Even this means wasted efforts if the game uses same 2 cores.

[CBJ]

The thing is I'm on linux and can see the games uses 2 cores (amd 7950: 16 equal cores/32 threads) during game and only one core during loading/saving.

I'm not sure where you're getting your data for this, but it's quite simply wrong, at least from the game's perspective. The game uses 2 threads heavily during normal gameplay, plus as many other threads as are available for peripheral tasks. In some cases the usage of those other threads can also get quite high. During loading and saving the game uses at least three threads, though since one of those will be the limiting factor the other two may be much more lightly used.

Of course it's up to the OS how to schedule those threads onto different cores, and it's possible that for some reason your installation is cramming them all onto 2, but this seems a little unlikely unless you've specifically limited it to doing that.

[Shark22]

I'm not sure where you're getting your data for this, but it's quite simply wrong, at least from the game's perspective. The game uses 2 threads heavily during normal gameplay, plus as many other threads as are available for peripheral tasks. In some cases the usage of those other threads can also get quite high. During loading and saving the game uses at least three threads, though since one of those will be the limiting factor the other two may be much more lightly used.

Of course it's up to the OS how to schedule those threads onto different cores, and it's possible that for some reason your installation is cramming them all onto 2, but this seems a little unlikely unless you've specifically limited it to doing that.

I wish it were true, but the only time the game uses 2 threads is when loading or launching the game. In all other cases, it doesn't even know there's a 2nd thread. If it used at least 1 fully, that would be great.

[CBJ]

It is true, and your continued insistence that you know better despite admitting that you are not a programmer makes it pointless for me to continue to respond to your posts.

[Shark22]

I have never said that I know anything better. I only draw conclusions based on my observations. Your phrase that there is no point in answering my messages sounds more like the speech of an offended child than a tech support employee. I will be very happy if you really stop answering my messages. Unlike the responses of other support employees, I did not see anything in your messages except megalomania. "If need to explain, then do not need to explain."

[CBJ]

I'm not a "tech support employee". I'm a developer who has been trying, apparently unsuccessfully, to give you information about how the game works. Your response has, on every occasion, to tell me that I'm wrong. Now you've turned to insults. Carry on like that, and this thread will be locked.

[af_2017]

The game uses 2 threads heavily during normal gameplay, plus as many other threads as are available for peripheral tasks. In some cases the usage of those other threads can also get quite high. During loading and saving the game uses at least three threads, though since one of those will be the limiting factor the other two may be much more lightly used.

I think this official statement is worth pinning and bookmarking for future reference.
Thank you very much for this @CBJ!

Considering this following both cases can be correct:
1.

the game runs faster, expanding on all 12 (3900X) and 16 (5950X currently) cores

if amount of "peripheral tasks" is big; that probably depends on "game situation"

2.

the games uses 2 cores (amd 7950: 16 equal cores/32 threads) during game

if the "game situation" does not create enough of "peripheral tasks" to set 30 of remaining execution threads of CPU under full load.

As a conclusion back to the advice:

Consider trying the game on Linux.

No, this won't help. Changing OS won't bring such difference in CPU usage for this game: This game situation on this hardware will perform equally both on linux and windowz.
I even can speculate slightly further: there's no "magic" schedulers that can break execution of single thread (by design) application into multiple threads.
There are differences between linux and windowz indeed in performance but not in this area.

Thank you!

[jlehtone]

Changing OS won't bring such difference in CPU usage for this game: This game situation on this hardware will perform equally both on linux and windowz.

There are differences between linux and windowz indeed in performance but not in this area.

I don't disagree with that, but isn't it the OS that decides when and where a thread gets a slice of execution time? That could differ (but not necessarily in ways that matter here). There are options to pin a thread to specific core. I think I did try that with X3R. Can't tell whether it made any difference.

What could differ is how the executables are built. It might not be with same compiler and hence same optimizations, so CPU instructions in the executable could differ.

I wish it were true, but the only time the game uses 2 threads is when loading or launching the game. In all other cases, it doesn't even know there's a 2nd thread. If it used at least 1 fully, that would be great.

I have never said that I know anything better. I only draw conclusions based on my observations.

That is where the issue is.
You observe thread usage. You see one or two. Your initial conclusion was that the game does not use more.
You get additional "observation" from developer that the code of the game does use more threads.

Lets recap: a game that uses multiple threads does not seem to do so in your system.
You did stick to the conclusion that the game does not use many threads.
A more plausible conclusion would be that you should figure out why you don't see the threads.

[Shark22]

That is where the issue is.
You observe thread usage. You see one or two. Your initial conclusion was that the game does not use more.
You get additional "observation" from developer that the code of the game does use more threads.

Lets recap: a game that uses multiple threads does not seem to do so in your system.
You did stick to the conclusion that the game does not use many threads.
A more plausible conclusion would be that you should figure out why you don't see the threads.

Does it use multiple threads for you? Can you post a screenshot showing this?
I've tested this on other games, used multiple threads and everything works fine in them. From this I conclude that the hardware is fine. Drivers are up to date. I'm at a loss as to why only 1 thread is used in X4.

[Daemonjax]

The game definitely uses a bunch of threads. I remember watching what it was doing a few months back on my 10700k machine (hyperthreading disabled) and all 8 cores were used constantly and with high utilization -- was using rtss osd overlays to watch each core.

If it's not doing that for you, something is wrong.

I don't really expect any existing game to use more than 8 physical cores effectively.

I haven't tested this game extensively on my 5700x3d machine (with SMT enabled) that I recently built for cheap, but it seemed to run fine... maybe even better than the 10700k @ 5ghz, but I dunno for sure yet (seemed to have less stuttering when at higher lod slider levels).

One thing I noticed with AMD+SMT is that the windows scheduler screws up the scheduling when "cppc prefered cores" is enabled in bios for apps that do not explicitly pin threads to specific cores -- like prime95. Even with it disabled, it's not ideal because it will often dance a thread between two sibling cores for no good reason (there's still a price to pay for doing that -- even when the cores are on the same CCX). This is where the linux schedulers have the advantage. It can be mitigated to some extent on windows by running as close to 0 background processes as possible, adjusting cpu priorities, making sure nothing is panicking (I've seen older steam versions panicking, but pretty sure that's fixed now in the release version) and writing to event logs or reading/writing to the registry (lgcore.exe from logitech's gaming software), etc. -- all basic gaming on windows optimization stuff that many already do to some extent.

I never bothered testing the 10700k with hyperthreading on to see if windows screws it up there too, since I never enable hyperthreading in bios. I imagine x4 (and all competently coded game engines) explicitly pin threads to cores appropriately, but I dunno how well it's coded to handle e-cores -- might make sense to try testing this game without allowing it to use the ecores or to disable ecores in bios. I'm familiar with the windows API used to get cpu core info, and it's a shockingly terrible API.

EDIT: Try loading up 4 worker threads (uncheck multithreading and disable all avx) in prime95 and look at how windows schedules the threads (task manager> performance > cpu > view logical cores) to see whether it's assigning them to cores reasonably. Also try 8 and then 16 worker threads. This might lead you somewhere interesting.

[Shark22]

The game does use multiple threads. I was wrong here. I ran some tests and found out:
I tried different thread configurations, setting them up via the task manager. When using 8 or fewer processor threads, the main load goes to the threads that I left enabled (although they still don't load completely). If you enable more threads, they are simply not used by the game. As far as I know, these are limitations of Vulkan, which uses only 8 threads. There is no difference in performance when 8 or 32 threads are enabled. Although, I will still mention Cyberpunk, which uses all the threads it has.

In general, I opened the wait chain analysis in the task manager and there are constantly about 40 threads hanging in wait. As far as I understand, the processor simply does not have time to process these threads, which is why the FPS suffers greatly. Perhaps, if the game was made on DX12, which uses all the CPU threads, this problem would not exist, but this is just a guess.

[jlehtone]

Does it use multiple threads for you? Can you post a screenshot showing this?

I don't know about use, but the process itself has 70 threads (according to Windows 10):
https://drive.google.com/file/d/1L0Ooh5 ... sp=sharing

[Shark22]

I don't know about use, but the process itself has 70 threads (according to Windows 10):
https://drive.google.com/file/d/1L0Ooh5 ... sp=sharing

I just looked at it the same way. I have more than 270. Now I want fewer threads. =D Apparently my processor can handle about 230, and about 40 more are hanging in anticipation.

[xrogaan]

The game does use multiple threads. I was wrong here. I ran some tests and found out:
I tried different thread configurations, setting them up via the task manager. When using 8 or fewer processor threads, the main load goes to the threads that I left enabled (although they still don't load completely). If you enable more threads, they are simply not used by the game. As far as I know, these are limitations of Vulkan, which uses only 8 threads. There is no difference in performance when 8 or 32 threads are enabled. Although, I will still mention Cyberpunk, which uses all the threads it has.

In general, I opened the wait chain analysis in the task manager and there are constantly about 40 threads hanging in wait. As far as I understand, the processor simply does not have time to process these threads, which is why the FPS suffers greatly. Perhaps, if the game was made on DX12, which uses all the CPU threads, this problem would not exist, but this is just a guess.

You've been spreading FUD all around. Either because you don't understand whatever you're talking about, or you want to deceptively promote other technologies. Vulkan doesn't limit you to 8 threads, Vulkan has the same capabilities than Dx12. It's an API that let you talk to the GPU hardware. Vulkan is GPU stuff, not CPU stuff.
More on that: https://vkguide.dev/docs/extra-chapter/multithreading/

If all you're looking at, to drive you conclusions, are CPU utilization. Then you're myopic. Raising the amount of thread will not raise performances, and in many cases it'll have the opposite effect and reduce performances. Using threads is like using a funnel. You can have a big funnel but you'll still be limited by the throughput of the end point. X4 is using as many threads as it can, whenever it can, however the workload given to those threads aren't as heavy that it would max out a core for an extended period of time. If that were the case, the whole game would stall: the main process would have to wait for a thread to finish its work.

It doesn't matter how many threads a process can handle. What matters is the amount of time it takes to synchronize the work from all those peripheral workload. If the sync time takes too long, it's detrimental to the game. And we are here talking in microseconds: if you have 60 frames drawn per seconds, you have 0.0167 second to compute one frame.

The game performances aren't suffering due to lack of threads. It's suffering because the game compute its economy (offer, demand, travel, war, etc) in real time, and that can hardly be done through threads.

[Sprengmeister]

Let me help understanding the matter of CPU Bottlenecks and use of Threads on X4, by providing you guys with my observations on 4 Different PC´s with different Hardware:

Disclamer: I´m also nowhere near an Expert on the Matter of CPU´s! There could be some incorrect Stuff comming up you can correct me, but you don´t have too.

by my limited knowledge there can be at least 3 types of CPU Bottlenecks
1. Singlethread/core limit
2. Multithread limit
3. Datatransfer limit (aka Bandwith, Memory, Cash, etc.)

1. PC, my old Desktop used until jan. 2023
Specs Intel i7 920 4C 4T overclocked to 3,65GHz; 16GB 1333MT´s(aka MHz) Ram; GTX 980; Setting min. 1080p 60Hz
Max FPS 60 (Monitor Max), Avg. ~30FPS, Min 5FPS
What should I say, I can "play" the game, but it´s not nice. The Min FPS happen quite often, and I am literally in all of the 3 Bottlenecks seemingly at once!
Example If I put Ram speed to 1111MT´s CPU is bottlenecked by Datatransfer and the overall CPU utilization is about 60%, if I put it to 1444MT´s overall performance got a bit better and CPU utilization got to 80-100%
However even on min Graphics settingings, the GPU did bottleneck at some Sectors.

2. PC, my Laptop
Specs Intel i7 3610Q; 1333MT 8GB Ram; GTX 1070M
FPS very similar to first PC, so not much to add here

3. PC, my brother´s old PC that I used form Feb. to Sep. 2023
Specs Intel i7 5820k 6C 12T OC to 4GHz; 32GB 2933MT´s Ram; GTX 1080 Setting medium to High SSAA 2x 1080p 60Hz
Max FPS 60, Avg. ~45 FPS, Min 15 FPS
About 50% better FPS overall compared to PC 1, so not bad. And again I could run into all 3 CPU limits when Hyperthreading turned off. Seeing 6 Thread at 3,9 to 4Ghz being used to 100% was something I didn´t expect at all.
Sometimes one Thread was at 100% and limiting FPS, and sometimes my FPS dropped to 20, with CPU not at any obvious Single or Multicore limit. So again Bandwithlimit
At that point I wanted a new PC, that can deliver good performence for at least 8-10 years, with a 4K monitor in mind. That made my choce pretty easy, so:

4. PC, my current Desktop PC
Specs Ryzen 7 7800X3D Curveoptimiser set -30 all cores; 32GB 6000MT´s CL32 Ram; RTX 4090 Settings Max SSAA 2x 1080p 60Hz (Yes, new 4K OLED Monitor coming next year)
Max FPS 60 Avg 60! Min, 45 for like 2 Seconds when I fly fast to my 400+ Module Station, At the Station 60FPS!
Finally a PC that can give full FPS almost everytime Even in Mapview with all Tradefilters on 60FPS! Amazing
Now the fun part: how many threads are utilized at least by 10%?
12! some about 50-70% and in the 45FPS case 1 Thread is at 100% for those 2 Seconds.
Overall CPU load is at 35-45%
And also thanks to 96MB L3 cache, no more Bandwith limits since cache is a lot faster than even the best DDR5 Memory.

And lastly from the Factorio FFF´s regarding Multithreading:

"Failed attempt: Multithreading electric network update"
This is one of those things that I keep hearing from everyone: make electric network update multithreaded.
The electric network update was already improved (FFF-209), however it was still performed by only one thread which was often observed to be the slowest one to finish. In most cases save games have just 1 huge network, so multithreading wouldn't make much benefit. However with Space age, there are many large networks (different planets), so the potential is greater.
As for anything that is multithreaded, the first step that has to be done is identifying the moving parts and how they interact with each other. All of this is required because the game needs to remain fully deterministic or a desync would happen.
At first glance you would think it should be pretty simple, each thread works on a different electric network and its done. This is however not the case because there is one game mechanic that makes it slightly more complicated: the ability to have an entity powered by multiple electric networks.
In this example when the oil refinery is crafting, its energy storage is discharged and the electric network needs to charge it back again. Here we have 2 possible cases that can happen:
Left electric network updates first - charging the oil refinery from the Steam engine, the boiler would burn fuel and the inserter would activate.
Right electric network updates first - charging the oil refinery from the solar panels (in which case boiler remains idle).
Because of this, the networks are dependant on each other, and must be updated by the same thread.
After finding all the cases like this (like power switch closing causing multiple networks to merge), I was able to define what should an update group contain. If two electric networks had at least one entity powered by both networks, those networks had to be in the same update group, be updated by the same thread and desyncs will not happen.
I was able to start doing measurements...
The results
And this is where this idea failed completely. I could see our playtesting save file having at least 4 large electric networks completely independent since they are on different planets, however the electric network update time remained the same while the CPU usage went significantly up.
As it turns out, electric network update does not really do much: it just reads two variables, does one or two additions and goes to the next entity. It was memory throughput limited and by having more threads reading the data from memory, processor cannot simply read data faster. With multithreading here, instead of having one thread wait for memory, all of the threads doing electric network update were waiting for memory, slowing each other.
To fully reject this idea I had to use additional profiling tools like Intel's VTune which allowed me to give more numeric arguments that showed that electric network is memory throughput limited. Our playtesting save file had electric network update time improved from 0.5ms to 0.39ms while CPU usage went up from 0.5% to 15%. Overall the save file was not running faster.

[Shark22]

In the very first post in this thread I posted a save. I'd be interested to know how your PC performs next to my station in the Abyss. Can you check it out?

[Sprengmeister]

Will do, ("wait that´s a 20 Day save... ok") and here the Results:
First of all at Mapview I got 60 FPS even scrolling around fast, if I put tradefilter on, I can get it to drop to 45-50 FPS
When I was 100 KM from your Station full 60 FPS even looking at it.
Than I flew to at with travel drive, at 40 KM the games froze for 2 Sec. , so most likely my "45 FPS drop" just as extreme as your Station would expect, so no suprises here.
Inside your Station 10-15 FPS looking direcly at it.
When flying 100 KM away from your station full 60FPS when not looking at it, when looking at it now 20-25FPS (probably most things still renderd).
CPU load at 10-20% Why you may ask? -> Limit numer one Singlethread. And again totaly expected at a 2000! Module station with hundreds of Ships, some combat and ALL the small station masstraffic you can think of!
And again read the post from Factorio FFF I quoted. Even if the Devs would try to somehow Multithread this specific Issue, it will most likly end up not improving at all, the only thing that would change is the power consumption of your CPU which is obviously suboptimal.
Why does my overall CPU load get lower at your Gigastation you may ask? Simple, when you are single thread limited all other things won´t need to be computed as fast anymore, so in short, 1/4 of the simulation speed -> 1/4 of overall CPU load, minus that one thread.
My stong advise would be as already mentioned not to build a 2000 Module Gigastation in one Sector, but rather 4 or 5 400-500 Modul Megastation in 4-5 Different Sectors.
And now my final Example:
In Factorio you can build a Megabase with 10.000 SPM and get easy 60 FPS. But what happens if you build a 50.000 or a 100.000 SPM Gigabase?
Exactly the same thing that happens to you in X4 with your Gigastation! 15 or so FPS!
And what is the solution to the same issue in Factorio? Build 4-5 Megabases on different Severs aka Clustorio.
So in summary no matter how good you optimise a Game, no matter how good your Hardware is, with Building Games your can always get to some limit no matter what!
You can only push the limit futher until FPS drop by optimisation or better Hardware, but you can always reach it at some point.
And here at your Gigastation (by the way it is mighy Impressive) you are far beyond that point.

Hope i could help

Oh, by the way if you are wondering: if your FPS are double that of mine 1/4 of the simspeed from 60FPS is 15FPS, a 1/4 of 120FPS is 30FPS in Simspeed. but I recon both of our CPU´s being pretty much equal, al all Benchmarks, Test and Reviews are saying so.

[Shark22]

It's sad. I have exactly the same picture. Maybe in 10 years there will be equipment capable of producing high FPS, next to the station. And it's too late to dismantle it. It will take forever to remove resources from it. I think it will soon have 3000+ modules.