Super-sized complexes - without the LAG - for all

General discussions about the games by Egosoft including X-BTF, XT, X², X³: Reunion, X³: Terran Conflict and X³: Albion Prelude.

Moderator: Moderators for English X Forum

glenmcd
Posts: 920
Joined: Sat, 16. Oct 10, 11:07
x3tc

Super-sized complexes - without the LAG - for all

Post by glenmcd » Wed, 17. Feb 16, 04:48

Click here -> THE DOWNLOADS (Benchmarks.txt, 24 save files for benchmarking Albion Prelude complexes)

In this thread you'll find a very usable workaround to what has been a major cause of lag and generally poor performance in mature X3 games. My initial post has been revised (twice) to get the essential information to those that need it in a more compact form and to reflect a second set of benchmarks performed aimed at excluding affects of NPC stations, ships, generic missions etc.

FOR PLAYERS:
When you first start an X3 game, framerates are good (perhaps 100 to 900 FPS), it's quick to pass through a gate to get to another sector (1.5 to 10 seconds depending on computer) and loading from a saved game is reasonable (7 to 30 seconds). As your game matures and you build factories and eventually complexes, performance all round decreases.
There are at least two sources of major lag that affect most players:
1. Generic Missions (the ones you can accept from NPC stations and some NPC ships). The performance degradation from this source in a vanilla game increases every time you enter a new sector but can spawn at any time. Unlike factory complex lag, GM lag is happening right from seconds into a new game.

Each generic mission spawned generally disappears within a couple of hours. Without two current (mission type) exceptions, if you stayed in an unknown or xenon sector for two hours or more, this generic mission induced lag would go away. But in Unknown and Xenon sectors, there is no mission spawning. Some mission types cause more lag than others and which missions spawn is pretty much random. This post isn't about the generic missions. It is important though to make the distinction between the different causes of high level lag in X3. This has been quite difficult to achieve in the past.

2. The use of "Complex Construction Kits" to connect factories together into a complex. The way that virtually every player to date has used these introduces lag in the game that grows exponentially with the factory count. In this post I reveal a different and somewhat counter intuitive method of using these same CCKs to reduce the lag from complexes to a level that will be not even measurable for most players. Others that strive for super sized complexes will now be able to achieve their goal without making their game unplayable.

In the folder linked to above ("THE DOWNLOADS") is a text file "Benchmarks.txt". There is also a bunch of X3 save files that allows you to do your own comparisons under equal environments. The environment chosen eliminates most of the significant causes of lag with exception of that caused by player factories and complexes. Thus, the benchmarks should be meaningful.

Whether your game is modified or vanilla, anyone can use the method described below.

CCKs have typically been used like this: Connect factories 1 and 2 to produce Complex Hub (CH) "A". Use a second CCK to connect CH "A" to factory 3. Use a third CCK to connect CH "A" to factory 4 and so on.

Now here's how to use CCKs so your game keeps running well: Connector factories 1 and 2 to produce Complex Hub "A". Use a second CCK to connect factories 3 and 4 to produce CH "B". Use a third CH to connect CH "A" to CH "B".

You connect factories and you connect CHs. Let's just call this connecting complexes, where a factory is considered a 1 factory complex. Basically the rule is this: Use a CCK to connect the two smallest complexes. The next CCK you use connects what is then the two remaining smallest complexes. So when connecting 256 factories, first use 128 CCKs to produce 128 Complex Hubs. Then use 64 CCKs to connect these 128 CHs to produce 64 CHs. Then use 32 CCKs to connect these 64 CHs to produce 32 CHs. Keep going until you finally use a single Complex Construction Kit to connect the two remaining Complex Hubs, to produce your final Complex Hub which you can dock at and which connect all 256 of your factories. If you are constructing a complex that is not an even power of 2, that’s okay. Just stick to the rule above and it will be fine.

FOR TECHNICALLY INTERESTED:
The X3 engine in the background updates ware counts in all factories in the Universe. It does this at all times, and thus framerate is affected by how much work is required, as factories need to be updated in game playing time and not slow down just because there’s more factories in the universe. Whenever you fly through a gate or use a jumpdrive to enter a new sector, all factories in the universe are updated in one go and not just the sector you are about to enter. Whenever you load a saved game, it’s a similar thing. So whatever load the X3 engine endures with maintaining ware levels in factories, effects these three things. It also affects the time it takes to quit back to the menu, build a new factory and various other actions in game.

So if we double the number of individual factories in the universe, the load this places on the X3 engine doubles. The overall effect of this is larger than you think, due to how CPU caches work. This has the potential of decreasing framerates and of increasing the time it takes to leave a sector, enter a sector, load up a saved game or quit your current game.

Currently at the start of a new game there’s around 1700 factories in the universe (Albion Prelude). If the player builds a factory, it adds very little to the CPU load. In fact, if your computer is relatively fast, you can build perhaps two thousand factories and not really notice much drop in performance. But when you use Complex Construction Kits to join factories, everything changes.

CPU execution requirements increase exponentially with each additional factory in a complex. What that exponent actually is, depends on the order in which you connect your factories and complex hubs. With the old way of using CCKs, the exponent is larger, perhaps somewhere between 2 and 4. With the new way, it’s only very slightly over one. When things scale exponentially, you can no longer use percentages to compare relationships. Execution requirements for factory complexes with a higher number of connected factories eventually place an overwhelmingly high burden on the CPU. The time to pass through a gate may increase by perhaps a few billionths of a second if you have a 5 factory complex instead of a 4 factory complex. But if the complex already has 450 factories, each additional factory can add over half a second to EACH gate pass and each loading of a saved game. If you ever managed to build a 1020 factory complex the old way (I doubt it), gate passing time would be measured in years, possibly even in decades or centuries. Such is the power of exponential relationships.

By using CCKs in the manner described above, you are effectively decreasing exponentially, the number of factories that the X3 engine checks as it updates ware levels in each player factory and each complex hub. This is because the number of links (from using a CCK) between it and the Complex Hub is always equal to the square root of the total number of factories in the complex. But using CCKs the old way, the number of links between each factory and the Complex Hub you can dock at vary between one and one less than the total number of factories. So this averages out at half the factory count. Multiplying these out for a 256 factory complex, and we get 2048 for the new way, and 65512 for the old. Although for this example you could say that the new is roughly 32 times faster, it’s not as simple as that. The caches in modern CPUs can be accessed many times faster than normal memory. Serially accessing longer linked lists can suddenly introduce cache misses. The new method greatly reduces CPU cache misses in maintaining ware levels in factories. So it does get complicated very quickly if you attempt to work out precisely how well the above works for any complex size. The best way of quickly learning how performance differs for various complex sizes and build methods, is to have a read of Benchmarks.txt above.

The downloads folder above focuses mainly on two complex sizes - 256 and 1020. It also includes save files for many "OLD STYLE" complexes, of sizes zero (of course!), 39, 49, 59, 69, 79, 92, 99, 109, 119, 129, 139, 149, 159, 169, 179, and 189. In one save you'll find multiple seperate complexes built the old way with sizes 64 and 256. These are then connected and each construction level has another save provided. Each and every save has you in a very fast M5 ship in an empty sector with neighbooring empty sector. They've been setup so as not to trigger any generic missions when you time gate passing. All asteroids, stations etc have been removed also. Actually they have been removed throughout all sectors!

Here's a few benchmark hilites:

No factories:
Framerate 1000+ frames per second (FPS)
Pass a gate: 0.6 seconds (allowing 1.2 seconds for the fixed overhead for fading screen to black and reverse in new sector)
Loading the save: 6.1 seconds. (time only while the loading screen is visible)

OLD -> NEW comparisions:

256 factory complex complete:
Framerate:
227 FPS -> 990-1000+ FPS
Pass a gate:
13.1 seconds -> 0.63 seconds
Load the saved game:
18.8 seconds -> 6.1 seconds

It's simply not possible to build a 1020 factory complex the old way. Performance decreases to such a point that further building becomes impossible. So here I compare the benchmarks for a 1020 factory complex built the new way to a 179 factory complex built the old way.

179 factory compex old -> 1020 factory complex new:
Framerate:
430 -> 650 FPS
Pass a gate:
2.47 seconds -> 1.3 seconds
Load the saved game:
8.2 seconds -> 6.8 seconds

I am expecting that the above will raise a number of questions regarding how performance will be affected on different computers, larger/smaller complexes, hybrid build methods etc. That's why I put together the two dozen saves available for everyone, so you can benchmark on your own system without having to go through building of each complex.

Provided that you stay within "Unseen Domain" and the "Unknown Sector" beside it, you may swap between these sectors as often as you wish without experiencing increasing lag from generic missions. In time however the universe will begin to respawn, at least for many of the ships and stations. So you have perhaps an hour at a time to get some accurate benchmarks attributable purely to the player complex(es).

Update 27/May/2023: Files have been inaccessible for some time, today I zipped them and updated the link above so you should be able to get to all of the original files now. The issue was server protocols related, not that the files not being there. My apologies for taking so long to provide the workaround.
Last edited by glenmcd on Sat, 27. May 23, 00:53, edited 8 times in total.

fireanddream
Posts: 384
Joined: Sun, 13. Dec 15, 07:15
xr

Re: Super-sized complexes - without the LAG - for all

Post by fireanddream » Wed, 17. Feb 16, 09:43

First of all, good to know people are still working on this... or at least, people are still complaining about this. For a game that's 5 years old and with its "sequel" released I pretty much expected the dev to just abort the supports and everything.
glenmcd wrote:Connect the 32- and 4- factory sized complexes giving you a 36 factory complex. Lastly, connect the 64 and 36 factory complexes to produce a single 100 factory complex. Don't connect the 64- to the 36- factory complex first as this will lead to a large mismatch in sizes for the final connect. If you can follow this, you can get excellent performance from any sized complex.
Did you mean to say "don't connect the 64- to the 32- ..."?

I thought the fact that low IS frame rate can be solved by a tubeless complex mod suggests the game recalculates tube placement every time we pass through a gate, and the time required to do that grows exponentially indeed. From your workaround I guess if we build a 10 station complex and connect the stations "normally", the game treats it as "complex A+3+4+5+6+7+8+9+10", but if we do it your way, the game treats it as "complex [(A+B)+(C+D)]+E"? So in the end it's 9 "objects" vs. 2?

But wait, are you sure the generic mission issue was fixed in the latest patch of AP? In my previous save I had ~150 stations, only one large complex (~90 stations) but my frame rate was no where near the frame rate at a fresh start after 6 in-game days. After reading this post (http://forum.egosoft.com/viewtopic.php?t=311890) I downloaded the empty mission directer file, saved and deleted it but my frame rate still improved a lot.

Sorry for funky grammar and everything. Not a native speaker.

glenmcd
Posts: 920
Joined: Sat, 16. Oct 10, 11:07
x3tc

Re: Super-sized complexes - without the LAG - for all

Post by glenmcd » Wed, 17. Feb 16, 13:46

fireanddream wrote:
glenmcd wrote:Connect the 32- and 4- factory sized complexes giving you a 36 factory complex. Lastly, connect the 64 and 36 factory complexes to produce a single 100 factory complex. Don't connect the 64- to the 36- factory complex first as this will lead to a large mismatch in sizes for the final connect. If you can follow this, you can get excellent performance from any sized complex.
Did you mean to say "don't connect the 64- to the 32- ..."?
You are correct, thank you (now edited).

fireanddream wrote: I thought the fact that low IS framerate can be solved by a tubeless complex mod suggests the game recalculates tube placement every time we pass through a gate, and the time required to do that grows exponentially indeed. From your workaround I guess if we build a 10 station complex and connect the stations "normally", the game treats it as "complex A+3+4+5+6+7+8+9+10", but if we do it your way, the game treats it as "complex [(A+B)+(C+D)]+E"? So in the end it's 9 "objects" vs. 2?
Not quite. The bottleneck that this thread is about, has nothing to do with IS lag, however sometimes with IS and OOS lag, one and one add up to three. IS lag has mostly to do with graphics and models, OOS all of that is totally ignored. Tubeless does help, but one time when I had 1000 factories visible with nothing connecting them at all, framerates IS dropped to low single digits. Yet OOS, there was really no drop in performance at all compared to no factories. I even tried 2000, pretty much same thing. That's what made me realise that it was the CCK code adding the OOS lag. If you look at the files for the tubeless mod, there's actually no code at all. And that's exactly why it manages to be vanilla compatible. Models only.

Regarding how the X3 engine performs the exponential execution thing: Firstly, I don't know for sure. I can only guess. Because I found something that worked, I may have guessed right. Maybe. Here's my best answer at the moment:

Each factory must communicate ultimately with the hub but it can only do so through whatever CCK it was joined to. With the OLD approach, the average number of hops between any one factory and the hub is equivalent to half of the (factory plus CCKs) count. For ANY complex, the required number of CCKs is exactly equal to the factory count minus one. In the case of a perfectly formed binary tree, the nodes furtherest from the top (which is effectively the Complex Hub in this case) are all at a distance equivalent to the square root of the total factory count. So for a 256 factory complex, the furtherest any factory is, is only 8 jumps. The number of CCKs in each case would be 255. With the NEW (binary tree) these are laid out as 128 just above the factories level. Next going towards the CH is a row of 64. Then 32, 16, 8, 4, 2 and then 1. Total these and you get 255. In the case of the OLD, there's a CCK in between EVERY factory and its neighboor, except of course the CH's single neighboor. So again the total is 255. The last factory added in OLD has to traverse through all 255 other factories, and all 254 CCKs before finally reaching the Complex Hub. At the other end, factory #1 has immediate access to Complex Hub. Half way, its half way in jumps also. So using averaging, we get total iterations for OLD as 65512. NEW is much easier to calculate, as all nodes have equal access - exactly 8 CCKs to pass through. So we get 256 * 8 = 2048 for NEW. Doubling the number of factories gets us figures of 261632 for OLD and 4608 for NEW. NEW had 32 times fewer iterations for 256 factories, but increases to 56 times fewer when complex size is doubled. Depending on how many times through these structures the code executes, the (32 times fewer iterations) and (56 times fewer iterations) add up. Multiple this by the factor for cache misses (I've read figures of around ten) and we're getting pretty close to what the benchmarks indicate. So what we've got in switching to a binary tree, is a reduction in structure parsing iterations that increases exponentially as factory count increases, effectively flattening out CPU usage as relative to factory count.

If Egosoft fix this then we can go back to connecting CCKs however we want. Until then, once you know what a difference you can make to performance for the remainder of your game (which in the X3 case can be years!), it becomes rather critical to be a bit more careful and create the binary tree described.
fireanddream wrote: But wait, are you sure the generic mission issue was fixed in the latest patch of AP? In my previous save I had ~150 stations, only one large complexes (~90 stations) but my framerate was no where near the framerate at a fresh start after 6 in-game days. After reading this post (http://forum.egosoft.com/viewtopic.php?t=311890) I downloaded the empty mission directer file, saved and deleted it but my framerate still improved a lot.
Okay this is where it gets quite interesting actually. I only gave a small snippet of the set of benchmarks I generated which researching this. The CCK code bottleneck has effects which tend to compound with other things:
1. War sectors, when Argon and Terran are really giving it to each other
2. Aldrin, Albion Gamma
3. Your CPU cache size. I believe that this in particular, is responsible for the quite sudden increase in lag when your complex gets to XX number of factories total (in one complex). On my computer, 50 factories adds 0.6 miliseconds per factory for gate passes, while a 30 factory complex adds zero. With 80 factories, it's up to 1.25 ms / factory. By the time you get to 450 factories, each additional factory is adding more than half a second to each gate pass!! All these are for classic CCK connections. NEW style, at 450 factories each additional factory adds only 0.35 miliseconds, roughly 1600 times less at that point. With a 1020 factory complex, the difference would likely be in the millions to billions.


What's happening here is that the code is iterating along these linked lists, and if the list is short enough, it fits into the CPU caches. But once you breach this limit, you're accessing memory which is at least ten times slower (depending on CPU). The cache sizes in various CPUs vary considerably, and this has all helped to muddy our understanding of just how we can avoid lag when it isn't obviously in sector, graphic related.

Even with the 450 factory size NEW complex, once I removed all NPC stations, ships, asteroids, debris, planets etc, my framerate in an empty sector doubled, from ~400 to ~800. Yet, doing the same with the old style 450 factory complex took it from 69 FPS to a quite unimpressive 74 FPS. Understand that what I'm comparing is one game that forked at a particular save (around 1.5 playing hours), so I was careful to ensure that nothing else affected the results. The actual complex builds were done in a couple of hours due to use of scripts and macros.

Please note that all benchmarks were performed with an empty generic missions file.

Regarding generic missions, there's still a lot of talk about it lagging games. Pretty amazing, considering that the big fix for these was released in January 2012. I emailed Bernd about the generic missions bottleneck and how to reproduce in April 2011. As they were busy on Rebirth at the time, it didn't get top priority for nearly a year. At that time, Rebirth was supposedly going to replace X3 and thus performance issues would disappear with X3 behind them. So you can understand their priorities at the time.

I may revisit the generic missions benchmarking again, but I'll wait until people have had a chance to use the new CCK method. We may then find that the talk about generic missions diminishes.

Joe McCracken
Posts: 397
Joined: Sat, 22. Aug 09, 18:55
x3tc

Post by Joe McCracken » Thu, 18. Feb 16, 17:12

Wow. Thanks for your research Glen. I will have to try it out.

This can be very useful. 8)
I'd like to shoot you in the butt with a EBC gun, hide in the asteroids and laugh at what I done, put a blood blister upon each bun, I'd like to shoot you in the butt with a EBC gun.
Some people are like slinkies, not worth a whole lot, but they bring a smile to your face when you push them down the stairs.

MrFiction
Posts: 215
Joined: Sat, 7. Jul 12, 19:16
x3ap

Post by MrFiction » Fri, 19. Feb 16, 10:53

Amazing info, just what I needed for my TC complexes. Well, I assume this works the same for TC.

patient zero
Posts: 329
Joined: Sun, 25. Mar 07, 19:19
x3tc

Post by patient zero » Fri, 19. Feb 16, 20:47

Thanks for the info.
Sounds good for new complexes but probably can't fix my old complexes.
Superimposed stations would explode before they could be connected.
This is only a virtual reality.

glenmcd
Posts: 920
Joined: Sat, 16. Oct 10, 11:07
x3tc

Post by glenmcd » Fri, 19. Feb 16, 22:17

patient zero wrote:Thanks for the info.
Sounds good for new complexes but probably can't fix my old complexes.
Superimposed stations would explode before they could be connected.
In the save game linked to above, all 1020 factories are superimposed. They're all inside an asteroid. Just have a look at the sector view and you'll see where everything is. Visually you can only see one Complex Hub and the usual asteroids in the sector. Placing factories inside asteroids didn't seem to give me any advantage - it was a test to see if my video card could retain a higher FPS in sector. You can layout the factories however you like, it won't make any difference to OOS performance.

With superimposing, try building exactly 3 factories before you connect two of these with CCK. So there's always one left over so you can easily line up the next two. When you need to go for more factories/CCKs do so when there's one factory unconnected not 3. Apart from the Complex Hub that will remain after everything is connected, the other complex hubs can go anywhere. Don't superimpose them, no point. They'll disappear as you connect the hubs together.

fireanddream
Posts: 384
Joined: Sun, 13. Dec 15, 07:15
xr

Post by fireanddream » Sat, 20. Feb 16, 04:08

I just tried your method on a 32-station complex last night... Well, 32 stations won't hit my fps that bad either way. I get the binary sorting and why it minimizes the lag, but for casual gaming the most "super" complex I'll be will probably have 80-90 stations, if I just connect them into 2-station complexes and then paste all the mini complexes together the lag should be drastically reduced, right? It's like I'm having ~40 "levels" instead of ~80, not optimal, but enough?

The optimal method is great but I messed up a few times, miscalculated the number of stations or ended up with the last two complexes too far from each other.

glenmcd
Posts: 920
Joined: Sat, 16. Oct 10, 11:07
x3tc

Post by glenmcd » Sat, 20. Feb 16, 07:46

fireanddream wrote:I just tried your method on a 32-station complex last night... Well, 32 stations won't hit my fps that bad either way. I get the binary sorting and why it minimizes the lag, but for casual gaming the most "super" complex I'll be will probably have 80-90 stations, if I just connect them into 2-station complexes and then paste all the mini complexes together the lag should be drastically reduced, right? It's like I'm having ~40 "levels" instead of ~80, not optimal, but enough?
Ignoring cache miss issues, you'll get a quartering of required CPU load to maintain wares in the complex for each level you make binary balanced. If doing so brings all operations to within cache size on your particular CPU, then you can multiply the benefits by something like ten times. With the 80 factory example you suggest, it would do exactly that on my system. It could be a good compromise for the casual gamer if binary balancing all the way is too difficult. There is no difference in cost either way.

MrFiction
Posts: 215
Joined: Sat, 7. Jul 12, 19:16
x3ap

Post by MrFiction » Sat, 20. Feb 16, 11:22

@glenmcd: two questions
1. When you say load a saved game, do you mean loading from the main menu to the game? Or after you've loaded the game and the time the controls don't work for a while? Because on my system each game load from the main menu takes about 20 seconds. This is on a high end system with an SSD. Every savegame, whether 10 seconds in or 200 hours takes about the same time from the main menu.

2. Loading your example save I get more stuttering, also in other sectors. Maybe that's because the neighboring sectors contain a lot of objects or do you think stuttering is inevitable with these complex sizes?

glenmcd
Posts: 920
Joined: Sat, 16. Oct 10, 11:07
x3tc

Post by glenmcd » Sat, 20. Feb 16, 13:31

MrFiction wrote:@glenmcd: two questions
1. When you say load a saved game, do you mean loading from the main menu to the game? Or after you've loaded the game and the time the controls don't work for a while? Because on my system each game load from the main menu takes about 20 seconds. This is on a high end system with an SSD. Every savegame, whether 10 seconds in or 200 hours takes about the same time from the main menu.

2. Loading your example save I get more stuttering, also in other sectors. Maybe that's because the neighboring sectors contain a lot of objects or do you think stuttering is inevitable with these complex sizes?
1. If you are playing a game and you simply load a saved game without going to menu first, then X3 does this:
a) fades to black
b) unloads the current universe. With classic complexes, this takes a long time. Not quite as long as loading though, maybe two thirds as long.
c) You'll see the loading screen. At this point, the old universe has been unloaded and the new one is being loaded

You can see these happen separately by first going back to menu, and then from menu you load a save. When you go back to menu, it does some loading of the menu itself as it is kind of a little 3D universe or at least a sector.

With loading X3 saves, it's latency that counts with your drive not bandwidth. My SSD reads at a sustained 1.7 gigabytes per second. But it doesn't load a 20MB X3 save in eleven milliseconds. If your SSD is on a (SATA) cable, it has latency in the low single digit milliseconds. Let's use 4ms for example. So each 4KB block that X3 reads, incurs at least two of these 4ms penalties. 8ms for each 4KB block works out to be one megabyte per second. That's ignoring the actual reading time for your drive. So for a medium sized (20MB) X3 save file, it's going to take at least five seconds from a SATA SSD regardless of its bandwidth. PCI Express SSDs can do it in under 9 seconds because of the low latency. So for same save, the 4KB blocks X3 reads sets a hard limit on mine to 40MB / second, which for a 20MB file is becoming insignificant. If the X3 code instead used say 256KB blocks for reading, loads and saves could possibly happen in less than a second even with a clunky old HDD.


2. I chose just over a thousand factories for the save as I thought that would seem impressive. But it isn't if you can't appreciate or measure how bad performance would be in a complex of that size made the old way. I think the lesson here is for me to make another demo save, but with just say 250 factories in it. It will run smooth and for many players be comparable complex wise with what they have in their game. But then there's the generic missions to consider..

Stuttering also comes from the spawning of generic (NPC station) missions. Virtually every time you change sectors, you incur additional lag as generic missions spawn. Around one to two hours after any individual mission spawns at a station or ship, that mission will dissapear and its individual contribution to lag will go also.

Unless you disable all generic missions, there's no point in measuring lag coming from factory complexes or from war sectors or whatever. During benchmarking I changed sectors 200 times in half an hour, dropping my framerate from 900 FPS to 2 FPS. That's with zero player factories. In a separate game, I built one hundred thousand (100,000) factories in a single sector, and this took my framerate initially from 830 down to 90, but then settled out at an average of around 300 FPS. All factories had resources and were producing. At 300 FPS, flying is smooth even in a Kestrel. In one test of generic missions, the very first sector swap I did caused framerate to drop by over twenty percent. In other test, it halved over the first five sector swaps. Limiting to "Destroy Convoy" type, in 12 sectors framerate went from 812 FPS to 97 FPS. That's probably the worst of them. This stuff is complicated, and trying to do something with it requires isolation of individual causes. We can achieve the performance we seek, but one needs to appreciate that unacceptably low performance in TC/AP has more than one cause. The above allows you to eradicate one of the causes. Not all of them.

i7 CPU, GTX-660 Ti:
0 factories: 830 FPS
1,000 factories: 800 FPS
2,000 factories: 760 FPS
5,000 factories: 680 FPS
10,000 factories: 620 FPS
15,000 factories: 520 FPS
20,000 factories: 400 FPS
30,000 factories: 280 FPS
40,000 factories: 190 FPS
50,000 factories: 140 FPS
100,000 factories: 90 FPS initially then averaged 300 FPS

MrFiction
Posts: 215
Joined: Sat, 7. Jul 12, 19:16
x3ap

Post by MrFiction » Sat, 20. Feb 16, 15:02

@glencmd: thanks for your hard work! Never had a complex with over 150 factories (yet) so I'll believe you when you say performance will be very bad the "old way".

Interesting stuff btw, I'm going to construct all my future complexes the way you described, because my current game is the one I'm planning on playing for a few years.

One last thing, I think your solution would also work if you did:
1 50 factory complex connected the old way
2 50 factory complex connected the old way
3 50 factory complex connected the old way
4 50 factory complex connected the old way
5 50 factory complex connected the old way

My theory is that connecting those 5 complexes together would also work like your solution because the exponent will never reach high numbers for the 5 smaller complexes. Performance will be worse than your solution but still very good (I suspect)

User avatar
TTD
Posts: 11165
Joined: Sun, 6. Jul 08, 10:29
x4

Post by TTD » Sat, 20. Feb 16, 21:38

Just thought I would look in the forums after a long absence.

It is nice to see you are still investigating the game's mechanics and publishing your results with good advice. :-)

I might try this idea when i restart playing again.

as for it being a 5 year old game, there are a number of games out there that have a very large following and been kicking around for 10 years or more

I have to say that X£ was the best and is the best in the series for me, so far.

I just can't get enthusiasm for Rebirth

glenmcd
Posts: 920
Joined: Sat, 16. Oct 10, 11:07
x3tc

Post by glenmcd » Sun, 21. Feb 16, 01:33

MrFiction wrote:@glencmd: thanks for your hard work! Never had a complex with over 150 factories (yet) so I'll believe you when you say performance will be very bad the "old way".

Interesting stuff btw, I'm going to construct all my future complexes the way you described, because my current game is the one I'm planning on playing for a few years.

One last thing, I think your solution would also work if you did:
1 50 factory complex connected the old way
2 50 factory complex connected the old way
3 50 factory complex connected the old way
4 50 factory complex connected the old way
5 50 factory complex connected the old way

My theory is that connecting those 5 complexes together would also work like your solution because the exponent will never reach high numbers for the 5 smaller complexes. Performance will be worse than your solution but still very good (I suspect)
I agree, what you suggest is much better than using the old way entirely and may draw perhaps 20% cache misses compared to 99%. There are people that would choose a GM over a Ferrari even when they are priced identically. No justification is necessary as long as they don't get to thinking that their complaints will be listened to later. If they bought before the cheap Ferraris came on the market, they got the best available at the time.

If a player already has five 50-factory complexes, connecting these together the old way won't be as bad as connecting all 250 together the old way.
[Update 3PM: On thinking about this again I'm not so sure. Need to test each way to know for certain.]

I get these figures:
Old: (99% cache miss) = 624375

Compromise above: (20 % cache miss) = 39700 = 15 times less load

Using 10 X 25-factory complexes (5% cache miss assumed) = 10875 = 57 times less load

Completely balanced: (1% cache miss) = 2200 = 283 times less load

Yesterday I made this observation: lag coming from factories affects equally FPS, gate passing time and game loading time. Lag coming from generic mission issues affects framerate well before any effect on gate passing and game loading times. But even with 2 FPS, if the lag is coming from generic missions, game loading time might double to triple whereas with CCK lag it could be 20 to 50 times slower.

So if you benchmark your FPS, gate passing (use empty or near empty sectors for this) and game loading (use a save done within minutes of new game) times in a newish game, compare them to benchmarks measured in your mature game. This should give you an indication as to what's affecting performance in your current game.
Last edited by glenmcd on Sun, 21. Feb 16, 06:40, edited 1 time in total.

glenmcd
Posts: 920
Joined: Sat, 16. Oct 10, 11:07
x3tc

Post by glenmcd » Sun, 21. Feb 16, 02:15

TTD wrote: It is nice to see you are still investigating the game's mechanics and publishing your results with good advice. :-)

I have to say that X£ was the best and is the best in the series for me, so far.
Thanks TTD. By chance I read some of your older posts yesterday and got to hoping I would see you again on here. I remember we had some great conversations on here some time back.

I've had really poor internet for a while but am moving house within a week so will be more active on here after that. One really needs youtube for good research!

One has to wonder how sales compare between X3 and Rebirth. Ratings and reviews are still quite bad for Rebirth.

It's said that Rebirth is being improved, but obviously there's still potential for X3 to be further polished and added to also. What you can do in X3 modding really does it for me!

As Egosoft must watch their bottom line I wonder whether they're thinking of moving some of the focus back onto future X3 versions. Who knows, there may be an X4 after all :)

kgkosio
Posts: 602
Joined: Sun, 8. Feb 04, 07:47
x4

Post by kgkosio » Sun, 21. Feb 16, 03:54

Thank you, great research.
Fan of the save game manager by badger
Best Steam DiD death: Fly into a TL repeatatly while repairing it

Cursed Ghost
Posts: 637
Joined: Sat, 2. Oct 04, 22:44
x3tc

Post by Cursed Ghost » Sun, 21. Feb 16, 13:13

this an interesting read a little over my head but interesting all the same

i do have to wonder though why even do complexes the way it is right now? why not just do away with the whole idea of complexes in there current form and simply implement a system that is much cleaner and simpler which doesn't incur this problem in the first place?

aka you build your complex hub you then you go in to the control panel and then just start adding production lines and for each production line you add you pay the associated cost just as you would if you where buying a new station and complex kit and then the station is updated to accommodate the new production line just the same way it would be upon adding a new station via a complex kit

what you end up with is something a kin to the multi food production factories found in Aldrin a single factory that has multiple production lines

all without having to connect dozens of stations together thus avoiding the associated performance issue plus the associated issues with actually trying to line stations up properly so you don't end up with spaghetti junction plus the issue of not being able to connect stations to the complex after the complex gets to big and the hub is to far away plus the general tedium involved in actually setting up a complex

the only stations this wouldn't work for are mines which would have to continue to use the current system but that's not a big issue.

is there any reason my way of doing this wouldn't work maybe some technical reason maybe ? if not then would it not simply be more sensible to crate a mod that would remove the current complex building system for all stations except mines and simply replace it with one that functions as described above.

glenmcd
Posts: 920
Joined: Sat, 16. Oct 10, 11:07
x3tc

Post by glenmcd » Sun, 21. Feb 16, 13:42

Many solutions have come forward from the modders to work around the complex lag issue. Litcubes Saturn Hub is one of the better ones. Some address OOS lag, some address IS lag, some both. There's at least one way of reducing some IS lag in the vanilla game and that's with the tubeless mod.

But for the OOS lag which increases exponentially with the number of CCKs used the current way, there's nothing in vanilla to help apart from keep your complex size down to a few dozen factories.

One good reason for posting my thread was to put forward an undeniable case to Egosoft that there's a (vanilla) bottleneck in there that if fixed by them will lead to a superior product for all. Better in a modded game, better in vanilla. Personally, I suspect that the solution you offer would lose a great deal of realism that is present in the vanilla game.

User avatar
TTD
Posts: 11165
Joined: Sun, 6. Jul 08, 10:29
x4

Post by TTD » Sun, 21. Feb 16, 19:17

yes...many ways to crack a nut.
Each person has ideas and favourite solutions.

Glen...I may get back here more regularly at some point, but am still enthralled by ARK: Survival Evolved.

But part of me misses this community. many people come and go. others linger. But unless RL forces it's hand, the best players remain active and so too the modders and moderators.

Your researches into game mechanics always gave me something to think about.
when i saw your name here I just had to read. :-)

glenmcd
Posts: 920
Joined: Sat, 16. Oct 10, 11:07
x3tc

Post by glenmcd » Mon, 22. Feb 16, 08:04

MrFiction wrote:Loading your example save I get more stuttering, also in other sectors. Maybe that's because the neighboring sectors contain a lot of objects or do you think stuttering is inevitable with these complex sizes?
It's taken me two solid days in AP to answer this one! I originally allowed generic missions to pollute the benchmarks and I should not have. I've re-written the original post and it now links to 24 different Albion Prelude save files. They are a mix of various complex sizes, old style and new style. There's even some hybrid versions. Lag coming from NPC stations and ships and generic missions has been avoided and thus testing results should be far more meaningful and accurately reflect what the complexes are doing to the X3 engine.

The only real surprises I got when performing the above benchmarks were:
1. Generic Missions are a real problem. Lag is very significant and "only few players are affected" is absolute crap. "only few players can accurately tell where the lag is coming from" is I believe an accurate statement.
2. Every NPC object (planets, asteroids, debris, ships, stations) in the universe when added up, produces OOS lag which is only barely measurable. IS is an entirely different matter.

Please let me know how your benchmarks compare in some of the saves?
Just stay within the UKS / Unseen Domain during..

Post Reply

Return to “X Trilogy Universe”