Super-sized complexes - without the LAG - for all

[MrFiction]

Wait, I thought the generic mission lag was fixed a long time ago in patch 1.1 or something like that?

edit: and isn't this lag temporary, i.e. the lag is gone when exiting and starting X3 again?

[Dome1]

Glenmcd , thanks for you hard work. I will be putting this method to the test and getting back to you .

[glenmcd]

Wait, I thought the generic mission lag was fixed a long time ago in patch 1.1 or something like that?

edit: and isn't this lag temporary, i.e. the lag is gone when exiting and starting X3 again?

"It" wasn't "fixed". Improvements were made that assisted specific mission types at the time. If you mean lag is gone when you give up current game and start a new one, sure it will reset to initial conditions. Reloading a save won't make GM lag go away. Try resetting the MD just before / after a reload to confirm this.

Following are some benchmarks focussing on generic missions I gathered in past week. Each run starts from the same save. It is possible to focus on one mission type even from same save, this is what I did. Started and ended in empty Unknown Sector, scripted FPS averaging test at start and end. Swapped sectors fast enough so that I could test FPS at end before the missions spawned disappeared. Actual path was UKS to Barren Shores and back. I could have spawned more missions had I not revisited sectors however players do often revisit sectors.

Only 11 sectors could spawn missions as first and last is unknown. Results follow. You'll notice that during these tests, the framerate always went up when no missions spawned, for whatever the reason. I've noticed this a lot lately, but you only notice it when you get rid of the major lag producers.

This block swapped sectors 12 times, last sector could not spawn missions as empty Unknown Sector. So effectively 11 sector swaps:
No missions: 815 FPS -> 903 FPS
No missions again: 816 FPS -> 893 FPS
Scan asteroids: 817 FPS -> 283 FPS
Scan asteroids again: 819 FPS -> 301 FPS
Dual Convoy: Didn't spawn and I don't know why. 816 FPS -> 899 FPS.
Dual Convoy again: Didn't spawn again. 814 FPS -> 899 FPS.
Escort Convoy: 813 FPS -> 99 FPS
Escort Convoy again: 821 FPS -> 110 FPS
Deliver Wares to Station in need: Didn't spawn. 822 FPS -> 894 FPS.
Deliver illegal wares to pirate station: Didn't spawn. 814 FPS -> 884 FPS.
Deliver illegal wares to trading station: 826 FPS -> 489 FPS.
Transport Cargo: 820 FPS -> 381 FPS.
Passenger Transport: 819 FPS -> 666 FPS
Assasination 1: 818 FPS -> 383 FPS
Xenon invasion: Didn't spawn (no Xenon sectors close) 817 FPS -> 903 FPS
Destroy Convoy: 812 FPS -> 97 FPS
Generic Patrol: 811 FPS -> 793 FPS
Defend Object: 816 FPS -> 387 FPS
Build Station: 824 FPS -> 660 FPS
Transport (one) Passenger: 806 FPS -> 519 FPS
Follow ship: 811 FPS -> 511 FPS
Deliver Matching Ship: 823 FPS -> 731 FPS
Freight Scan: 813 FPS -> 533 FPS
Repair Station: Didn't spawn. 812 FPS -> 900 FPS
Multiple Transport: Didn't spawn. 822 FPS -> 921 FPS
Tour of a lifetime: Didn't spawn. 804 FPS -> 906 FPS
Notoriety Hack: Didn't spawn as no pirate stations near. 846 FPS -> 906 FPS
Buy Asteroid Survey: 826 FPS -> 295 FPS
Buy sector data: 824 FPS -> 491 FPS
Buy used ship: 828 FPS -> 240 FPS
Buy Blueprints: 839 FPS -> 573 FPS

Escort Convoy only, one sector swap only: 817 FPS -> 564 FPS
Escort Convoy only, 142 sector swaps: 781 FPS -> 5 FPS
Escort Convoy only, 228 sector swaps (toured the universe): 805 FPS -> 2 FPS

All generic missions (original), 48 sector swaps: 825 FPS -> 211 FPS
All generic missions (original), one sector swap only: 823 FPS -> 632 FPS


So as you can see, framerate is significantly affected on spawning of any and all generic mission types. Those that failed to spawn at all always resulted in an increase in framerate, at least over the 12 sector tests.

When all generic missions are enabled, the resulting performance degradation is associated with the sum of the effects from each mission spawned. Because this happens using a combination of intentional chance setting as well as some randomness, the only way to properly tell what's going on is to focus on one mission type at a time. This doesn't eliminate randomness but it assists.

I finished the tests with allowing all missions to spawn as per vanilla. The results cannot possibly be accurate as randomness plays such a big part in determining which missions spawn, how many and when.

I do know that when all generic missions are disabled, that I can swap sectors hundreds of times without seeing this significant framerate degradation.

The tests I did measured more than framerate. They timed gate passing, compared game time and real time, counted frames in each sector passed and logged the lot. Unless you push the generic missions issue a LONG way, you won't notice longer gate passing times, nor longer gamesave loading times. It will happen eventually, but by then your framerate is down to single digits.

Performance degradation from our big old complexes is quite different in this respect. It doesn't grow with sector swaps or with time. It's a little more if you run out of resources but this is hardly significant. If anything, it's gate passing time that suffers first, but it's a close race with the old factory complex issue.

To put this in perspective, changing sector one time triggers the generic missions spawned (averaged) to place the same additional load on the CPU as building an additional ten thousand factories, all actively producing in parallel. That's more factories than any X3 game is ever likely to have. Even all the ships in the universe present only a fraction of the load OOS that GMs do, again with only one sector swap. This is the case even when the mission spawned is in some other sector and you didn't accept it or even ask about it. Each time you swap sector again, it adds to lag again. And again. And I'm not just referring to the GM types that are currently recognised as broken/worst.

The fact that most missions take back some of this lag after an hour or two does little more than discourage players from exploring the universe and encourage one to quit their current game. X3 has always been like this. That can change.

[MrFiction]

You sure put a lot of time into this research, very interesting. Two "bold" statements, which I think are true.

1. Reading the post about GM lag suggest that waiting in a sector for long enough will get rid of the lag (for a while). So, idling in an empty unknown sector at 10xseta for 10 minutes will clear most of the lag in the current game.

2. Disabling GM from the start and building complexes in pairs of 2, connecting those pairs together, and so on gets rid of the lag completely in a game.

[enenra]

I think people need to make an important distinction here. What is being talked about is not lag per se. What this is about is FPS reduction. And FPS reduction happens naturally as more is going on in a game.

Yes you are going to get more FPS when you don't have any generic missions. But then you also don't have any generic missions. The game will also run faster if you prevent any and all ship spawns in the universe. That doesn't mean you should do that.

The issues are:

a) Short time FPS reductions that are disproportionate to what is actually happening.

b) Long time FPS reductions which accumulate over time.

And here we have to bear in mind that short time reductions is absolutely normal just by nature of something actually happening. The part where it becomes problematic and really important to fix is with case b) because by the very nature of the missions being generic, they shouldn't have long-term impact on the game. If they do, something is going wrong badly and it probably means missions aren't being cleaned up properly.

[pref]

Thanks glenmcd, nice research.
Funny how we have to build the tree here, normally the CPU should handle such things
\still, i like even the issues of X3 more then XR's...

[glenmcd]

FPS reduction happens naturally as more is going on in a game.

This to me is in sync with long held views held (and often promoted) by Egosoft, many regular posters here (especially moderators) as well as many players that read the forums but don't actually do their own research. If what you are saying (that game requirements in TC and AP justify being overwhelmingly CPU-bound) is true that's fine. But what is "true"? Something that isn't yet proven as false, and this can in some cases take a long time to happen.

Well, it has happened.

I'm not quite done yet, but once Egosoft applies the fixes suggested in this thread and the post linked to above, the mistruths about performance in TC and AP will finally be laid to rest. And that's when I will take a rest also.

There is nothing in Terran Conflict or Albion Prelude that justifies framerates below 120 FPS
because of CPU load, in any sector and regardless of state or "maturity" of game or missions, on any modern gaming computer.

[enenra]

FPS reduction happens naturally as more is going on in a game.

This to me is in sync with long held views held (and often promoted) by Egosoft, many regular posters here (especially moderators) as well as many players that read the forums but don't actually do their own research. If what you are saying (that game requirements in TC and AP justify being overwhelmingly CPU-bound) is true that's fine. But what is "true"? Something that isn't yet proven as false, and this can in some cases take a long time to happen.

Well, it has happened.

I'm not quite done yet, but once Egosoft applies the fixes suggested in this thread and the post linked to above, the mistruths about performance in TC and AP will finally be laid to rest. And that's when I will take a rest also.

There is nothing in Terran Conflict or Albion Prelude that justifies framerates below 120 FPS
because of CPU load, in any sector and regardless of state or "maturity" of game or missions, on any modern gaming computer.

Performance reduction happens as the player builds more stations, has more ships, etc. simply due to the fact that there's more objects than before.

I'm not quite sure how OOS works for sectors that have not been discovered yet but I could imagine that with uncovering more and more sectors the performance will also reduce somewhat.

There is optimization potential, absolutely, and there's probably also some memory leaks. But it's just simple math that more (player-owned) objects in the universe will decrease FPS to some degree.

[glenmcd]

But it's just simple math that more (player-owned) objects in the universe will decrease FPS to some degree.

I'm actually using my experience in software engineering and three decades of research into how software executes to measure performance, find bottlenecks etc. I've reverse engineered thousands of applications of all sizes including games, generated millions of lines of assembler code, tested as assembling and executing as per the original, then profiling and speeding up, many times to dozens to hundreds to thousands of times faster than the original. As well as fixing known bugs and bugs I found during these processes. My favourite language to reverse engineer was C++ due to how incredibly wasteful it is and how it promotes exponential increases in CPU loads using simple, brief and good-looking source code lines. Classic case of shielding the user from things he needs to know.

I assemble with my own assembler, written in 100% hand coded assembler. Every tool I have used in this work, I hand coded in assembler. That includes a number of profilers. As well as dozens of tools that don't have names because I conceived and developed them myself. All of these had a focus on software execution measurement and efficiency. All up, a bit over ten millions lines of code. There's nothing between C code and silicon in a computer that I'm not intimately familiar with in conceptual terms. I did hardware courses first but switched to software as I enjoyed it so much more.

I haven't reverse engineered in many years, but my experiences allow me now to make reasonably accurate predictions based on simply running and using an application. X3 is gifted with a decent scripting language and this has given me what I needed to automate the process of benchmarking, testing and proofing these games that I enjoy playing.

A few months after I bought Egosoft's superbox, I realised that Terran Conflict was seriously suffering in performance in certain areas and that it was almost unique as a 3D game that was CPU bound. I communicated my thoughts to Egosoft. I also used what I could work out about the issues to make my own game more enjoyable. That's when I disabled both Convoy generic missions - early 2011. I didn't know why they were so bad at the time, but know more now. In between, I've learned the scripting language and had a lot of fun, mostly playing Albion Prelude.

When I discovered for myself the large factory complex CCK issue, I initially switched to using Complex Cleaner, but eventually wrote my own CCK code. After a few re-writes and revisions, it successfully and efficiently joined tens of thousands of player factories which in most of my mature games, used up every asteroid in the universe in a balanced "complex" that spread across most sectors in the universe. So I knew years ago that even when you take the basic CCK code requirements and execute it in a higher level language (X3 scripts instead of partly within the AP executable), that it was possible to grow the player empire pretty much as big as it could go given the initial asteroid quota, without hitting any CPU availability walls.

All that's required of Egosoft is to choose appropriate algorithms and data structures as well as effective testing which includes performance topics. Obvious to some, black magic to others. You are of course correct in your statement that adding ships to the universe adds some CPU load. You would be incorrect if you are suggesting that this is happening at a level that could be called significant or noticable. If so interested, use selective disabling of GMs to spawn escort convoy missions in a few dozen sectors. Save game. Then reset the MD and you'll find that all of those escort ships can still get to their destinations with the FPS meter in the high hundreds. Reload game, destroy or destruct these ships without touching the MD and you'll see framerate remain at pathetic, until finally the MD catches on some ten minutes later. Adding a few dozen or even few hundred ships is a PITO.

If you don't understand - and (no offense meant) you appear not to - I suggest just sit back and watch as the changes happen. Now that players know for sure that top performance can be had out of X3, there is no turning back. Egosoft no longer have a choice and they no longer have an excuse. Unless you call abandoning X3 an excuse. Fairly recently Bernd was asking players for guidance on further work on X3. Knowing that good performance truly is possible and achievable, this will further encourage the decision to develop and release revisions of TC and AP. Any delay means more players becoming aware of what they can have but don't have.

When do we want it?

[enenra]

Oh don't worry gelnmcd, I do recognize someone who knows what they're talking about and I don't doubt that you know a lot more about this than me.

I wasn't aware that increasing the amount of objects in a sector does that little to the framerate. That's certainly news for me.

As for MD, that's more my area of expertise. However, from what you describe this might actually be an issue with the implementation of the language itself to a certain degree. Not sure tbh. There's definitely a lot of optimization potential in the MD code itself as well. The thing is that actually a lot of the current basic MD code was written while the language MD was still not quite finished yet. So there's a lot of really old code in there that may be really unoptimized or even not working at all. Sadly, rewriting all of that is... not something that can be done that easily now.

I've personally toyed with the idea and have started a complete rewrite a couple years ago which I've been working on now and then. That's promising but until all of that would be implemented and tested I'd need a lot more time. Like a year plus of working on it full time kind of time probably.

Updating basic MD files would also create issues with savegame compatability which is another issue Egosoft will be limited in. We'll see what can be done. The person to talk to about this is actually probably CBJ. I believe he knows the most about the actual implementation of MD.


Have you looked at the actual MD code at all during your tests? I feel if we could work out the places where performance is most degraded by we could probably narrow down the reasons for the unreasonable decrease in performance. I could well imagine that some of the conditions used are not really properly optimized for what they do, for example.

By the way, consider signing up for the DevNet through your profile if you haven't already. That's usually an a bit more direct line to the devs.

[glenmcd]

Thank you.

>> I wasn't aware that increasing the amount of objects in a sector does that little to the framerate.
To some degree this is a hunch of mine but partly observation too. I'll come back with some numbers. Pretty sure they'll be an eye opener for some.

With MD it seems that there is enormous resistance against doing a rewrite or anything major on it. My contacts so far have been Owen and Bernd. So my post in the tech support thread was focussed on an Egosoft stamped workaround that would require only days to weeks and perhaps two months including testing before a release that would be save file compatible. Here's what I had in mind (for someone else to do lol):

1. Don't allow Convoy missions to spawn. No problem with save compatibility here. That gets rid of perhaps 50 percent or more of the framerate problem.
2. Get rid of "buy asteroid survey" offered from flying ship in space. We already can do this from a station mission, and it behaves itself. A mission that doesn't spawn doesn't bother us with cleanup issues.
3. Two generic mission types need to be fixed to respect quota. I don't really spend any significant time on MD stuff but I can't imagine this could be too hard. Surely must have been the intent. These mission types are ones that players would miss (freight scan, buy blueprints).
4. Set quota in globals.txt to 1, for all four items. 20 second job and except for those missions that ignore it, is working fine in my tests.
5. Something I was toying with today: At the moment, most missions cleanup after four sector swaps. With all above done, performance is pretty good, however halving this to two sector swaps would give significantly higher resistance on slow machines to ridiculous framerate drops when other stuff is added on: war sectors, Albion Gamma etc.

I see the GM stuff as being far easier and quicker to get to release stage compared to the full fix on Complex CCKs. Complex CCKs does have a viable vanilla workaround, GMs don't. GM fixes will be felt by all and in all games, new or mature. Those that thought performance was already okay will get a nice surprise. Players that build the biggest complexes and hit the CCK FPS wall are also those most likely to visit these forums and see the workaround - or be pointed to it.

I've been on Devnet a few years, but TBH haven't got much out of it. It seems that one voice isn't enough to get some things done, but many voices - which are found here in the open forum - are listened to.

I do have time on my hands at the moment, and would be willing to see what I can do with MD while retaining compatibility. Understand that the only edits I've done to MD files were done in a text editor and mostly restricted to flipping individual bits or bytes or commenting stuff out. Not exactly ideal for XML I imagine. I'm no noob so no need for detailed coaching, perhaps just mentioning the best app to use for editing XML and any "gotyas" I'm likely to come across should suffice.

You speak of "optimization potential" as something that's a big job. It doesn't need to be. I've found that if you spend more time working out the smallest changes possible to make the biggest impact possible, that it's the best way to go.

If you can point me in the right direction with MD I'll see if I can put my workaround into MD code. I dislike rewrites unless really necessary. If I can flip a bit and get a good result, that's what I prefer. I got the impression from my workaround above that especially given the maturity of X3, it may not be necessary with the MD.

[glenmcd]

I wasn't aware that increasing the amount of objects in a sector does that little to the framerate. That's certainly news for me.

As promised, here are some numbers comparing CPU load with and without ships throughout Universe.

Albion Prelude 3.1
All asteroids, planets and debris removed from universe.
Computer used was hand built with fastest available parts 3.5 years ago, not overclocked.

In UKS East of Unseen Domain. All framerate measurements in here, same position same rotation. I converted FPS to time between frames, assuming that most if not all of this was required to generate the frame.


Baseline:
Vanilla jobs.txt, around 8090 ships.

As framerate was >1000 without missions, in both tests I first generated 2 missions of same type (Passenger Transport). This brought the FPS down below 1000. Dozens of tests using same mission types confirmed repeatability of FPS.

Framerate with 8090 ships: 780 FPS = 1282 microSeconds / frame
Framerate with empty Jobs.txt (no ships): 882 FPS = 1133 microSeconds / frame
Difference is 1282-1133=149 microSeconds per frame.
That's 18 nanoSeconds per ship average added to each frame time.
At 60 FPS, it would take twice this overhead just to drop one FPS.
At 1200 FPS, it's enough to drop by 150 FPS to 1050 FPS.

Comparison with generic mission overheads:

Each generic mission that spawns ("on offer") with exception of Convoy types, adds roughly 79 microSeconds to each frame time. With two generic missions on offer somewhere/anywhere in the universe, you've already added more time to each frame than all the ships in existence at the start of an AP game, and that's throughout the entire universe. We're still a number of times short of the GM overhead likely experienced in my workaround linked to above. And this in turn is dozens of times less than that experienced in most AP games today, once the player has explored half the universe or more.

Others? I found generally that ships add more overhead to (OOS) frame time than asteroids, planets, debris and even stations. Without the CCK issue, player factories will not even come close to the overhead that 8090 ships add. And it's still hardly even worth a mention.

Summary: When it comes to OOS performance in TC/AP, only large complex CCKs and generic missions matter at the moment. Truly, just forget the rest until these are fixed.

[MrFiction]

...until these are fixed.

Not likely, time to move on to other games

[enenra]

Thank you.

>> I wasn't aware that increasing the amount of objects in a sector does that little to the framerate.
To some degree this is a hunch of mine but partly observation too. I'll come back with some numbers. Pretty sure they'll be an eye opener for some.

With MD it seems that there is enormous resistance against doing a rewrite or anything major on it. My contacts so far have been Owen and Bernd. So my post in the tech support thread was focussed on an Egosoft stamped workaround that would require only days to weeks and perhaps two months including testing before a release that would be save file compatible. Here's what I had in mind (for someone else to do lol):

1. Don't allow Convoy missions to spawn. No problem with save compatibility here. That gets rid of perhaps 50 percent or more of the framerate problem.
2. Get rid of "buy asteroid survey" offered from flying ship in space. We already can do this from a station mission, and it behaves itself. A mission that doesn't spawn doesn't bother us with cleanup issues.
3. Two generic mission types need to be fixed to respect quota. I don't really spend any significant time on MD stuff but I can't imagine this could be too hard. Surely must have been the intent. These mission types are ones that players would miss (freight scan, buy blueprints).
4. Set quota in globals.txt to 1, for all four items. 20 second job and except for those missions that ignore it, is working fine in my tests.
5. Something I was toying with today: At the moment, most missions cleanup after four sector swaps. With all above done, performance is pretty good, however halving this to two sector swaps would give significantly higher resistance on slow machines to ridiculous framerate drops when other stuff is added on: war sectors, Albion Gamma etc.

I see the GM stuff as being far easier and quicker to get to release stage compared to the full fix on Complex CCKs. Complex CCKs does have a viable vanilla workaround, GMs don't. GM fixes will be felt by all and in all games, new or mature. Those that thought performance was already okay will get a nice surprise. Players that build the biggest complexes and hit the CCK FPS wall are also those most likely to visit these forums and see the workaround - or be pointed to it.

I've been on Devnet a few years, but TBH haven't got much out of it. It seems that one voice isn't enough to get some things done, but many voices - which are found here in the open forum - are listened to.

I do have time on my hands at the moment, and would be willing to see what I can do with MD while retaining compatibility. Understand that the only edits I've done to MD files were done in a text editor and mostly restricted to flipping individual bits or bytes or commenting stuff out. Not exactly ideal for XML I imagine. I'm no noob so no need for detailed coaching, perhaps just mentioning the best app to use for editing XML and any "gotyas" I'm likely to come across should suffice.

You speak of "optimization potential" as something that's a big job. It doesn't need to be. I've found that if you spend more time working out the smallest changes possible to make the biggest impact possible, that it's the best way to go.

If you can point me in the right direction with MD I'll see if I can put my workaround into MD code. I dislike rewrites unless really necessary. If I can flip a bit and get a good result, that's what I prefer. I got the impression from my workaround above that especially given the maturity of X3, it may not be necessary with the MD.

Your suggestions certainly will get rid of the problems you've found but there is the issue of changing gameplay quite significantly by influencing the amount of missions available or completely removing a mission type. Those are not really the kind of solutions that we'd want to implement into a game, or only if there really is no other option.

1. The issue here lies with what the mission does. I haven't looked at its code outside of a cursory glance but there must be something that's going terribly wrong for it to have such an impact. You mentioned before that the ships of these missions take ages to despawn, but just them existing can't really be the issue considering other missions handle way more ships than these. I'd assume there's either something wrong where its ships don't get cleaned up properly or way too slowly, although again, ordering ships to fly to a station where they are then despawned shouldn't have as much of an impact.

On the top of my head there's a couple things that could be done if the issue lies with ships not despawning quickly enough: a) Make sure to not use really slow ships for convoys so that they can reach their destination in a more reasonable timeframe, be that the mission target or the station where they are to be cleaned up at. b) Spawn convoy ships docked at a station. That way they can just be destroyed if the player doesn't accept the mission and don't have to first fly to a station. c) Additional rule where if the player is not in the sector and there is no player owned object near (hence not in radar range), instantly despawn ships.

The issue might also be in the way that conditions are used inside the code. Sometimes there's delays missing inbetween checks, etc. This is something that has to be analyzed with the MD debugger.

2. Seems reasonable. Doesn't really result in a loss of content either. Though again I'd be curious what exactly is the issue here and whether it could be resolved without removing the ships completely. There ought to be something very wrong with its handling to cause such an impact.

3. To make a mission respect quota is indeed not a big change, but I fear the issue lies more in a buggy check rather than it not having the code for it. Not sure.

4. That would influence the amount of missions available quite drastically and therefore alter gameplay. Can't imagine any player doing a lot of missions would be happy about this. IMO not really a viable solution. Need to find and fix the bugs instead. And figure out why multiple instances of the same mission have such an insane effect. (likely outside of what can be affected in the MD itself)

5. From what I remember the cleanup is not necessarily dependant on sector swaps but on mission offer timeouts. Not sure though. In any case though that is certainly something that could be looked at since the gameplay impact won't be large.


Devnet lvl 3 is not much atm. Well, not since XR testing has been made available to everyone. What you might want to consider is signing up for lvl5 by signing the NDA. lvl5 gets more access into the development, earlier betatests, and is also able to contribute to patches etc.

What do you need regarding MD? I can help you set it up but it's really not much more than extracting the files. Pretty much any text editor will do that can read the xml stylesheets that you'll find in the same directory. I've been using Visual Studio which has worked well for me. I'm pretty sure Sublime works as well. Also: mdfiles.txt lists (if present) all files that are loaded by the game. That's pretty much it.

I agree that relatively small changes can make a big difference but it's really not good practice to remove functionality from the game in the process of doing so unless it can absolutely not be helped. ^^

Thanks for the object analysis btw. though I gotta say I was actually more thinking about specifically player ships, since they run on a different set of scripts etc. than the ships spawned by the Jobs file.

[shanrak]

So I've been making complexes using this method in my new game ever since I saw this. I have a 128 station energy loop in savage spur, and a few 80+ station complexes in other areas. So far load times seems to be great, haven't noticed much increase during jumping at all.

The only downside of this method is that its a huge hassle while connecting the stations:

[ external image ]

Connecting stations the old way was much easier

[glenmcd]

The only downside of this method is that its a huge hassle while connecting the stations:

Connecting stations the old way was much easier

Thanks for the feedback @shanrak. I must agree with you that knowing which complex hubs to connect next is not straight forward. But here is an idea that may ease this task a little:

1. Joining factories initially is easy so do all of these first. One of these must be positioned carefully as it will become the one that ships will dock at (the only surviving complex hub).
2. Before you join any complex hubs together, rename ALL complex hubs, adding a string of digits to the start of current names, where the numbers are incrementing. So instead of all being "Your Complex Hub alpha", the first would be "100Your Complex Hub alpha", the second "101Your Complex Hub alpha", the third "102Your Complex Hub alpha" etc. If you plan a really huge complex, use a 4 digit sequence. The hub you select to be first in this list should be the hub that you positioned carefully in step (1) above. See below for reasons.
3. Because all hubs now have unique names, they will no longer jump around in sector lists as well as when asked which things to connect using CCKs.
4. Connect the two hubs at start of list of hubs.
5. The first in the list now will be the first hub you selected in step 4 above and it will contain 4 factories. Rename this hub, adding a "z" to the start of the name. So "100Your Complex Hub alpha" would change to "z100Your Complex Hub alpha". It will drop to end of list at this point.
6. Repeat steps 4 and 5. Eventually, all the hubs containing 2 factories will instead contain 4, and all will have names that start with "z". Next, it's actually just as simple as repeating steps 4 and 5 again until you get down to just one hub.

Ensuring that the correct hub becomes the final one:
This is easy. Choose which hub you want to be the one that you dock at, and make this the one that is first alphabetically. In above example, this would be named "100Your Complex Hub alpha". When you select hubs to connect, the first one you select survives, the second is consumed. If when you connect "100Your Complex Hub alpha" it's always the first selected, then it will survive all connects and thus be the last remaining hub. The only real gotya here is when you have an odd number of hubs to join in a particular level. This won't be the case if your initial number of factories is an even power of 2 (32 factories, 64 factories etc). No problem, just be aware as you approach the end of each level of connects.

[Cursed Ghost]

Personally, I suspect that the solution you offer would lose a great deal of realism that is present in the vanilla game.

Perhaps but I would think this is one of them situations where you would want to make a small compromise for better performance and better simpler game play for players because I know that is something that a lot of players complain about particularly those new to the series and even as a veteran of the series I have to say they have a point there are a lot of elements which could do with being simplified for the sake of improved game play

Yes it detracts a little from the realism but you have to ask your self the question is it not worth sacrificing a little realism for improved game play especially when those changes not only improve game play by removing unnecessary complexity which really doesn’t add very much to the game anyway but also removes the very performance issues you are positing about

Now I don’t know about anyone else but personally I’d say that’s a trade worth making as long as it’s handled properly

don't get me wrong i realize that its not always so simple and depending on the nature of the issue it could well be one of those things that would require a massive rewrite of code which simply isn't practical because of the time and cost involved having said that however if they just wrote clean efficient code and properly tested stuff to make sure it works correctly before releasing it then this wouldn't even be an issue in the first place

now i know how that sounds and I don't know maybe if its just me but i was always taught to take my time and do it right first time and don't release work until I'm satisfied it's done correctly.

admittedly in the real world where you have dead lines and schedules etc you don't always have that luxury having said that that's no excuse for taking short cuts and not doing what you are supposed to do this is why so meany games these days come out so bug ridden that they are completely unplayable Stalker Shadows of Chernobyl being a prime example that game was so bug ridden that it crashed continually was unplayable because it.

although to Ego's credit they are for the most part good at fixing stuff when it doesn't work right which is more than can be said for a lot of other game developers who simply wont fix things even when they know its broken because they are only interested ££££ and once they have it they don't care anymore.

[Rabiator der II.]

@Glenmcd:
Thanks for putting in the effort to research this issue .

And without having done the research myself, I strongly suspect that Egosoft could solve the performance problem in complexes by picking a better algorithm. Simply because the first idea that comes to mind (going through a linked list of the connected stations) should require time roughly proportional to the number of connected stations. That is without spending time on finding a better solution, such as aggregating all station in the hub into one super-station for resource calculations.

How Egosoft managed to end up with exponential time is beyond me.

Edit:
I've reread your post from the beginning, and if the vanilla algorithm looks up every factory in the linked list in a separate search, with going through the list again for the next factory, an O(n²) complexity of the algorithm would result. But that would be stupid indeed .

Edit: corrected my Big O notation

[glenmcd]

I've reread your post from the beginning, and if the vanilla algorithm looks up every factory in the linked list in a separate search, with going through the list again for the next factory, an O(n²) complexity of the algorithm would result. But that would be stupid indeed

That's the only thing I can think of that would be happening. Although to us (outsider devs and ex devs looking on) it certainly looks like a stupid choice of algorithms, you have to consider that at the time of choosing, computers were much slower graphically, and nobody would have expected us to be building complexes as big as we do now. The graphic adapters of the day would have made frame-rates so low after say 200 factories were in sector, that building an even bigger complex would have been impossible. As graphic adapters are much more capable now, we've finally come to moving the stress onto other components in the X3 engine. Not just for some, but for all players that can afford very capable graphic adapters. In 2016, I'd say that would be the majority.

[RAVEN.myst]

I'd like to add my voice to the many others thanking you, glenmcd, for your stellar (heheh!) work on this - it is greatly appreciated!

(Your research is particularly well timed for me, as I am now almost done with my latest visit to Rebirth-land and nearing my inevitable return to X3AP - armed with a constructively different way to do something!)

I hope that your efforts nudge ES in the right direction - you've proven that certain significant (to say the least!) performance improvements can be effected with relatively little investment of effort.