EnglishGermanFrenchRussianPolishItalianSpanish
Log inRegister
 
Super-sized complexes - without the LAG - for all
Post new topic Reply to topic Goto page Previous  1, 2, 3  Next
View previous topic :: View next topic
Author Message
kgkosio





Joined: 08 Feb 2004
Posts: 596 on topic

Thank you for registering your game
PostPosted: Sun, 21. Feb 16, 04:54    Post subject: Reply with quote Print

Thank you, great research.


_________________
Fan of the save game manager by badger
Best Steam DiD death: Fly into a TL repeatatly while repairing it
Back to top
View user's profile Send private message
Cursed Ghost





Joined: 02 Oct 2004
Posts: 585 on topic

Thank you for registering your game
PostPosted: Sun, 21. Feb 16, 14:13    Post subject: Reply with quote Print

this an interesting read a little over my head but interesting all the same

i do have to wonder though why even do complexes the way it is right now? why not just do away with the whole idea of complexes in there current form and simply implement a system that is much cleaner and simpler which doesn't incur this problem in the first place?

aka you build your complex hub you then you go in to the control panel and then just start adding production lines and for each production line you add you pay the associated cost just as you would if you where buying a new station and complex kit and then the station is updated to accommodate the new production line just the same way it would be upon adding a new station via a complex kit

what you end up with is something a kin to the multi food production factories found in Aldrin a single factory that has multiple production lines

all without having to connect dozens of stations together thus avoiding the associated performance issue plus the associated issues with actually trying to line stations up properly so you don't end up with spaghetti junction plus the issue of not being able to connect stations to the complex after the complex gets to big and the hub is to far away plus the general tedium involved in actually setting up a complex

the only stations this wouldn't work for are mines which would have to continue to use the current system but that's not a big issue.

is there any reason my way of doing this wouldn't work maybe some technical reason maybe ? if not then would it not simply be more sensible to crate a mod that would remove the current complex building system for all stations except mines and simply replace it with one that functions as described above.

Back to top
View user's profile Send private message Send e-mail
glenmcd





Joined: 16 Oct 2010
Posts: 895 on topic
Location: Australia
Thank you for registering your game
PostPosted: Sun, 21. Feb 16, 14:42    Post subject: Reply with quote Print

Many solutions have come forward from the modders to work around the complex lag issue. Litcubes Saturn Hub is one of the better ones. Some address OOS lag, some address IS lag, some both. There's at least one way of reducing some IS lag in the vanilla game and that's with the tubeless mod.

But for the OOS lag which increases exponentially with the number of CCKs used the current way, there's nothing in vanilla to help apart from keep your complex size down to a few dozen factories.

One good reason for posting my thread was to put forward an undeniable case to Egosoft that there's a (vanilla) bottleneck in there that if fixed by them will lead to a superior product for all. Better in a modded game, better in vanilla. Personally, I suspect that the solution you offer would lose a great deal of realism that is present in the vanilla game.

Back to top
View user's profile Send private message
TTD





Joined: 06 Jul 2008
Posts: 10894 on topic
Location: Wiltshire,UK
Thank you for registering your game
PostPosted: Sun, 21. Feb 16, 20:17    Post subject: Reply with quote Print

yes...many ways to crack a nut.
Each person has ideas and favourite solutions.

Glen...I may get back here more regularly at some point, but am still enthralled by ARK: Survival Evolved.

But part of me misses this community. many people come and go. others linger. But unless RL forces it's hand, the best players remain active and so too the modders and moderators.

Your researches into game mechanics always gave me something to think about.
when i saw your name here I just had to read. Smile


_________________
Not My Sites...
TCplots-AP
Customize your start
XDownloads (a Mod Library)
Xadrian'sComplexCalc
REBIRTH HELP
Back to top
View user's profile Send private message Send e-mail
glenmcd





Joined: 16 Oct 2010
Posts: 895 on topic
Location: Australia
Thank you for registering your game
PostPosted: Mon, 22. Feb 16, 09:04    Post subject: Reply with quote Print

MrFiction wrote:
Loading your example save I get more stuttering, also in other sectors. Maybe that's because the neighboring sectors contain a lot of objects or do you think stuttering is inevitable with these complex sizes?


It's taken me two solid days in AP to answer this one! I originally allowed generic missions to pollute the benchmarks and I should not have. I've re-written the original post and it now links to 24 different Albion Prelude save files. They are a mix of various complex sizes, old style and new style. There's even some hybrid versions. Lag coming from NPC stations and ships and generic missions has been avoided and thus testing results should be far more meaningful and accurately reflect what the complexes are doing to the X3 engine.

The only real surprises I got when performing the above benchmarks were:
1. Generic Missions are a real problem. Lag is very significant and "only few players are affected" is absolute crap. "only few players can accurately tell where the lag is coming from" is I believe an accurate statement.
2. Every NPC object (planets, asteroids, debris, ships, stations) in the universe when added up, produces OOS lag which is only barely measurable. IS is an entirely different matter.

Please let me know how your benchmarks compare in some of the saves?
Just stay within the UKS / Unseen Domain during..

Back to top
View user's profile Send private message
MrFiction



MEDALMEDAL

Joined: 07 Jul 2012
Posts: 164 on topic

Thank you for registering your game
PostPosted: Mon, 22. Feb 16, 12:06    Post subject: Reply with quote Print

Wait, I thought the generic mission lag was fixed a long time ago in patch 1.1 or something like that?

edit: and isn't this lag temporary, i.e. the lag is gone when exiting and starting X3 again?

Back to top
View user's profile Send private message
Dome1





Joined: 10 Mar 2004
Posts: 8 on topic

Thank you for registering your game
PostPosted: Mon, 22. Feb 16, 12:28    Post subject: Reply with quote Print

Glenmcd , thanks for you hard work. I will be putting this method to the test and getting back to you . Smile

Back to top
View user's profile Send private message
glenmcd





Joined: 16 Oct 2010
Posts: 895 on topic
Location: Australia
Thank you for registering your game
PostPosted: Mon, 22. Feb 16, 14:36    Post subject: Reply with quote Print

MrFiction wrote:
Wait, I thought the generic mission lag was fixed a long time ago in patch 1.1 or something like that?

edit: and isn't this lag temporary, i.e. the lag is gone when exiting and starting X3 again?


"It" wasn't "fixed". Improvements were made that assisted specific mission types at the time. If you mean lag is gone when you give up current game and start a new one, sure it will reset to initial conditions. Reloading a save won't make GM lag go away. Try resetting the MD just before / after a reload to confirm this.

Following are some benchmarks focussing on generic missions I gathered in past week. Each run starts from the same save. It is possible to focus on one mission type even from same save, this is what I did. Started and ended in empty Unknown Sector, scripted FPS averaging test at start and end. Swapped sectors fast enough so that I could test FPS at end before the missions spawned disappeared. Actual path was UKS to Barren Shores and back. I could have spawned more missions had I not revisited sectors however players do often revisit sectors.

Only 11 sectors could spawn missions as first and last is unknown. Results follow. You'll notice that during these tests, the framerate always went up when no missions spawned, for whatever the reason. I've noticed this a lot lately, but you only notice it when you get rid of the major lag producers.

This block swapped sectors 12 times, last sector could not spawn missions as empty Unknown Sector. So effectively 11 sector swaps:
No missions: 815 FPS -> 903 FPS
No missions again: 816 FPS -> 893 FPS
Scan asteroids: 817 FPS -> 283 FPS
Scan asteroids again: 819 FPS -> 301 FPS
Dual Convoy: Didn't spawn and I don't know why. 816 FPS -> 899 FPS.
Dual Convoy again: Didn't spawn again. 814 FPS -> 899 FPS.
Escort Convoy: 813 FPS -> 99 FPS
Escort Convoy again: 821 FPS -> 110 FPS
Deliver Wares to Station in need: Didn't spawn. 822 FPS -> 894 FPS.
Deliver illegal wares to pirate station: Didn't spawn. 814 FPS -> 884 FPS.
Deliver illegal wares to trading station: 826 FPS -> 489 FPS.
Transport Cargo: 820 FPS -> 381 FPS.
Passenger Transport: 819 FPS -> 666 FPS
Assasination 1: 818 FPS -> 383 FPS
Xenon invasion: Didn't spawn (no Xenon sectors close) 817 FPS -> 903 FPS
Destroy Convoy: 812 FPS -> 97 FPS
Generic Patrol: 811 FPS -> 793 FPS
Defend Object: 816 FPS -> 387 FPS
Build Station: 824 FPS -> 660 FPS
Transport (one) Passenger: 806 FPS -> 519 FPS
Follow ship: 811 FPS -> 511 FPS
Deliver Matching Ship: 823 FPS -> 731 FPS
Freight Scan: 813 FPS -> 533 FPS
Repair Station: Didn't spawn. 812 FPS -> 900 FPS
Multiple Transport: Didn't spawn. 822 FPS -> 921 FPS
Tour of a lifetime: Didn't spawn. 804 FPS -> 906 FPS
Notoriety Hack: Didn't spawn as no pirate stations near. 846 FPS -> 906 FPS
Buy Asteroid Survey: 826 FPS -> 295 FPS
Buy sector data: 824 FPS -> 491 FPS
Buy used ship: 828 FPS -> 240 FPS
Buy Blueprints: 839 FPS -> 573 FPS

Escort Convoy only, one sector swap only: 817 FPS -> 564 FPS
Escort Convoy only, 142 sector swaps: 781 FPS -> 5 FPS
Escort Convoy only, 228 sector swaps (toured the universe): 805 FPS -> 2 FPS

All generic missions (original), 48 sector swaps: 825 FPS -> 211 FPS
All generic missions (original), one sector swap only: 823 FPS -> 632 FPS


So as you can see, framerate is significantly affected on spawning of any and all generic mission types. Those that failed to spawn at all always resulted in an increase in framerate, at least over the 12 sector tests.

When all generic missions are enabled, the resulting performance degradation is associated with the sum of the effects from each mission spawned. Because this happens using a combination of intentional chance setting as well as some randomness, the only way to properly tell what's going on is to focus on one mission type at a time. This doesn't eliminate randomness but it assists.

I finished the tests with allowing all missions to spawn as per vanilla. The results cannot possibly be accurate as randomness plays such a big part in determining which missions spawn, how many and when.

I do know that when all generic missions are disabled, that I can swap sectors hundreds of times without seeing this significant framerate degradation.

The tests I did measured more than framerate. They timed gate passing, compared game time and real time, counted frames in each sector passed and logged the lot. Unless you push the generic missions issue a LONG way, you won't notice longer gate passing times, nor longer gamesave loading times. It will happen eventually, but by then your framerate is down to single digits.

Performance degradation from our big old complexes is quite different in this respect. It doesn't grow with sector swaps or with time. It's a little more if you run out of resources but this is hardly significant. If anything, it's gate passing time that suffers first, but it's a close race with the old factory complex issue.

To put this in perspective, changing sector one time triggers the generic missions spawned (averaged) to place the same additional load on the CPU as building an additional ten thousand factories, all actively producing in parallel. That's more factories than any X3 game is ever likely to have. Even all the ships in the universe present only a fraction of the load OOS that GMs do, again with only one sector swap. This is the case even when the mission spawned is in some other sector and you didn't accept it or even ask about it. Each time you swap sector again, it adds to lag again. And again. And I'm not just referring to the GM types that are currently recognised as broken/worst.

The fact that most missions take back some of this lag after an hour or two does little more than discourage players from exploring the universe and encourage one to quit their current game. X3 has always been like this. That can change.

Back to top
View user's profile Send private message
MrFiction



MEDALMEDAL

Joined: 07 Jul 2012
Posts: 164 on topic

Thank you for registering your game
PostPosted: Mon, 22. Feb 16, 16:11    Post subject: Reply with quote Print

You sure put a lot of time into this research, very interesting. Two "bold" statements, which I think are true.

1. Reading the post about GM lag suggest that waiting in a sector for long enough will get rid of the lag (for a while). So, idling in an empty unknown sector at 10xseta for 10 minutes will clear most of the lag in the current game.

2. Disabling GM from the start and building complexes in pairs of 2, connecting those pairs together, and so on gets rid of the lag completely in a game.

Back to top
View user's profile Send private message
enenra





Joined: 08 Apr 2005
Posts: 6152 on topic

Thank you for registering your game
PostPosted: Mon, 22. Feb 16, 16:53    Post subject: Reply with quote Print

I think people need to make an important distinction here. What is being talked about is not lag per se. What this is about is FPS reduction. And FPS reduction happens naturally as more is going on in a game.

Yes you are going to get more FPS when you don't have any generic missions. But then you also don't have any generic missions. The game will also run faster if you prevent any and all ship spawns in the universe. That doesn't mean you should do that.

The issues are:

a) Short time FPS reductions that are disproportionate to what is actually happening.

b) Long time FPS reductions which accumulate over time.

And here we have to bear in mind that short time reductions is absolutely normal just by nature of something actually happening. The part where it becomes problematic and really important to fix is with case b) because by the very nature of the missions being generic, they shouldn't have long-term impact on the game. If they do, something is going wrong badly and it probably means missions aren't being cleaned up properly.


_________________
Back to top
View user's profile Send private message
pref





Joined: 10 Nov 2012
Posts: 3408 on topic

Thank you for registering your game
PostPosted: Thu, 25. Feb 16, 14:37    Post subject: Reply with quote Print

Thanks glenmcd, nice research.
Funny how we have to build the tree here, normally the CPU should handle such things Very Happy
\still, i like even the issues of X3 more then XR's...

Back to top
View user's profile Send private message AIM Address
glenmcd





Joined: 16 Oct 2010
Posts: 895 on topic
Location: Australia
Thank you for registering your game
PostPosted: Mon, 29. Feb 16, 18:06    Post subject: Reply with quote Print

enenra wrote:
FPS reduction happens naturally as more is going on in a game.

This to me is in sync with long held views held (and often promoted) by Egosoft, many regular posters here (especially moderators) as well as many players that read the forums but don't actually do their own research. If what you are saying (that game requirements in TC and AP justify being overwhelmingly CPU-bound) is true that's fine. But what is "true"? Something that isn't yet proven as false, and this can in some cases take a long time to happen.

Well, it has happened.

I'm not quite done yet, but once Egosoft applies the fixes suggested in this thread and the post linked to above, the mistruths about performance in TC and AP will finally be laid to rest. And that's when I will take a rest also.

There is nothing in Terran Conflict or Albion Prelude that justifies framerates below 120 FPS
because of CPU load, in any sector and regardless of state or "maturity" of game or missions, on any modern gaming computer.


Back to top
View user's profile Send private message
enenra





Joined: 08 Apr 2005
Posts: 6152 on topic

Thank you for registering your game
PostPosted: Mon, 29. Feb 16, 20:02    Post subject: Reply with quote Print

glenmcd wrote:
enenra wrote:
FPS reduction happens naturally as more is going on in a game.

This to me is in sync with long held views held (and often promoted) by Egosoft, many regular posters here (especially moderators) as well as many players that read the forums but don't actually do their own research. If what you are saying (that game requirements in TC and AP justify being overwhelmingly CPU-bound) is true that's fine. But what is "true"? Something that isn't yet proven as false, and this can in some cases take a long time to happen.

Well, it has happened.

I'm not quite done yet, but once Egosoft applies the fixes suggested in this thread and the post linked to above, the mistruths about performance in TC and AP will finally be laid to rest. And that's when I will take a rest also.

There is nothing in Terran Conflict or Albion Prelude that justifies framerates below 120 FPS
because of CPU load, in any sector and regardless of state or "maturity" of game or missions, on any modern gaming computer.


Performance reduction happens as the player builds more stations, has more ships, etc. simply due to the fact that there's more objects than before.

I'm not quite sure how OOS works for sectors that have not been discovered yet but I could imagine that with uncovering more and more sectors the performance will also reduce somewhat.

There is optimization potential, absolutely, and there's probably also some memory leaks. But it's just simple math that more (player-owned) objects in the universe will decrease FPS to some degree.


_________________
Back to top
View user's profile Send private message
glenmcd





Joined: 16 Oct 2010
Posts: 895 on topic
Location: Australia
Thank you for registering your game
PostPosted: Tue, 1. Mar 16, 02:09    Post subject: Reply with quote Print

enenra wrote:
But it's just simple math that more (player-owned) objects in the universe will decrease FPS to some degree.

I'm actually using my experience in software engineering and three decades of research into how software executes to measure performance, find bottlenecks etc. I've reverse engineered thousands of applications of all sizes including games, generated millions of lines of assembler code, tested as assembling and executing as per the original, then profiling and speeding up, many times to dozens to hundreds to thousands of times faster than the original. As well as fixing known bugs and bugs I found during these processes. My favourite language to reverse engineer was C++ due to how incredibly wasteful it is and how it promotes exponential increases in CPU loads using simple, brief and good-looking source code lines. Classic case of shielding the user from things he needs to know.

I assemble with my own assembler, written in 100% hand coded assembler. Every tool I have used in this work, I hand coded in assembler. That includes a number of profilers. As well as dozens of tools that don't have names because I conceived and developed them myself. All of these had a focus on software execution measurement and efficiency. All up, a bit over ten millions lines of code. There's nothing between C code and silicon in a computer that I'm not intimately familiar with in conceptual terms. I did hardware courses first but switched to software as I enjoyed it so much more.

I haven't reverse engineered in many years, but my experiences allow me now to make reasonably accurate predictions based on simply running and using an application. X3 is gifted with a decent scripting language and this has given me what I needed to automate the process of benchmarking, testing and proofing these games that I enjoy playing.

A few months after I bought Egosoft's superbox, I realised that Terran Conflict was seriously suffering in performance in certain areas and that it was almost unique as a 3D game that was CPU bound. I communicated my thoughts to Egosoft. I also used what I could work out about the issues to make my own game more enjoyable. That's when I disabled both Convoy generic missions - early 2011. I didn't know why they were so bad at the time, but know more now. In between, I've learned the scripting language and had a lot of fun, mostly playing Albion Prelude.

When I discovered for myself the large factory complex CCK issue, I initially switched to using Complex Cleaner, but eventually wrote my own CCK code. After a few re-writes and revisions, it successfully and efficiently joined tens of thousands of player factories which in most of my mature games, used up every asteroid in the universe in a balanced "complex" that spread across most sectors in the universe. So I knew years ago that even when you take the basic CCK code requirements and execute it in a higher level language (X3 scripts instead of partly within the AP executable), that it was possible to grow the player empire pretty much as big as it could go given the initial asteroid quota, without hitting any CPU availability walls.

All that's required of Egosoft is to choose appropriate algorithms and data structures as well as effective testing which includes performance topics. Obvious to some, black magic to others. You are of course correct in your statement that adding ships to the universe adds some CPU load. You would be incorrect if you are suggesting that this is happening at a level that could be called significant or noticable. If so interested, use selective disabling of GMs to spawn escort convoy missions in a few dozen sectors. Save game. Then reset the MD and you'll find that all of those escort ships can still get to their destinations with the FPS meter in the high hundreds. Reload game, destroy or destruct these ships without touching the MD and you'll see framerate remain at pathetic, until finally the MD catches on some ten minutes later. Adding a few dozen or even few hundred ships is a PITO.

If you don't understand - and (no offense meant) you appear not to - I suggest just sit back and watch as the changes happen. Now that players know for sure that top performance can be had out of X3, there is no turning back. Egosoft no longer have a choice and they no longer have an excuse. Unless you call abandoning X3 an excuse. Fairly recently Bernd was asking players for guidance on further work on X3. Knowing that good performance truly is possible and achievable, this will further encourage the decision to develop and release revisions of TC and AP. Any delay means more players becoming aware of what they can have but don't have.

When do we want it? Smile

Back to top
View user's profile Send private message
enenra





Joined: 08 Apr 2005
Posts: 6152 on topic

Thank you for registering your game
PostPosted: Tue, 1. Mar 16, 10:33    Post subject: Reply with quote Print

Oh don't worry gelnmcd, I do recognize someone who knows what they're talking about and I don't doubt that you know a lot more about this than me. Smile

I wasn't aware that increasing the amount of objects in a sector does that little to the framerate. That's certainly news for me.

As for MD, that's more my area of expertise. However, from what you describe this might actually be an issue with the implementation of the language itself to a certain degree. Not sure tbh. There's definitely a lot of optimization potential in the MD code itself as well. The thing is that actually a lot of the current basic MD code was written while the language MD was still not quite finished yet. So there's a lot of really old code in there that may be really unoptimized or even not working at all. Sadly, rewriting all of that is... not something that can be done that easily now.

I've personally toyed with the idea and have started a complete rewrite a couple years ago which I've been working on now and then. That's promising but until all of that would be implemented and tested I'd need a lot more time. Like a year plus of working on it full time kind of time probably.

Updating basic MD files would also create issues with savegame compatability which is another issue Egosoft will be limited in. We'll see what can be done. The person to talk to about this is actually probably CBJ. I believe he knows the most about the actual implementation of MD.


Have you looked at the actual MD code at all during your tests? I feel if we could work out the places where performance is most degraded by we could probably narrow down the reasons for the unreasonable decrease in performance. I could well imagine that some of the conditions used are not really properly optimized for what they do, for example.

By the way, consider signing up for the DevNet through your profile if you haven't already. That's usually an a bit more direct line to the devs. Smile


_________________
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic Reply to topic Goto page Previous  1, 2, 3  Next
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
Control Panel
Login Data
The time now is Mon, 23. Oct 17, 20:56

All times are GMT + 2 Hours


Board Security

Copyright © EGOSOFT 1989-2017
Powered by phpBB © 2001, 2005 phpBB Group
Template created by Avatar & BurnIt!
Debug: page generation = 1.07010 seconds, sql queries = 29