EnglishGermanFrenchRussianPolishItalianSpanish
Log inRegister
 
Super-sized complexes - without the LAG - for all
Post new topic Reply to topic Goto page Previous  1, 2, 3
View previous topic :: View next topic
Author Message
glenmcd





Joined: 16 Oct 2010
Posts: 895 on topic
Location: Australia
Thank you for registering your game
PostPosted: Tue, 1. Mar 16, 11:51    Post subject: Reply with quote Print

Thank you.

>> I wasn't aware that increasing the amount of objects in a sector does that little to the framerate.
To some degree this is a hunch of mine but partly observation too. I'll come back with some numbers. Pretty sure they'll be an eye opener for some.

With MD it seems that there is enormous resistance against doing a rewrite or anything major on it. My contacts so far have been Owen and Bernd. So my post in the tech support thread was focussed on an Egosoft stamped workaround that would require only days to weeks and perhaps two months including testing before a release that would be save file compatible. Here's what I had in mind (for someone else to do lol):

1. Don't allow Convoy missions to spawn. No problem with save compatibility here. That gets rid of perhaps 50 percent or more of the framerate problem.
2. Get rid of "buy asteroid survey" offered from flying ship in space. We already can do this from a station mission, and it behaves itself. A mission that doesn't spawn doesn't bother us with cleanup issues.
3. Two generic mission types need to be fixed to respect quota. I don't really spend any significant time on MD stuff but I can't imagine this could be too hard. Surely must have been the intent. These mission types are ones that players would miss (freight scan, buy blueprints).
4. Set quota in globals.txt to 1, for all four items. 20 second job and except for those missions that ignore it, is working fine in my tests.
5. Something I was toying with today: At the moment, most missions cleanup after four sector swaps. With all above done, performance is pretty good, however halving this to two sector swaps would give significantly higher resistance on slow machines to ridiculous framerate drops when other stuff is added on: war sectors, Albion Gamma etc.

I see the GM stuff as being far easier and quicker to get to release stage compared to the full fix on Complex CCKs. Complex CCKs does have a viable vanilla workaround, GMs don't. GM fixes will be felt by all and in all games, new or mature. Those that thought performance was already okay will get a nice surprise. Players that build the biggest complexes and hit the CCK FPS wall are also those most likely to visit these forums and see the workaround - or be pointed to it.

I've been on Devnet a few years, but TBH haven't got much out of it. It seems that one voice isn't enough to get some things done, but many voices - which are found here in the open forum - are listened to.

I do have time on my hands at the moment, and would be willing to see what I can do with MD while retaining compatibility. Understand that the only edits I've done to MD files were done in a text editor and mostly restricted to flipping individual bits or bytes or commenting stuff out. Not exactly ideal for XML I imagine. I'm no noob so no need for detailed coaching, perhaps just mentioning the best app to use for editing XML and any "gotyas" I'm likely to come across should suffice.

You speak of "optimization potential" as something that's a big job. It doesn't need to be. I've found that if you spend more time working out the smallest changes possible to make the biggest impact possible, that it's the best way to go.

If you can point me in the right direction with MD I'll see if I can put my workaround into MD code. I dislike rewrites unless really necessary. If I can flip a bit and get a good result, that's what I prefer. I got the impression from my workaround above that especially given the maturity of X3, it may not be necessary with the MD.

Back to top
View user's profile Send private message
glenmcd





Joined: 16 Oct 2010
Posts: 895 on topic
Location: Australia
Thank you for registering your game
PostPosted: Wed, 2. Mar 16, 10:45    Post subject: Reply with quote Print

enenra wrote:
I wasn't aware that increasing the amount of objects in a sector does that little to the framerate. That's certainly news for me.

As promised, here are some numbers comparing CPU load with and without ships throughout Universe.

Albion Prelude 3.1
All asteroids, planets and debris removed from universe.
Computer used was hand built with fastest available parts 3.5 years ago, not overclocked.

In UKS East of Unseen Domain. All framerate measurements in here, same position same rotation. I converted FPS to time between frames, assuming that most if not all of this was required to generate the frame.


Baseline:
Vanilla jobs.txt, around 8090 ships.

As framerate was >1000 without missions, in both tests I first generated 2 missions of same type (Passenger Transport). This brought the FPS down below 1000. Dozens of tests using same mission types confirmed repeatability of FPS.

Framerate with 8090 ships: 780 FPS = 1282 microSeconds / frame
Framerate with empty Jobs.txt (no ships): 882 FPS = 1133 microSeconds / frame
Difference is 1282-1133=149 microSeconds per frame.
That's 18 nanoSeconds per ship average added to each frame time.
At 60 FPS, it would take twice this overhead just to drop one FPS.
At 1200 FPS, it's enough to drop by 150 FPS to 1050 FPS.

Comparison with generic mission overheads:

Each generic mission that spawns ("on offer") with exception of Convoy types, adds roughly 79 microSeconds to each frame time. With two generic missions on offer somewhere/anywhere in the universe, you've already added more time to each frame than all the ships in existence at the start of an AP game, and that's throughout the entire universe. We're still a number of times short of the GM overhead likely experienced in my workaround linked to above. And this in turn is dozens of times less than that experienced in most AP games today, once the player has explored half the universe or more.

Others? I found generally that ships add more overhead to (OOS) frame time than asteroids, planets, debris and even stations. Without the CCK issue, player factories will not even come close to the overhead that 8090 ships add. And it's still hardly even worth a mention.

Summary: When it comes to OOS performance in TC/AP, only large complex CCKs and generic missions matter at the moment. Truly, just forget the rest until these are fixed.

Back to top
View user's profile Send private message
MrFiction



MEDALMEDAL

Joined: 07 Jul 2012
Posts: 164 on topic

Thank you for registering your game
PostPosted: Wed, 2. Mar 16, 13:10    Post subject: Reply with quote Print

glenmcd wrote:
...until these are fixed.

Not likely, time to move on to other games

Back to top
View user's profile Send private message
enenra





Joined: 08 Apr 2005
Posts: 6152 on topic

Thank you for registering your game
PostPosted: Wed, 2. Mar 16, 16:55    Post subject: Reply with quote Print

glenmcd wrote:
Thank you.

>> I wasn't aware that increasing the amount of objects in a sector does that little to the framerate.
To some degree this is a hunch of mine but partly observation too. I'll come back with some numbers. Pretty sure they'll be an eye opener for some.

With MD it seems that there is enormous resistance against doing a rewrite or anything major on it. My contacts so far have been Owen and Bernd. So my post in the tech support thread was focussed on an Egosoft stamped workaround that would require only days to weeks and perhaps two months including testing before a release that would be save file compatible. Here's what I had in mind (for someone else to do lol):

1. Don't allow Convoy missions to spawn. No problem with save compatibility here. That gets rid of perhaps 50 percent or more of the framerate problem.
2. Get rid of "buy asteroid survey" offered from flying ship in space. We already can do this from a station mission, and it behaves itself. A mission that doesn't spawn doesn't bother us with cleanup issues.
3. Two generic mission types need to be fixed to respect quota. I don't really spend any significant time on MD stuff but I can't imagine this could be too hard. Surely must have been the intent. These mission types are ones that players would miss (freight scan, buy blueprints).
4. Set quota in globals.txt to 1, for all four items. 20 second job and except for those missions that ignore it, is working fine in my tests.
5. Something I was toying with today: At the moment, most missions cleanup after four sector swaps. With all above done, performance is pretty good, however halving this to two sector swaps would give significantly higher resistance on slow machines to ridiculous framerate drops when other stuff is added on: war sectors, Albion Gamma etc.

I see the GM stuff as being far easier and quicker to get to release stage compared to the full fix on Complex CCKs. Complex CCKs does have a viable vanilla workaround, GMs don't. GM fixes will be felt by all and in all games, new or mature. Those that thought performance was already okay will get a nice surprise. Players that build the biggest complexes and hit the CCK FPS wall are also those most likely to visit these forums and see the workaround - or be pointed to it.

I've been on Devnet a few years, but TBH haven't got much out of it. It seems that one voice isn't enough to get some things done, but many voices - which are found here in the open forum - are listened to.

I do have time on my hands at the moment, and would be willing to see what I can do with MD while retaining compatibility. Understand that the only edits I've done to MD files were done in a text editor and mostly restricted to flipping individual bits or bytes or commenting stuff out. Not exactly ideal for XML I imagine. I'm no noob so no need for detailed coaching, perhaps just mentioning the best app to use for editing XML and any "gotyas" I'm likely to come across should suffice.

You speak of "optimization potential" as something that's a big job. It doesn't need to be. I've found that if you spend more time working out the smallest changes possible to make the biggest impact possible, that it's the best way to go.

If you can point me in the right direction with MD I'll see if I can put my workaround into MD code. I dislike rewrites unless really necessary. If I can flip a bit and get a good result, that's what I prefer. I got the impression from my workaround above that especially given the maturity of X3, it may not be necessary with the MD.


Your suggestions certainly will get rid of the problems you've found but there is the issue of changing gameplay quite significantly by influencing the amount of missions available or completely removing a mission type. Those are not really the kind of solutions that we'd want to implement into a game, or only if there really is no other option.

1. The issue here lies with what the mission does. I haven't looked at its code outside of a cursory glance but there must be something that's going terribly wrong for it to have such an impact. You mentioned before that the ships of these missions take ages to despawn, but just them existing can't really be the issue considering other missions handle way more ships than these. I'd assume there's either something wrong where its ships don't get cleaned up properly or way too slowly, although again, ordering ships to fly to a station where they are then despawned shouldn't have as much of an impact.

On the top of my head there's a couple things that could be done if the issue lies with ships not despawning quickly enough: a) Make sure to not use really slow ships for convoys so that they can reach their destination in a more reasonable timeframe, be that the mission target or the station where they are to be cleaned up at. b) Spawn convoy ships docked at a station. That way they can just be destroyed if the player doesn't accept the mission and don't have to first fly to a station. c) Additional rule where if the player is not in the sector and there is no player owned object near (hence not in radar range), instantly despawn ships.

The issue might also be in the way that conditions are used inside the code. Sometimes there's delays missing inbetween checks, etc. This is something that has to be analyzed with the MD debugger.

2. Seems reasonable. Doesn't really result in a loss of content either. Though again I'd be curious what exactly is the issue here and whether it could be resolved without removing the ships completely. There ought to be something very wrong with its handling to cause such an impact.

3. To make a mission respect quota is indeed not a big change, but I fear the issue lies more in a buggy check rather than it not having the code for it. Not sure.

4. That would influence the amount of missions available quite drastically and therefore alter gameplay. Can't imagine any player doing a lot of missions would be happy about this. IMO not really a viable solution. Need to find and fix the bugs instead. And figure out why multiple instances of the same mission have such an insane effect. (likely outside of what can be affected in the MD itself)

5. From what I remember the cleanup is not necessarily dependant on sector swaps but on mission offer timeouts. Not sure though. In any case though that is certainly something that could be looked at since the gameplay impact won't be large.


Devnet lvl 3 is not much atm. Well, not since XR testing has been made available to everyone. What you might want to consider is signing up for lvl5 by signing the NDA. lvl5 gets more access into the development, earlier betatests, and is also able to contribute to patches etc.

What do you need regarding MD? I can help you set it up but it's really not much more than extracting the files. Pretty much any text editor will do that can read the xml stylesheets that you'll find in the same directory. I've been using Visual Studio which has worked well for me. I'm pretty sure Sublime works as well. Also: mdfiles.txt lists (if present) all files that are loaded by the game. That's pretty much it.

I agree that relatively small changes can make a big difference but it's really not good practice to remove functionality from the game in the process of doing so unless it can absolutely not be helped. ^^

Thanks for the object analysis btw. though I gotta say I was actually more thinking about specifically player ships, since they run on a different set of scripts etc. than the ships spawned by the Jobs file.


_________________
Back to top
View user's profile Send private message
shanrak





Joined: 25 Feb 2009
Posts: 587 on topic

Thank you for registering your game
PostPosted: Thu, 3. Mar 16, 20:41    Post subject: Reply with quote Print

So I've been making complexes using this method in my new game ever since I saw this. I have a 128 station energy loop in savage spur, and a few 80+ station complexes in other areas. So far load times seems to be great, haven't noticed much increase during jumping at all.

The only downside of this method is that its a huge hassle while connecting the stations:



Connecting stations the old way was much easier Razz

Back to top
View user's profile Send private message
glenmcd





Joined: 16 Oct 2010
Posts: 895 on topic
Location: Australia
Thank you for registering your game
PostPosted: Sat, 5. Mar 16, 13:27    Post subject: Reply with quote Print

shanrak wrote:
The only downside of this method is that its a huge hassle while connecting the stations:

Connecting stations the old way was much easier Razz

Thanks for the feedback @shanrak. I must agree with you that knowing which complex hubs to connect next is not straight forward. But here is an idea that may ease this task a little:

1. Joining factories initially is easy so do all of these first. One of these must be positioned carefully as it will become the one that ships will dock at (the only surviving complex hub).
2. Before you join any complex hubs together, rename ALL complex hubs, adding a string of digits to the start of current names, where the numbers are incrementing. So instead of all being "Your Complex Hub alpha", the first would be "100Your Complex Hub alpha", the second "101Your Complex Hub alpha", the third "102Your Complex Hub alpha" etc. If you plan a really huge complex, use a 4 digit sequence. The hub you select to be first in this list should be the hub that you positioned carefully in step (1) above. See below for reasons.
3. Because all hubs now have unique names, they will no longer jump around in sector lists as well as when asked which things to connect using CCKs.
4. Connect the two hubs at start of list of hubs.
5. The first in the list now will be the first hub you selected in step 4 above and it will contain 4 factories. Rename this hub, adding a "z" to the start of the name. So "100Your Complex Hub alpha" would change to "z100Your Complex Hub alpha". It will drop to end of list at this point.
6. Repeat steps 4 and 5. Eventually, all the hubs containing 2 factories will instead contain 4, and all will have names that start with "z". Next, it's actually just as simple as repeating steps 4 and 5 again until you get down to just one hub.

Ensuring that the correct hub becomes the final one:
This is easy. Choose which hub you want to be the one that you dock at, and make this the one that is first alphabetically. In above example, this would be named "100Your Complex Hub alpha". When you select hubs to connect, the first one you select survives, the second is consumed. If when you connect "100Your Complex Hub alpha" it's always the first selected, then it will survive all connects and thus be the last remaining hub. The only real gotya here is when you have an odd number of hubs to join in a particular level. This won't be the case if your initial number of factories is an even power of 2 (32 factories, 64 factories etc). No problem, just be aware as you approach the end of each level of connects.

Back to top
View user's profile Send private message
Cursed Ghost





Joined: 02 Oct 2004
Posts: 585 on topic

Thank you for registering your game
PostPosted: Sun, 6. Mar 16, 00:27    Post subject: Reply with quote Print

Quote:
Personally, I suspect that the solution you offer would lose a great deal of realism that is present in the vanilla game.


Perhaps but I would think this is one of them situations where you would want to make a small compromise for better performance and better simpler game play for players because I know that is something that a lot of players complain about particularly those new to the series and even as a veteran of the series I have to say they have a point there are a lot of elements which could do with being simplified for the sake of improved game play

Yes it detracts a little from the realism but you have to ask your self the question is it not worth sacrificing a little realism for improved game play especially when those changes not only improve game play by removing unnecessary complexity which really doesn’t add very much to the game anyway but also removes the very performance issues you are positing about

Now I don’t know about anyone else but personally I’d say that’s a trade worth making as long as it’s handled properly

don't get me wrong i realize that its not always so simple and depending on the nature of the issue it could well be one of those things that would require a massive rewrite of code which simply isn't practical because of the time and cost involved having said that however if they just wrote clean efficient code and properly tested stuff to make sure it works correctly before releasing it then this wouldn't even be an issue in the first place

now i know how that sounds and I don't know maybe if its just me but i was always taught to take my time and do it right first time and don't release work until I'm satisfied it's done correctly.

admittedly in the real world where you have dead lines and schedules etc you don't always have that luxury having said that that's no excuse for taking short cuts and not doing what you are supposed to do this is why so meany games these days come out so bug ridden that they are completely unplayable Stalker Shadows of Chernobyl being a prime example that game was so bug ridden that it crashed continually was unplayable because it.

although to Ego's credit they are for the most part good at fixing stuff when it doesn't work right which is more than can be said for a lot of other game developers who simply wont fix things even when they know its broken because they are only interested ££££ and once they have it they don't care anymore.

Back to top
View user's profile Send private message Send e-mail
Rabiator der II.





Joined: 14 Nov 2011
Posts: 891 on topic

Thank you for registering your game
PostPosted: Fri, 18. Mar 16, 11:18    Post subject: Reply with quote Print

@Glenmcd:
Thanks for putting in the effort to research this issue Smile.

And without having done the research myself, I strongly suspect that Egosoft could solve the performance problem in complexes by picking a better algorithm. Simply because the first idea that comes to mind (going through a linked list of the connected stations) should require time roughly proportional to the number of connected stations. That is without spending time on finding a better solution, such as aggregating all station in the hub into one super-station for resource calculations.

How Egosoft managed to end up with exponential time is beyond me.

Edit:
I've reread your post from the beginning, and if the vanilla algorithm looks up every factory in the linked list in a separate search, with going through the list again for the next factory, an O(n²) complexity of the algorithm would result. But that would be stupid indeed Headbang .

Edit: corrected my Big O notation Embarassed


_________________
Gazz in the LT forum:
In X3, piracy is not implemented at all. All the "pirates" that fly around are bands of roaming psychopaths that destroy everything they see without even trying to loot anything.
Back to top
View user's profile Send private message
glenmcd





Joined: 16 Oct 2010
Posts: 895 on topic
Location: Australia
Thank you for registering your game
PostPosted: Sun, 20. Mar 16, 22:31    Post subject: Reply with quote Print

Rabiator der II. wrote:
I've reread your post from the beginning, and if the vanilla algorithm looks up every factory in the linked list in a separate search, with going through the list again for the next factory, an O(n²) complexity of the algorithm would result. But that would be stupid indeed

That's the only thing I can think of that would be happening. Although to us (outsider devs and ex devs looking on) it certainly looks like a stupid choice of algorithms, you have to consider that at the time of choosing, computers were much slower graphically, and nobody would have expected us to be building complexes as big as we do now. The graphic adapters of the day would have made frame-rates so low after say 200 factories were in sector, that building an even bigger complex would have been impossible. As graphic adapters are much more capable now, we've finally come to moving the stress onto other components in the X3 engine. Not just for some, but for all players that can afford very capable graphic adapters. In 2016, I'd say that would be the majority.

Back to top
View user's profile Send private message
RAVEN.myst





Joined: 20 Jun 2011
Posts: 2226 on topic

Thank you for registering your game
PostPosted: Mon, 4. Apr 16, 18:19    Post subject: Reply with quote Print

I'd like to add my voice to the many others thanking you, glenmcd, for your stellar (heheh!) work on this - it is greatly appreciated!

(Your research is particularly well timed for me, as I am now almost done with my latest visit to Rebirth-land and nearing my inevitable return to X3AP - armed with a constructively different way to do something!)

I hope that your efforts nudge ES in the right direction - you've proven that certain significant (to say the least!) performance improvements can be effected with relatively little investment of effort.


_________________
-
Any loose animals will be vaporized on sight.
-
Born on Lave, raised on Freeport 7...
-
The Write Stuff
Back to top
View user's profile Send private message
bounty_hunter66





Joined: 15 Aug 2006
Posts: 465 on topic

Thank you for registering your game
PostPosted: Thu, 15. Jun 17, 09:04    Post subject: Reply with quote Print

How are you supposed to build like this?

Considering the fact that Complex Hubs have the extremely retarded tendency to misplace themselves when you build them too close to any object because they think they interfere with other collision boxes or whatever, but they aren't even remotely close to begin with(unlike normal stations witch do not give a damn where they are placed and can even be overlapped/stacked)

And considering the 20km maximum range for connections.

I'm having the most difficult time to build a measly 33 station complex with this method. Each time I connect Hubs, the next Hubs that will contain X^2 stations are farther and farther apart and go beyond the 20km range.

Back to top
View user's profile Send private message Yahoo Messenger
Snafu_X3





Joined: 28 Jan 2009
Posts: 3823 on topic
Location: London, UK
Thank you for registering your game
PostPosted: Fri, 16. Jun 17, 01:06    Post subject: Reply with quote Print

The last hub you connect to will become the only hub for that plex. If building plexes, by all means connect them with hubs if necessary but then leave the last connection to the plex (your trade dock) until.. last!


_________________
Wiki X:R 1st Tit capping
Wiki X3:TC vanilla: Guide to generic missions, Guide to finding & capping Aran
Never played AP; all X3 advice is based on vanilla+bonus pack TC or before: AP has not changed much WRT general advice.

I know how to spell teladiuminumiumium, I just don't know when to stop!

Dom (Wiki Moderator) Cool DxDiag
Back to top
View user's profile Send private message
RAVEN.myst





Joined: 20 Jun 2011
Posts: 2226 on topic

Thank you for registering your game
PostPosted: Fri, 16. Jun 17, 01:16    Post subject: Reply with quote Print

bounty_hunter66 wrote:
How are you supposed to build like this?

Considering the fact that Complex Hubs have the extremely retarded tendency to misplace themselves when you build them too close to any object because they think they interfere with other collision boxes or whatever, but they aren't even remotely close to begin with(unlike normal stations witch do not give a damn where they are placed and can even be overlapped/stacked)

And considering the 20km maximum range for connections.

I'm having the most difficult time to build a measly 33 station complex with this method. Each time I connect Hubs, the next Hubs that will contain X^2 stations are farther and farther apart and go beyond the 20km range.


I recommend building in a structured matrix, and ideally somewhat flat - if you are not deep in one of the three dimensions, you will have plenty of space in reserve. I've built plexes with 64 and more stations this way, no problem. Keep in mind that when you link two hubs, the first one you select will be the one that stays, the second one selected will connect to it. This way, you can easily predict where your resultant hubs will be. When you do the "real" one (last, as Snafu pointed out), you choose it out of the two remaining hubs, and then link the other to it. However, I suggest selecting the "true" hub's location early, so that you can plan accordingly.

To stay within the requisite maximum distance, try to end up with a sort of regular alternating pattern of hubs (sorry, I can't really explain it better than that, at least not until I've slept Razz ) - you end up with something almost resembling a caterpillar (depending on your 'plex layout), and then you join pairs of "legs" to each other, and then join those links to each other.

Best of luck!


_________________
-
Any loose animals will be vaporized on sight.
-
Born on Lave, raised on Freeport 7...
-
The Write Stuff
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic Reply to topic Goto page Previous  1, 2, 3
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
Control Panel
Login Data
The time now is Sun, 22. Oct 17, 01:07

All times are GMT + 2 Hours


Board Security

Copyright © EGOSOFT 1989-2017
Powered by phpBB © 2001, 2005 phpBB Group
Template created by Avatar & BurnIt!
Debug: page generation = 1.21793 seconds, sql queries = 27