[MOD coming soon] X3-Reunion performance callback

[Observe]

Based on investigation into reasons for Terran Conflict poor fps performance, we have decided to embark on a "performance callback" mod. This will result in similar performance to that of X3-Reunion.

The main reason Terran Conflict has such horrendous performance is because of too many AI Jobs being performed - with consequent demand on the cpu.

To prove this concept, I have performance the following rough experiments:

1. Eliminate Jobs.txt: Result is incredibly good performance (fps). Of course this is not an option because it eliminates virtually all AI "life" in the galaxy.

2. Reduce Jobs by half: Performance is equivalent to X3-R.

If you compare Jobs between X3-R and X3-TC, you will notice the following:

Maximum Jobs total:
X3-R = 3580
X3-TC = 7746

Maximum Jobs per sector:
X3-R = 442
X3-TC = 872

Number of sectors:
X3-R = 163 (plus a few more added with various patches)
X3-TC = 227

Note: Jobs per sector is not quite what is sounds like. Consider many Jobs are related to specific sectors, and won't spawn in all sectors. Therefore, these numbers can be misleading if simply taken at face value.

My conclusion is that Terran Conflict is suffering from Jobs "glut". I really don't think there is much Egosoft can do about the situation short of significant Jobs reduction.

Therefore, we will be releasing a callback mod for Terran Conflict. This will possibly consist of X3-Reunion galaxy map, and associated Jobs converted for Terran Conflict. Performance is (proved) MUCH improved (similar to X3-R).

Finally smooth performance equivelent to our beloved X3-Reunion is within our grasp on the Terran Conflict plateform!

[LTerSlash]

Yeah but most of the performance problems are more GPU-Ralated than CPU related... its very wierd, i mean if i jump to senator badlands i will have a very very very bad fps.... not reason at all because the sector its empy, if i jump to argon prime, well the fps are about x4-6 than in senator badlands...

Also, keep running gpu-z on background doing a sensor log of my video card, this is the only game that carry my GPU (a HD4850) at 100% activity... not even Crysis at High settings could do that, because my CPU bottlenecks my GPU... the performance problems of TC are more in how it hadles the video driver than CPU tasks...

[Observe]

Yeah but most of the performance problems are more GPU-Ralated than CPU related... its very wierd, i mean if i jump to senator badlands i will have a very very very bad fps.... not reason at all because the sector its empy, if i jump to argon prime, well the fps are about x4-6 than in senator badlands...

Clearly the game places demands on both gpu and cpu. One or the other (gpu or cpu) will present the major bottleneck depending on what's going on with the game in total. All I'm saying is Terran Conflict performs equivalent to X3-R when AI is reduced to similar level. Possibly ES can solve some of these problems with new patch, but I really don't think they can reduce AI overhead to the level it was with X3-R without significant alteration in the way Jobs are handled by the cpu.

[ttl]

I'll shamelessly hijack your thread to note three things (at least one which you surely already know).

1)
Most of those jobs are civilians, and militaries. The civilians can be removed, with minimal gameplay effect outside of lack of scenery. Traders are likely best not greatly reduced, to keep the economy humming.

2)
As far as the militaries go, I've been working on that, see :

http://forum.egosoft.com/viewtopic.php?t=224070

It's a script that only spawns heavy military presence only when needed, thus allowing the removal of most military Jobs. It's still work in progress, after conversion from X3R XTM mod (which had Job numbers similar to X3TC).

In case you are interested, I am willing to offer my assistance and co-operation with this project. Minimally, my script would likely work well together with your reduced Jobs as it stands. But furthermore, it would probably allow for almost complete removal of military Jobs, while keeping the same or better military performance, for the same or less CPU load.

3)
If you wish to get back to X3R galaxy map, that's fine, but are you sure that's really necessary if all you want is the performance? Bottom line is, I'd think these three things would get the performance at least close to X3R levels, without need for custom galaxy (you can keep the plot and all) :

1) remove civilians from Jobs
2) replace most militaries from Jobs with something smart (which I strived to accomplished with my script)
3) remove most non-asteroid rocks from the universe (these cause heavy load IS only, but rocky sectors can easily get really bad)

Furthermore, probably Xenon and pirate Jobs could be replaced by a smart script along the same lines, to get a universe that acts more smart, thus seems more immersive, _and_ gain performace.

[ThisIsHarsh]

I totally agree with ttl here. Though I will also add that it isn't all just about the amount of jobs, but the complexity of the job scripts themselves. Civilian ships scripts are very very simple, whereas traders and police are much more complex.

I personally never noticed any performance increase with either X3R or X3TC with a no civ mod... but it must help, even if only marginally.

[Cycrow]

Yeah but most of the performance problems are more GPU-Ralated than CPU related... its very wierd, i mean if i jump to senator badlands i will have a very very very bad fps.... not reason at all because the sector its empy, if i jump to argon prime, well the fps are about x4-6 than in senator badlands...

Also, keep running gpu-z on background doing a sensor log of my video card, this is the only game that carry my GPU (a HD4850) at 100% activity... not even Crysis at High settings could do that, because my CPU bottlenecks my GPU... the performance problems of TC are more in how it hadles the video driver than CPU tasks...

think ur mistaken abit.

X games are most likly more CPU dependant.

the amount of load doesn't really prove that ur GPU is struggling.

most ppl have found that they are more CPU limited, you can usually prove this by changing the graphics mode, in most cases when using good graphics card, the change in modes makes very little difference in performance.

thats because the GPU can keep up fine but is having to wait for the CPU, which makes the frame rates drop.

[Cycrow]

I personally never noticed any performance increase with either X3R or X3TC with a no civ mod... but it must help, even if only marginally.

thats because civilians dont really do much

if they are on screen, then of course its additional rendering, and of course the addition collosion checks per ship. But overal it doesn't make an aweful lot of difference.

especially as civilian ships are only active IS so there isn't a huge amount of AI being done, unlike the other ships which are running in most sectors all at the same time

[fud]

If you want a real example of just how CPU-intensive TC is, run Cycrow's "SWEN" script, and set it to watch ALL sectors.

Then enjoy the slideshow.

[LTerSlash]

Yeah but most of the performance problems are more GPU-Ralated than CPU related... its very wierd, i mean if i jump to senator badlands i will have a very very very bad fps.... not reason at all because the sector its empy, if i jump to argon prime, well the fps are about x4-6 than in senator badlands...

Also, keep running gpu-z on background doing a sensor log of my video card, this is the only game that carry my GPU (a HD4850) at 100% activity... not even Crysis at High settings could do that, because my CPU bottlenecks my GPU... the performance problems of TC are more in how it hadles the video driver than CPU tasks...

think ur mistaken abit.

X games are most likly more CPU dependant.

the amount of load doesn't really prove that ur GPU is struggling.

most ppl have found that they are more CPU limited, you can usually prove this by changing the graphics mode, in most cases when using good graphics card, the change in modes makes very little difference in performance.

thats because the GPU can keep up fine but is having to wait for the CPU, which makes the frame rates drop.

ofcourse.. but there is no way that a empy sector to has a lot more cpu heavy than a central sector, like Argon prime... as it happends to me... Senator badlands is almost imposible to play inside that sector for me... and the only thing that are inside are a nebula and asteroids... and maybe 1 or 2 patrols of yaki fighters... you cant compare it with a centrla sector like argon prime or kingdom end... im having a lot more fps in kingdom end/argon prime that has a lot of stations, merchants, military and civilians... senator badland is empy and i have 5 fps in there, at most... so...

i know that x-series are highly cpu-intensive, but right now there are a big optimisation problem in how the game hadles the driveo driver.

[Graxster]

I will also add that it isn't all just about the amount of jobs, but the complexity of the job scripts themselves. ... traders and police are much more complex.

Agreed. Just take a look at the number of scripts. When I first heard about the UT bug, I looked at the scripts and compared them to X3R. In R there were 22, in TC there are 28. Also quite a bit of additional scripting for "missile boats", and missile firing in general, as well as a bunch of "auto-resupply" stuff going on. Just take a look at "script statistics" in the editor. !lib.resupply.rearm.fromdockedat has been called over 100,000 times in my current game that's only a bit over 1 day old.

Another thing that puts a strain on the script engine is this new "fights between the races" thing they have going on. You may or may not have noticed/seen it, but every so often the game will spawn huge armadas of Military ships and send them to a different race's sectors to start attacking things. A couple of days ago in my other saved game, I deleted at least 500 Paranid/Argon/Split/Teladi Military ships that were stacking up in preparation for war.

It's jobs, complexity of scripts, and a number of other things that the game is trying to do all at the same time. And don't forget that the Mission Director is also busy putting a load on the machine too.

-Grax

[Cycrow]

Yeah but most of the performance problems are more GPU-Ralated than CPU related... its very wierd, i mean if i jump to senator badlands i will have a very very very bad fps.... not reason at all because the sector its empy, if i jump to argon prime, well the fps are about x4-6 than in senator badlands...

Also, keep running gpu-z on background doing a sensor log of my video card, this is the only game that carry my GPU (a HD4850) at 100% activity... not even Crysis at High settings could do that, because my CPU bottlenecks my GPU... the performance problems of TC are more in how it hadles the video driver than CPU tasks...

think ur mistaken abit.

X games are most likly more CPU dependant.

the amount of load doesn't really prove that ur GPU is struggling.

most ppl have found that they are more CPU limited, you can usually prove this by changing the graphics mode, in most cases when using good graphics card, the change in modes makes very little difference in performance.

thats because the GPU can keep up fine but is having to wait for the CPU, which makes the frame rates drop.

ofcourse.. but there is no way that a empy sector to has a lot more cpu heavy than a central sector, like Argon prime... as it happends to me... Senator badlands is almost imposible to play inside that sector for me... and the only thing that are inside are a nebula and asteroids... and maybe 1 or 2 patrols of yaki fighters... you cant compare it with a centrla sector like argon prime or kingdom end... im having a lot more fps in kingdom end/argon prime that has a lot of stations, merchants, military and civilians... senator badland is empy and i have 5 fps in there, at most... so...

i know that x-series are highly cpu-intensive, but right now there are a big optimisation problem in how the game hadles the driveo driver.

have you actually tryed reduceing your graphics to see if it speeds it up signfically ?

maybe you are having specific graphics related problems which could be problem drivers on your PC, there have been drivers in the past that can reducded perforance. But this is nothing the game can do anything about, as the drivers are seperate.

The problem is that your assuming that the more stuff in sector will only effect your GPU. Which is false. There are things like the AI for all those objects, the collosion detection required for them all, etc. Which are all done on the CPU. So increase the number of ships in a sectors increases the processing on both the CPU and GPU. So if the CPU cant compute the collosions and ai quick enough, then its not taking longer to tell the GPU to render and the slower the game appears to be.

obviosuly theres a chance that some models have problems and causing slow down, which will be a bug in the model iteself, rather than in the actual rendering engine of the game which you seem to suggest

[ezra-r]

Yeah but most of the performance problems are more GPU-Ralated than CPU related... its very wierd, i mean if i jump to senator badlands i will have a very very very bad fps.... not reason at all because the sector its empy, if i jump to argon prime, well the fps are about x4-6 than in senator badlands...

Also, keep running gpu-z on background doing a sensor log of my video card, this is the only game that carry my GPU (a HD4850) at 100% activity... not even Crysis at High settings could do that, because my CPU bottlenecks my GPU... the performance problems of TC are more in how it hadles the video driver than CPU tasks...

I'd say its quite the opposite, and it is just CPU related but it seems we have all flavours.

Why? Because same sectors get me different fps depending if game is just started or it has been a while running.

[ezra-r]

Hey Observe, how is it going?

Im back for X3:TC.

About your MOD it is a great idea, just I'd like to let you know I wouldnt be in for a MOD erasing systems from X3:TC

[OOZ662]

I'd probably use this if it didn't change the universe map. That's kinda a big point of this game...

[Observe]

1) Most of those jobs are civilians, and militaries. The civilians can be removed, with minimal gameplay effect outside of lack of scenery. Traders are likely best not greatly reduced, to keep the economy humming.

I'm not sure I like the idea of removing entirely, but certainly those are clearly areas which could probably stand reduction.

The total max number of Jobs using the local.stationtostation function = 2127. These include things like: Civilians, Media, Cleaning, Locksmith, Passenger cruise, Emergency doctor, Archeology, Salesman, and many others. Consider the total number of Jobs for TC = 7746, the local.stationtostation function accounts for a substantial percentage of overall Jobs. Other Jobs using functions special.* account for potentially another 673. These include mainly taxi's and some prisoner transports. Not of these affect the economy from what I can determine.

In case you are interested, I am willing to offer my assistance and co-operation with this project. Minimally, my script would likely work well together with your reduced Jobs as it stands. But furthermore, it would probably allow for almost complete removal of military Jobs, while keeping the same or better military performance, for the same or less CPU load.

Thanks for the offer ttl! Right now I want to see how much Jobs alone can be optimized before applying any additional scripts. The simple fact is that X3-R has less than half the maximum number of potential Jobs compared to TC. Clearly TC hasn't added all that much to the game over Reunion. Sure it has a few more sectors, but nothing warranting the enormous number of Jobs additions from what I can determine.

It's a script that only spawns heavy military presence only when needed, thus allowing the removal of most military Jobs. It's still work in progress, after conversion from X3R XTM mod (which had Job numbers similar to X3TC).

Actually, I've never played XTM, but I did download to verify your assertion. It turns out that while XTM has Job numbers larger than X3-R, it is still about half of what TC has.

...just I'd like to let you know I wouldn't be in for a MOD erasing systems from X3:TC

I'd probably use this if it didn't change the universe map. That's kinda a big point of this game...

Agreed. I think we can keep the TC map, while achieving significant performance improvement, without entirely eliminating any Job type.

FYI: As a side note, my correspondence with Doubleshadow indicates he is unlikely to provide compatibility for his Editor with TC Jobs. Therefore, I will be providing a separate Jobs Editor for those interested.

[Moonrat]

Way to go, this mod is exactly what I was thinking X:TC needs. X:TC + X:TC/XTM map + smoooooth(er) frame rates.....If I had that, I'd believe that Santa Claus existed again.

[ttl]

If you put together a Jobs editor, I'd volunteer to be first in line to fire it up

However, Jobs is not so difficult to modify with just wordpad. Each line starts with the job-name, and I think the next two numbers are "Max Jobs" (of this type in universe) and "Max per sector" (of this type per sector). The name on the beginning of the line, and the script the Job runs, pretty much tell you what it does.

It gets more complicated when wings are involved, and Jobs call other Jobs to assist them. That is, if you want to decrease the number of escorts per average trader and/or military ship, in addition to the number of the "lead" ships themselves, which likely is something you should to get most performance for the best-behaving universe. Never done that, and don't know how.

[Observe]

However, Jobs is not so difficult to modify with just wordpad. Each line starts with the job-name, and I think the next two numbers are "Max Jobs" (of this type in universe) and "Max per sector" (of this type per sector). The name on the beginning of the line, and the script the Job runs, pretty much tell you what it does.

Editing Jobs in Wordpad (or other text editor) is easy enough if all you want is adjustment of "Max Jobs", "Jobs per sector", and other minor changes as you point out. However, if you are adding custom jobs, or making more involved adjustments, using a text editor can be a major pain.

For X3-R there are 118 Jobs fields, and X3-TC has 128. Most of the new fields for TC are related to Terran and new weapons additions. Also, the order of some of the old fields has changed with new Jobs. I probably wouldn't bother making a full-blown Jobs editor if not for our Transcend II mod having a completely custom galaxy, and accompanying Jobs file. Even Transcend I (soon to be released) has quite a few Jobs adjustments to accomodate the Mech's and others.

[amirite]

I don't think the problem is strictly CPU-based. I mean I am sure there's a performance increase, but the real problem is that this game is terribly unoptimized in every fiber of it's being. This mod is an interesting idea, but if you're only able to address about 30% of the entire overall problem, while being required to actually CUT things from the gameplay, what's the point?

I don't mean to crap on your idea though. It's a good one. However maybe it would be better to approach it as a general performance-enhancing mod, rather than a 'back to X3' thing.

Not to undercut the work you've already done either. These are some pretty interesting findings especially the fact that all those numbers are twice what they once were.

[Observe]

I don't mean to crap on your idea though. It's a good one. However maybe it would be better to approach it as a general performance-enhancing mod, rather than a 'back to X3' thing.

You are right amirite. The thread title "X3-Reunion performance callback" is intended to highlight the fact that Reunion ran sooo much better than TC does, and that Jobs adjustment is (based on experiments) one of the primary differences between the two. The simple fact is that by running TC with Reunion-level Jobs results in HUGE TC performance gain.

You are right that there are other factors involved, and our team is exploring some of them as well. [ external image ]