[Mod/Patch] X3-Unleashed [RC2] [Aug-06-2010] - X3 Performance gains up to 38%

The place to discuss scripting and game modifications for X³: Terran Conflict and X³: Albion Prelude.

Moderators: Moderators for English X Forum, Scripting / Modding Moderators

Post Reply
User avatar
Posts: 430
Joined: Sat, 16. Sep 06, 22:48

[Mod/Patch] X3-Unleashed [RC2] [Aug-06-2010] - X3 Performance gains up to 38%

Post by Jaga_Telesin » Mon, 28. Sep 09, 07:13


Current Version: [RC2] Release Candidate 2 NEW
Dated: August 6, 2010
Compatibility: X3:TC 2.1 - 2.7.1
Tested speed gains: Up to 38% more FPS (due to reduced CPU load) > Latest Test Results <
Notes: All bugs and economy issues have been fixed. RC2 contains additional tweaks and optimizations. This should be the final release candidate.

Ship Rebalance Mod (SRM) Compatible Version (by vkerinav, v3.1):

Older Versions:
RC1 - July 26, 2010

Beta 1 - Sept 28, 2009


X3-Unleashed improves performance for X3:Terran Conflict by reducing load on your computer's CPU. It does this by selectively reducing the total number of jobs (flights) in the universe to be closer to realistic and manageable levels. All flights were classified and weighed accordingly when adjusting their numbers, so that the universe as a whole still feels alive, and balanced.
  • Reduced flights in the universe means a return on FPS. In the case of X3-Unleashed, reducing overall flights by 47% provided our test system a return of +38% FPS (tested with repeatable low-graphic-demand conditions).
  • Less ships moving between sectors (ex. traders) means less traffic jams at gates and in busy traffic lanes.
  • Less traffic jams means flights to destinations take less time. For a NPC trader that means more trades per hour, making up directly for having less traders in the universe.
  • Less NPC flights (especially civilians) means less stations with completely full docking rings, enabling the player to dock without hassle, and to dock more of their own ships.
  • Fewer flights in the player's sector (on average) means less wasted CPU cycles when those ships utilize collision avoidance routines.
  • Keeping civilian flights going (a small but noticable number) adds flavor and texture to the universe, while not significantly adding to CPU overhead. Typically these flights do not have escorts, do not trade, stay in one sector, do not interact with other ships (except to flee hostiles), and simply move station to station. Combine their low CPU use with smaller collision and hitboxes than most ships, and their impact on FPS in X3-Unleashed is almost unnoticable.
  • Far less civilians overall means less avoidance calculations when the player is in a sector, less radar tracking for the player, and less docking/undocking at stations.
  • Less traders means far less overhead where that set of scripts is concerned. Searching the universe for best buys/sells, flight management, purchasing and selling, etc. And given that they now fly more routes per hour (see previous point), they attempt to make up for their reduced numbers.
  • Rogues (Yaki & Pirates) remain untouched, with numbers equalling those from vanilla.
  • Classic Enemies remain between 85% and 100% of original numbers.
  • Military numbers were only marginally tweaked downwards, in anticipation of most players using a script like Race Response Fleets (RRF) or Improved Races (IR). The immersive impact of military ships in a vanilla game was rather minimal at best as it was. Their relative population in the universe however, has actually risen from 8% in vanilla to 11% in Unleashed.
  • When determining which flights to trim and by how much, all were categorized both by type and function, and given due consideration to their impact on the gaming experience, the X3:TC engine, and their rendering requirements.

X3:TC v2.1 or higher (2.7.1 recommended)
Knowledge of "Fake Patch" installation technique

Choosing a Version of Unleashed:
You need to chose which version you would like to run before copying the files. Only one can be installed and run at a time:

- Unleashed - No Civilians (no civilian flights - lowest possible CPU load)
- Unleashed - With Civilians (with reduced civilian flights to add flavor)

New Game or Existing Game?
X3:Unleashed works perfectly fine with both scenarios - either a brand new game or one in progress.

When adding Unleashed to a game currently in progress, flights will be slowly reduced to Unleashed's numbers, as the X3 engine cleans them up naturally over time.

Unleashed installs in your X3:Terran Conflict directory as a fake patch.
  1. Copy the .cat and .dat files to your Terran Conflict directory (you'll see other cats/dats there)
  2. Rename Unleashed's .cat and .dat files so they are now either the highest number, or 1 less than the Plugin Manager's (if used). For example: if adding to a vanilla installation of 2.7.1 X3:TC, Unleashed's cat/dat pair would be copied into the main directory and renamed 13.cat and 13.dat. If you install other fake patches with number pairs higher than Unleashed, be absolutely certain that they do not modify the game's Jobs.txt and JobWings.txt files. If they do, rename Unleashed's cat/dat pair so it once again has the highest number.
  3. Placeholder .txt files have been included to help you remember what cat/dat pair you just installed. Be sure to change the ## at the beginning to whatever you used when renaming the cat/dat for Unleashed.
Reverse the install process detailed above: remove Unleashed's cat and dat files, and if you had any other cat/dat pairs with higher numbers, reduce them all by 1.

NOTE: Removing Unleashed and then loading a savegame will populate the universe with additional ships of all kinds over time (generally this takes 30-60 minutes).

Mod Behavior / Universe Changes:
The overall number of flights in the universe will be lower than those found in a vanilla X3:TC game, but the universe will still feel populated and alive. You may notice more free docking ports at stations, less traffic jams at gates, less collisions at gates and in nearby traffic lanes, and higher FPS among other improvements.

Performance Notes:
X3:Reunion and Terran Conflict have been known to be harder on your CPU than your graphics card. Most systems for this reason have the CPU as the bottleneck when you play X3, and you notice FPS skyrocket when you look away from stations and ships into empty space. Unleashed helps alleviate the CPU bottleneck, though it cannot completely remove it. If your graphics card is being stressed heavily (such as in a view with lots of ships and/or stations), the gains noticed from Unleashed will be less, since it only affects your CPU's workload.

This is why metrics testing on Unleashed was performed with different views, and the numbers averaged to give a general idea of expected improvement. It will not be the same on all systems, nor will it be the same from sector to sector within X3:TC. But it *will* help your game run smoother in nearly all circumstances, except those with extreme demands on your graphics card for rendering large objects or a large number of objects.

If your system is among the few where CPU power vastly exceeds graphic rendering power (new CPU, old video card), you may find that your performance gains are minimal at best. This simply means its time to upgrade to a new video card!

Last but not least - absolutely no guarantees are made regarding performance gains on your machine, your copy of X3:TC, or your blend of scripts/mods and savegames. All setups are unique (PCs do have personalities after all), and differences can be due to the smallest imaginable thing. For suggestions on testing the effects on your machine, see the following post regarding metrics. While our testing showed gains between 20% and 35%, it could very well be different for others.

Known Problems:
None at this time - please report any suspected issues in the official forum topic.

No flights (jobs) were completely removed from the Jobs.txt file. This was intentional, in case they are required by scripts or the Mission Director for quests. It also helps with future compatibility, when trying to merge changes from two Jobs.txt files. In the case of Civilian flights in the NoCiv version of Unleashed (for example), their numbers were simply set to 0/0.

Conflicts with other Mods/Scripts:
If you use the "No Civilians" mod by Graxter (or a similar variant), you should uninstall it prior to using Unleashed. Unleashed comes with a NoCiv cat/dat if you would like the absolute fastest game possible at the expense of not seeing those ships around. Similarly, any other Mod that makes changes to the Jobs.txt file (in the types directory) should either be uninstalled, or have it's Jobs.txt file merged with Unleashed's by someone who knows what changes were made by the other mod.

Total Conversions like Transcend that use custom Jobs.txt files are definitely not compatible with Unleashed at this time, as it was designed for use with a standard Plot (or non-Plot) game of Terran Conflict. It uses it's own custom Jobs file, which is also highly optimized.

Used Files:
  • types/Jobs.txt
  • types/JobWings.txt
Credits and Kudos
- Observe
- Litcube
- Serial Kicked
- mark_a_condren
- vkerinav
- madmartyr72
- Aragon Speed
- Targ Collective
- Doubleshadow
- The X community
- Egosoft


Developer Notes:

Feedback is welcome. Merging with other packages is encouraged and supported - PM prior for details.
Last edited by Jaga_Telesin on Sat, 7. Aug 10, 18:36, edited 20 times in total.

User avatar
Posts: 430
Joined: Sat, 16. Sep 06, 22:48

Post by Jaga_Telesin » Mon, 28. Sep 09, 07:16


The Premise:

For many months it was suspected that X3 engine jobs (flights) produced unnecessary strain on the engine; dropping frame rates proportionately. Several people (Observe among the most persistent) looked into the issue, doing benchmark tests and trials. They proved fruitless however, and the idea was left untapped for standard X3:TC games. No longer - the idea was revived, refined, and tested with the latest 2.5 patch for Terran Conflict.. with surprising results.

While results were costly in terms of flights removed, it was found that the X3:TC engine, while under light graphic rendering requirements for a scene, responded very well to changes in total flights. An investment of -10% flights yielded nearly 6% FPS improvements, among additional benefits like reduced traffic jams, free docking slots, etc. For those reasons alone, it was put up on the drawing board and revisited.

One of the very first tests done to determine the viability of the mod, was using a "blank" Jobs file, with a single token flight in it to simply give the engine something to do. When I noticed performance over 200 FPS, I knew gains were possible.

The Method:
The first step in knowing where to "trim the fat" was in knowing what I had to work with in the first place - what number of flights, what types, how many wingmen they have, what scripts they run, what kind of travel do they perform in the universe, etc. The Jobs.txt file was loaded into a spreadsheet, and summarily dissected and categorized. The largest of those categories was type of flight, and it's proportion to other flight types in the universe:


As can be seen, the overwhelming number of flights in the universe were due to two groups: civilians, and traders. We already knew we could cut a large number of civilian flights and still retain the "flavor" they gave Terran Conflict, and that the scripts traders use often increase their impact on the CPU due to constant activity. Other groups were also selectively put on the table, each one reviewed and some trimmed slightly. When all was said and done, I ended up with some new totals and their relative percentages in the universe:


Our new snapshot of the universe, and a comparison from where we were to where we ended up, looks like this:




Believing this was as far as I wanted to trim the universe (more would be detrimental), I stopped, reviewed the work, made a few adjustments, and started testing. Incidentally - the final # of flights ended up fairly close to those from X3:Reunion, which is a great milestone to hit. Performance from X3:R is much better than X3:TC in a vanilla game, and others have done some testing on the difference between the two engines in the past.

Several test games were started and played for several hours, to gauge the number, type, and distribution of flights across different sectors. Everything looked good, and didn't feel a whole lot different than vanilla, which was great to see.

The Metrics:
All benchmarking metrics were performed on the following system:
  • Q6700 @ 2.66ghz, 8gb RAM
  • Vista 64-bit Ultimate, SP2
  • Nvidia GTX280 @ stock speeds
  • Core Affinity Resident used to lock software to specific cores
  • X3:TC installed to a dedicated, high performing SSD drive
  • X3:TC patched to v2.5 RC3
  • No other applications (other than FRAPS) running during testing
  • X3:TC graphic settings set to realistic levels for typical gameplay (but not enough to push the GPU hard for this testing)

In trying to come up with a good benchmark and test routine for Unleashed, I decided that the most important factor was the change in CPU load that the change in flights produced. To that end, I didn't want other ships and/or stations in the view when recording FPS, but I *did* want to have an average number of ships in the same sector as the player's ship. The reason for this is that overall ship density in most sectors should be lower, and hence collision avoidance, radar tracking, and a number of other factors would influence the test results. Well, we wanted them to after all, since most players are in sectors with ship activity during the course of a normal game. However I decided against games with the plot enabled (for initial testing), to eliminate another unknown.

To satisfy those needs, and give an accurate number of scenarios to test against, I decided to gather three points of data for each test done:
  • Within radar range of other ships, but not viewing them.
  • Outside of radar range of other ships, and not viewing them.
  • Outside of radar range of other ships, having had SETA running for 25 in-game minutes .
This gives me an accurate idea of how flights affect CPU load, both at the start of a new game, when nearby ships are out of radar range, and how the game "levels out" after a small period of time.

Because the non-plot game start lands the player in a central sector with lots of activity (Argon Prime), I decided to use it and found a decent routine to obtain my data for recording:
  • Start new Custom game: X-Universe
  • Adjust ship nose up until the right hud mark near the center just touches the edge of the visible moon.
  • Ensure no other ships are visible (easy to double check against expected FPS)
  • Select South Gate (in lieu of the tutorial ship which flies around).
  • Record average FPS seen in the next ten seconds (Data Point 1)
  • Hit booster on ship, stop moving when distance from South Gate reaches 35km.
  • Record average FPS seen in the next ten seconds (Data Point 2)
  • Display Pilot statistics (which includes time flown), engage 600% SETA, and wait until time played hits 25 minutes.
  • Disengage SETA, clear pilot menu visibility.
  • Record average FPS seen in the next ten seconds (Data Point 3)

After doing around 10 test runs with this and vanilla, I came to the realization that because it is a truly dynamic system with very random results for ships in a new game's sector, I decided to toss out the top 2 and bottom 3 results, so that they didn't skew the averages . This was done for each test and each set of tests, without exception.

The Results:
Vanilla results looked very regular, as expected. Data Point 1 had other ships present in the radar, so was more impactive on FPS. Data Point 2 was outside range of any ship in the radar, so FPS went up. Allowing the game to normalize and staying out of radar range showed an expected dip in FPS, but not completely back to Data Point 1 levels.


Now that I had a baseline for Vanilla X3:TC, I wanted to see what the absolute upper limit was, so I could establish a reference for any FPS gains the mod might see. To that end, I created a completely empty Jobs.txt file (no flights, no content, nothing), and ran the same set of tests that were run against Vanilla.


It was easy to see the headroom in FPS that zero flights afforded. Framerates skyrocketed to almost double vanilla rates, simply due to no ships in the game. Next up, I decided to test an absolutely minimal number of flights, to see what impact it would have on FPS. I already knew that even 1 flight made performance dip considerably, so I had to ballpark what I thought the game could run with as an absolute minimum:


As can be seen, even with only 800 or so flights in the universe, performance dipped from our wonderful 85-93% gain, down to 54-59%. A loss of 30% of our gains, by adding in a nearly unplayable number of flights. Obviously the X3:TC engine is very touchy in response to changes made to the Jobs.txt file. And - no one in their right mind would have a universe with only 800-ish flights in it, since the economy would be dead, and sectors would have a few ships in them on average. This is good for testing, bad for gameplay.

Lastly, it was time to test Unleashed's flight numbers with the same benchmark procedure:


As expected, performance was up between 29% and 33% on average, with an established game. And the best part - the universe was still alive with traders, military, police, hostiles, rogues, civilians, and other ships.

Since we have more than two data points, and four separate situations, we can try and get a "big picture" view of what is happening with the X3:TC engine in terms of Flights vs FPS, in a low-GPU-demand situation (i.e. no ships/stations are drawn, load on the CPU is the primary consideration):


Putting both sets of numbers into percentages and on the same graph yielded very interesting results, and tells us exactly what is happening behind the scenes:


By looking at the second graph, it is easy to tell that the initial "cost" in gaining FPS, is a significant investment in lost flights (due to the steeper blue line). Had it been any more steep, it would probably not have been nearly as successful in reaping additional FPS gains. And, since additional flight cuts would still yield small improvements in performance (and be detrimental to immersion and the function of the universe), flight levels remain close to 50% in Unleashed.

Clearly there is a direct relationship between # of flights in the universe, and the impact on the CPU and its ability to deliver higher performance within the X3:TC engine.

This set of tests was completed in real-world scenarios with actual game starts, and concentrated on normalized data. Having said that, it isn't an actual "gameplay test", with visible ships on the HUD, combat situations, etc. That wasn't the main goal in the end, but rather finding out what impact flights do have on the universe.

As a sidenote - enabling the plot by using the "Anonymous Argon" game start saw very little overhead added to the CPU. Because of that, I discarded any idea of testing with the plot enabled, as the percentages should have stayed nearly the same.

Actual use during the course of gameplay may tell different stories, since ships and stations will be visible, putting variable demands on the GPU and CPU. However, as was shown - Unleashed can and will lift extra processing load from your system, and thereby allow your system to run faster (if it can). Systems which are more CPU-bottlenecked will benefit more from Unleashed, as will those that are fairly CPU/GPU balanced, as X3:TC is more CPU-bound than anything else.
Last edited by Jaga_Telesin on Tue, 29. Sep 09, 04:42, edited 3 times in total.

User avatar
Posts: 4245
Joined: Fri, 20. Oct 06, 19:02

Post by Litcube » Mon, 28. Sep 09, 18:10

I wish more mods were attributed with descriptions like this one. Very clear mission statement.

Looks promising.

Posts: 12
Joined: Thu, 17. Nov 05, 20:18

Post by Tiroth » Mon, 28. Sep 09, 20:09

Very nice description, I wish more mods would try to reduce the performance footprint of X3:TC because in my opinion X3 plays best with hundreds(if not thousands) of ships per battle.

Are you planning to do anything graphics wise? Because I still get alot of problems with too many weapons(read missiles, m7m, and bombers) being fired at once.

Defiantly a mod to watch.

P.S. I have a modded game going right now (IR, CC, MARS and cheats pack) I think it may conflict with CC though ( I remember its manual saying something about it overriding and other fake patched mods) but I'll try it out and confirm when I have more time

User avatar
Posts: 430
Joined: Sat, 16. Sep 06, 22:48

Post by Jaga_Telesin » Mon, 28. Sep 09, 20:25


Thanks for the words of encouragement. :) I've already managed to do two full days of metrics testing to gauge the effects on X3:TC, so I'm fairly certain it'll do just fine in the long run. I need to find time after work today to put together the methods and metrics post, so that people can see where the mod makes changes, and how the test results were compiled.


Thank you as well. Not planning on doing graphical adjustments at this time, as they attempt to change what the engine does from a different perspective. Unleashed will work with the # of ships in the universe, their escorts, and their eventual effect on the game long term. I am attempting to find someone to help with a script that will further enhance Unleashed, though that will also affect the flights and not the rendering of any objects.

Posts: 12
Joined: Tue, 22. Jan 08, 01:09

Post by Dezeal » Mon, 28. Sep 09, 22:47

I'll give it a try with the civilian version. If this works like it should, it will make the one advantage Reunion has over TC disappear (For me personally) Thank you :D I'll report back on any side effects (Though if this does make the game better I probably won't be seen for a good long while :P )

User avatar
Posts: 430
Joined: Sat, 16. Sep 06, 22:48

Post by Jaga_Telesin » Tue, 29. Sep 09, 02:11

Hope it turns out well for you Dezeal. I'm about to start a new game myself (just for fun), and am looking forward to some silky smooth framerates. :)

NOTE: Just finished the additional post for Unleashed regarding Methods, Metrics, and other juicy tidbits. Feedback is welcome, as always. Enjoy.

User avatar
Posts: 4245
Joined: Fri, 20. Oct 06, 19:02

Post by Litcube » Tue, 29. Sep 09, 02:48

I just read your second post. Jaga, you win the award for best documentation.

Very interesting read. I had simliar results myself when I played around with my benchmark script (there's a thread here somewhere with the script posted). The script allowed me to have exact positions when benchmarking. I didn't have to touch the mouse.

Anyway, I'm going to use this, and I rarely trust 3rd party modifications.

Also, take a look at some of the bugs that the Unoffcial Bug Fix mod found. You might be interested in seeing if they've been fixed in 2.5, or if you'd want to implement the corrections within your jobs.txt.

User avatar
Posts: 430
Joined: Sat, 16. Sep 06, 22:48

Post by Jaga_Telesin » Tue, 29. Sep 09, 02:58

Good point about the Unofficial Bug Fix Litcube, I hadn't considered they might be making alterations to the Jobs file. Looking at it they made minor alterations to things like Argon One, Boron M8, Pirate reactions, etc. I'll do a file comparison against vanilla Jobs and see.

If the community thinks enough people are using the Unofficial Bug Fix Mod, I'll incorporate the changes into Unleashed by default. It is very simple and easy changes to make, on my end.

User avatar
Posts: 2948
Joined: Fri, 30. Dec 05, 18:47

Post by Observe » Tue, 29. Sep 09, 04:09

Excellent work Jaga_Telesin! Image

I am happy to see you applying your performance improvements to vanilla TC. Your fine efforts mean I can now abandon my "X3-Reunion callback mod" - which has been languishing because of my focus on Transcend. :)

Keep up the good work!

Targ Collective
Posts: 1733
Joined: Wed, 4. Feb 09, 22:42

Post by Targ Collective » Tue, 29. Sep 09, 06:24

Oh, well done Jaga.

One to look at, especially for low-end machines. One wonders why Egosoft didn't consider dynamically altering the population of the universe to make things easier for low-end machines... And less competition means *I* get a bigger slice of the pie.

I like what I see!
Learn to use stickies. Look here for mods for X3TC and here for guides across everything since X2. If you haven't read the Forum Rules do so before we beat you with sticks. Above all have fun.

User avatar
Posts: 2948
Joined: Fri, 30. Dec 05, 18:47

Post by Observe » Tue, 29. Sep 09, 06:47

I think part of the problem is "sector creep". With X3-R, the game may have reached an optimum balance between number of sectors and AI Jobs.

In the case of TC we have quite a few additional sectors. Perhaps the TC map (and associated Jobs) has simply tipped the balance too far in the "wrong" direction.

Personally, I think the map could be modified with fewer Terran sectors with good result. In any case, I believe Jaga_Telesin is bringing performance balance back to the game with his efforts. :)

Logain Abler
Posts: 2254
Joined: Mon, 31. Oct 05, 09:44

Post by Logain Abler » Tue, 29. Sep 09, 10:17

Great work, I'll give this a try tonight :)


User avatar
Posts: 430
Joined: Sat, 16. Sep 06, 22:48

Post by Jaga_Telesin » Tue, 29. Sep 09, 22:13

Thanks for the positive feedback everyone, though it is based largely on numbers and presentation. :D

It will be interesting to see what people who use Unleashed think of the results. I started a new modded game last night for fun, played it for a few hours, and had a blast with almost zero chop in my framerate. Only the most densely packed sectors (like PTNI) were sluggish when viewed in their entirety, which is actually due to graphical lag.

Note: I would like to do a script to remove the "orphaned" flights from the galaxy instead of waiting for the God engine to do it over time, but don't have the X3 scripting experience necessary. I've looked at a few possible solutions, none of which worked when I tried. If there is someone with decent scripting experience who has a small bit of time, it would be great getting help writing a cleanup script for the mod. There are other uses it could be put to as well, in the long run.

Posts: 1132
Joined: Sun, 19. Oct 08, 18:46

Post by ThisIsHarsh » Tue, 29. Sep 09, 23:13

Excellent work! Must have taken a lot of time and effort. I have some experience gathering and presenting performance statistics... not fun.

I've often wondered, though, how much of a performance hit is caused by the complexity of scripts, rather than the number of ships...

In an effort to complicate your life yet further - what would be really awesome is some performance metrics for each job type. For example: eliminate all military, civillians, etc, leaving just freetraders, then run your performance benchmark. Can also then divide by number of freetraders to get average FPS used per ship, as a metric for CPU load of that job script. This isn't a foolproof test since there could be interaction between jobs, but should give a good overview.

Flagging job scripts that are badly coded, unnecessarily bloated, or could be replaced with 'smarter' scripts running on less ships, would be a great cue for us scripters to get to work on better solutions. I've seen the code of several vanilla scripts which I thought could cause a performance problem, but of course never tested it.

Anyway, again, very good work. Keep it up!
There are 10 types of people in the S&M forums - those who understand binary, and those who don't.

Black holes are where God divided by zero.

Post Reply

Return to “X³: Terran Conflict / Albion Prelude - Scripts and Modding”