[TOOL] X3 Customizer 3.16 (added FL support)

The place to discuss scripting and game modifications for X³: Terran Conflict and X³: Albion Prelude.

Moderators: Moderators for English X Forum, Scripting / Modding Moderators

Post Reply
Lone Machete
Posts: 29
Joined: Sun, 3. Dec 17, 06:01
x3ap

Post by Lone Machete » Sun, 7. Jan 18, 10:58

to stop god removement entirely is a bad idea as you can read in the relative topics since number of stations is limited and god is more likely to build up stations in some areas of the map than in others, designing a very poor universe with only one truely "prospering" area while the rest stays more or less empty.

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Mon, 8. Jan 18, 00:51

Can you provide links to such topics? I didn't run across any good documentation of GoD behavior in my searches; mostly it was just extrapolations based on individual experiences.

At any rate, the transform is reversible, for players to use as they see fit. Eg. turn off station removal when starting a new game, eventually place m5s at every station the player wants to keep, take a minute to turn station removal back on (save, rerun the customizer without the transform, reload), and let it go to work.

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Sun, 15. Apr 18, 03:46

Major update to v3.2.

Support for compiling to a standalone executable has been added, removing the python requirement. The binary release can be found on the github releases tab, as a zip file. Note: this is still a command line tool at the time, so don't be surprised if you double click to launch it and nothing happens. :)

Backwards compatibility may be slightly messy, as the older versions did not keep a log of which files they had written. Consider doing an empty run of the prior version first to undo its changes (using a command script with the correct path but no transforms). Alternatively, do at least one run on the new version before moving files out of the old sources folder.


The internal file management system has been reworked; it can now find source data from the standard game directory or from the cat/dat files, and no longer requires (though still supports) a high priority source folder. Existing files will be renamed (suffixed with .x3c.bak) when they would conflict with modified files; they have their name restored if the modified file is not created on the next run. A json log will be generated stating which files were written (with their sha256 hashes), as well as which renamed, to inform later runs.

The general source code structure of the project has been reorganized in a number of ways, though this is mostly transparent if you don't want to edit source. Various misc refinements have been made. The dependency on the scipy package has been removed, though if that package is available it can be used to create smoother scaling equations for transforms that support them.


As a bonus, an Allow_Valhalla_To_Jump_To_Gates transform has been added. Use at your own risk; I recommend some sort of ship size mod or gate mod to avoid crashes. This requires a KC edit, so currently will not work on mods that edit x3story.obj (eg. LU).

Thanks to akruppa (https://forum.egosoft.com/viewtopic.php?t=361709) for an example on how to decode the cat/dat pairs.

User avatar
Joubarbe
Posts: 4796
Joined: Tue, 31. Oct 06, 12:11
xr

Post by Joubarbe » Sun, 15. Apr 18, 05:14

Nice!

User avatar
RoverTX
Posts: 1426
Joined: Wed, 16. Nov 11, 18:37
x4

Post by RoverTX » Tue, 17. Apr 18, 06:38

Lone Machete wrote:to stop god removement entirely is a bad idea as you can read in the relative topics since number of stations is limited and god is more likely to build up stations in some areas of the map than in others, designing a very poor universe with only one truely "prospering" area while the rest stays more or less empty.
What relative topics? I also have seen people say this alot, but I have never seen GoD actually add in any stations to fill missing gaps. It just seems to cull stations.

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Sun, 10. Jun 18, 07:47

Updated to 3.3. This fixes a bug that snuck into Adjust_Weapon_Fire_Rate (thanks andu for catching it), adds some documentation on which transforms have been tested to not work with LU (mostly obj and script edits), and adds more graceful handling of any transform errors.

andu
Posts: 17
Joined: Thu, 21. Jun 07, 23:35
x4

Post by andu » Sun, 10. Jun 18, 21:51

Hi, i've downloaded the latest version and using this config file

Code: Select all

'''
Example for using the Customizer, setting a path to
the X3 directory and running some simple transforms.
'''

# Import all transform functions.
from Transforms import *

Set_Path(
    # Set the path to the X3 installation folder.
    path_to_addon_folder=r'[...]\x3 terran conflict\addon',
    # Optional subfolder with high priority source files to be modified.
    # By default, this is relative to the addon folder.
    #  source_folder='addon'
)

Adjust_Weapon_Fire_Rate(scaling_factor = 4)
It doesn't change the rate of fire.

command used

Code: Select all

X3_Customizer.exe User_Transforms.py

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Sun, 10. Jun 18, 22:32

Can you report what gets printed to the console when you try to run the script? If it works correctly, the printout should look like:
Attempting to run User_Transforms.py
Successfully ran Adjust_Weapon_Fire_Rate
Transforms Complete
If there are pathing problems to the user script or to the game directory, those should print out error messages.

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Mon, 11. Jun 18, 10:58

Updated to 3.4. This changes how the obj file edits are carried out to use a dynamic pattern match instead of hard offsets, to make it more robust against different obj file layouts (due to patches or other mods).

Adjust_Max_Speedup_Rate and Stop_GoD_From_Removing_Stations (untested) now apply to LU, along with Adjust_Max_Seta (which redirects to a t file edit if the LU t file found). Most other obj patches don't really map to LU since it hardcodes many of the same changes.

Stop_Events_From_Disabling_Seta has a new option, 'on_frame_input', which will prevent seta from turning off when opening menus.

GageDragon
Posts: 59
Joined: Tue, 26. Apr 16, 01:21
x4

Post by GageDragon » Mon, 11. Jun 18, 22:58

Hi. Trying to figure this out but keep getting path error. I am using xrm and have set the path to the xrm addon folder and it works fine using the example_transforms but when I try using the authors_transform and change it to the same path I get the path error.

Exception: Path to the AP/addon folder appears invalid.
[1372] Failed to execute script X3_Customizer

This is copy straight from both files, as you can see the paths are the same.

example path, Set_Path(
# Set the path to the X3 installation folder.
path_to_addon_folder = r'G:\X3xrm\addon',

author path, elif XRM:
Set_Path(
path_to_x3_folder = r'G:\X3xrm\addon',


Any idea what I am doing wrong other than I am an idiot (this I already know lol). Have tried changing some other lines at the other 2 paths and I get other errors then like indent error or syntax.

edit: or do I need to set it to the base folder and not add on.. hmm I will try that now...

edit2: well that was progress but still gave an error after running alot of things

OSError: Not a gzipped file (b'Q\x86')
[6272] Failed to execute script X3_Customizer

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Mon, 11. Jun 18, 23:25

Interesting. The path thing is as you say, taking either the x3 or addon folder, though I can touch up the error message a little bit. The gzip thing I will have to read up on. The x3 .pck files are gzipped versions of text files; when searching for the file to modify, if the tool finds a pck file then it uses the python gzip package to unpack it for reading/editing. I have never had trouble with it on my end, so I need to learn more about what gzip is doing to get an idea of what is going wrong for you. Since the file loader is outside my normal transform exception handler, the problem stops the entire tool run; that I could touch up as well.

GageDragon
Posts: 59
Joined: Tue, 26. Apr 16, 01:21
x4

Post by GageDragon » Mon, 11. Jun 18, 23:42

A couple things so you dont waste time looking into anything if its all my fault..
I notice when I run the authors_transform it is looking at python in programdata, which I do not have, it's after that the gzip error hits. I also do not have a source folder since I am lazy and have not opened xeditor yet and tried to figure out how to do that lol. If either of those are relevant.

Still it is odd that it partially runs at G:\X3xrm but not at all at G:\X3xrm\addon

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Tue, 12. Jun 18, 01:27

3.4.1 has been pushed with some tweaks including nicer gzip error catching.


Just as a warning, the Authors_Transforms includes some options that are hard to roll back once added to the game if the game is saved, such as the factory and ship variants (which get added to shipyards). I mostly make that file available for extended examples of calling transforms.

For the game path, these should hopefully be functionally equivalent:
path_to_x3_folder = r'G:\X3xrm'
path_to_addon_folder = r'G:\X3xrm\addon'

For the ProgramData/Python thing, it might just be a relic of how the executable is packaged up using PyInstaller. At least, if I rename my python distribution folder, the packaged customizer still runs okay.

For the gzip issue, I tracked it down to the X3 Plugin Manager, which will generate a TWareT.pck file but compresses it with something other than gzip. My current best guess from digging around and looking at some source code is that older X games and tools used to use deflate through zlib, so my current theory is that the plugin manager is using deflate still. I am poking around looking for a working solution now.

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Tue, 12. Jun 18, 03:15

3.4.2 is now up, and should support the X3 Plugin Manager generated pck file.

After spending some time going through x3 editor source code looking for how it handles decompression, and then the x2fd.dll source code (thanks doubleshadow for making source available), I found that the plugin manager appears to use an x2 pck format. (My theory about deflate instead of gzip was wrong.) Anyway, the customizer now tries x2 style decompression as a fallback when direct gzip fails.

GageDragon
Posts: 59
Joined: Tue, 26. Apr 16, 01:21
x4

Post by GageDragon » Tue, 12. Jun 18, 03:36

Aha will try 3.4.2 then. Thanks for this.

Was just coming on to say I kinda found a work around for the gzip thing in 3.4.1 by deleting the file created by plugin manager then running customizer and then rerunning plugin manager so it recreates that file. No error then. It might even be better off this way if customizer makes any ware size changes then plugin manager would need to recreate the file anyhow I believe.

Also not sure if I did it right but I grabbed the source and input_script files from the master and just put those into my folder. If that was correct then it is still erroring when I try g:\x3xrm\addon. I do see an x3_customizer_logs folder in addon so I guess it is working correctly regardless.

As far as changes being permanent I dont mind. I plan on starting over once I get this working anyway :D

I noticed in my rummaging through authors_transforms some settings for restoring ammo based weapons. I look forward to breaking my game more trying to figure that out next because I miss ammo weapons lol...

edit: well it still errors with the plugin file in folder as well as the path error, that is of course if I am doing things correctly by grabbing from the master on git and adding the changed files into my folder. Docu and readme both have 3.4.2 in them

edit2: grabbed the compiled version and it runs with no gzip error now but g:\x3xrm\addon now gives error Exception: Path to the AP/addon folder appears invalid(path: G:\X3xrm\addon\addon).. kinda funny but no not really lol

edit3: it is renaming the file to TWareT.pck.x3c.bak, maybe it really should be left alone and just rerun plugin manager after customizer?

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Tue, 12. Jun 18, 05:14

If you want to run the python code directly, I won't worry too much about putting out new releases for minor changes.

3.4.3 source code is uploaded, and changes handling of path problems a little bit: it will now raise a hard exception if the x3/addon path looks wrong instead of attempting to still run (with a command line option to override this behavior, if desired). The error messages should show what paths are being used for the base x3 and addon folders, so hopefully that teases out whatever problem you are having.

Are you sure you aren't using something like:
path_to_x3_folder = r'G:\X3xrm\addon' ?
Because that will make the tool think the path is to the base x3 folder, and 'addon' will get appended again. (Perhaps I will add a check for addon\addon to catch this.)

For the plugin manage integration, the plugin manager should run first, though it may work in either order. The customizer will see the twaret.pck file and use it as the base for any edits, then will rename the pck file when outputting twaret.txt to avoid conflicts (x3 prioritizes pck files over unpacked txt or xml files). The plugin manager might do the same thing in reverse, loading the customizer txt file for editing, but I haven't tested that yet.


Edit:
The plugin manager does not capture changes in the twaret.txt file when generating its own, so it will need to be run first, with the customizer being run after the plugin manager is closed. It appears user edits to twaret.txt in general are ignored by the plugin manager.

GageDragon
Posts: 59
Joined: Tue, 26. Apr 16, 01:21
x4

Post by GageDragon » Tue, 12. Jun 18, 18:32

Yes path_to_x3_folder = r'G:\X3xrm\addon' was exactly what I was trying to use with authors_transform because that worked with example. That explains the problem at least :D


Good to know about plugin manager then thanks for the fixes.

I would prefer compiled but there is no need to release the smaller fixes imo.

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Sat, 23. Jun 18, 09:54

Updated to 3.5.2.

The big addition deals with excess spaceflies generated accidentally by some mods, which accumulate over time and slow down the game. Eg. 85% slowdown of Seta after 10 game days in my own case. Mods affected include Improved Races 2.0 v1.08, Salvage Command Software v4.11, possibly Gazz' Missile Defense Mk2, and likely others.

Kill_Spaceflies will clean out the accumulation by editing the KC to swap the "is disabled" script command (only ever used on spaceflies) to instead kill them after a 10 second delay. (Without this delay, some other X3 engine problems crop up when the mods try to access the dead spacefly.) This uses some hardcoded addresses and so it does not work on LU currently, but that support could be added easily if wanted.

Prevent_Accidental_Spacefly_Swarms will edit the KC such that the "create ship" command no longer performs an undocumented spawn of a spacefly swarm. This will cause it to behave the way the affected mods were expecting, putting a stop to spacefly accumulation. This one does work on LU, though I don't have enough experience with LU to know if spaceflies are a problem in any of those mods.

You can check for spacefly accumulation in the script editor, checking how many of the spacefly scripts (idle or follow) are active. It should generally be 0 up to a couple dozen or so (depending on sector asteroids); problematic mods build up hundreds or thousands.


Some non-spacefly changes are included as well.

Remove_Factory_Build_Cutscene will edit the time delays to 0 for the cutscene played when placing a factory. Future refinements may allow a user adjustable delay (eg. make it take 25% as long).

Keep_TLs_Hired_When_Empty will stop a hired TL from wondering off when you drop the last factory in its hold.

_Benchmark_Gate_Traversal_Time is a toy transform which removes the inter-sector warp loading delay in games with large complexes by cutting out a low level function call. Side effects are unexplored, so it is not part of the main documentation, and running it requires a full path: Transforms.T_Obj_Code._Benchmark_Gate_Traversal_Time().


Other than transforms, this update converts the python code fully into a multi-layer package, making it generally easier to work with and matching the style I use elsewhere in my larger programs. A by-product is that user input scripts need to swap from "from Transforms import *" to "from X3_Customizer import *".

The documentation generator now supports a BB code format, and the top post in this thread has been updated to be somewhat pleasant.


Edit:
I thought XRM had a spacefly problem, but was mistaken and the problem was in SCS.

User avatar
Aldebaran_Prime
Posts: 1386
Joined: Sat, 20. Feb 10, 17:47
x4

Post by Aldebaran_Prime » Sun, 24. Jun 18, 00:21

I just tried version 3.05.1 to get running and to try your new Kill_Spaceflies() module.
I never used Python - so it's my first try.

I extracted your archive X3_Customizer_v3.05.1.zip to "C:\Users\Harald\Desktop\Neuer Ordner"

Then, I created a new transform-configuration "Spacefly_transform" in folder "input scripts" with the following content:

Code: Select all

# Import all transform functions.
from X3_Customizer import *

Set_Path(
    # Set the path to the X3 installation folder.
    path_to_x3_folder = r'C:\Users\Harald\Desktop\AP QA',
)

# Kill spaceflies spawned before Prevent_Accidental_Spacefly_Swarms
#  was added, lagging the game.
Kill_Spaceflies()
Then I opened a cmd window and changed the directory into the "...Neuer ordner" where I extracted your archive. There I tried to start the Transformation, but get a "Obj_Patch_Exception":
C:\Users\Harald\Desktop\Neuer Ordner>Launch_X3_Customizer.bat input_scripts\Spac
efly_transform.py
Attempting to run input_scripts\Spacefly_transform.py
Skipped Kill_Spaceflies due to a Obj_Patch_Exception exception.
Do I need to extract any obj.files from a .cat/dat before I start your converter?

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Sun, 24. Jun 18, 01:29

Hm, do you know if you are running any mods which change x3story.obj, or perhaps have an obj file from before AP 3.3?

That exception is thrown when a search is made on the obj file for a reference code section (basically a series of bytes); if that section is not found, or an unexpected number of such sections is found, the tool throws an error and skips the transform.

Different versions of the obj file will change around internal addresses for function calls. Since I swap out some function calls for the spacefly killing, the transform will only work on the version of x3story.obj I made it for (vanilla AP 3.3). Creating versions for other obj files wouldn't be hard, as long as I can get a copy the file to pick out the different addresses.

Post Reply

Return to “X³: Terran Conflict / Albion Prelude - Scripts and Modding”