[TOOL] X3 Customizer 3.16 (added FL support)
Moderators: Moderators for English X Forum, Scripting / Modding Moderators
Interesting. The path thing is as you say, taking either the x3 or addon folder, though I can touch up the error message a little bit. The gzip thing I will have to read up on. The x3 .pck files are gzipped versions of text files; when searching for the file to modify, if the tool finds a pck file then it uses the python gzip package to unpack it for reading/editing. I have never had trouble with it on my end, so I need to learn more about what gzip is doing to get an idea of what is going wrong for you. Since the file loader is outside my normal transform exception handler, the problem stops the entire tool run; that I could touch up as well.
-
GageDragon
- Posts: 59
- Joined: Tue, 26. Apr 16, 01:21
A couple things so you dont waste time looking into anything if its all my fault..
I notice when I run the authors_transform it is looking at python in programdata, which I do not have, it's after that the gzip error hits. I also do not have a source folder since I am lazy and have not opened xeditor yet and tried to figure out how to do that lol. If either of those are relevant.
Still it is odd that it partially runs at G:\X3xrm but not at all at G:\X3xrm\addon
I notice when I run the authors_transform it is looking at python in programdata, which I do not have, it's after that the gzip error hits. I also do not have a source folder since I am lazy and have not opened xeditor yet and tried to figure out how to do that lol. If either of those are relevant.
Still it is odd that it partially runs at G:\X3xrm but not at all at G:\X3xrm\addon
3.4.1 has been pushed with some tweaks including nicer gzip error catching.
Just as a warning, the Authors_Transforms includes some options that are hard to roll back once added to the game if the game is saved, such as the factory and ship variants (which get added to shipyards). I mostly make that file available for extended examples of calling transforms.
For the game path, these should hopefully be functionally equivalent:
path_to_x3_folder = r'G:\X3xrm'
path_to_addon_folder = r'G:\X3xrm\addon'
For the ProgramData/Python thing, it might just be a relic of how the executable is packaged up using PyInstaller. At least, if I rename my python distribution folder, the packaged customizer still runs okay.
For the gzip issue, I tracked it down to the X3 Plugin Manager, which will generate a TWareT.pck file but compresses it with something other than gzip. My current best guess from digging around and looking at some source code is that older X games and tools used to use deflate through zlib, so my current theory is that the plugin manager is using deflate still. I am poking around looking for a working solution now.
Just as a warning, the Authors_Transforms includes some options that are hard to roll back once added to the game if the game is saved, such as the factory and ship variants (which get added to shipyards). I mostly make that file available for extended examples of calling transforms.
For the game path, these should hopefully be functionally equivalent:
path_to_x3_folder = r'G:\X3xrm'
path_to_addon_folder = r'G:\X3xrm\addon'
For the ProgramData/Python thing, it might just be a relic of how the executable is packaged up using PyInstaller. At least, if I rename my python distribution folder, the packaged customizer still runs okay.
For the gzip issue, I tracked it down to the X3 Plugin Manager, which will generate a TWareT.pck file but compresses it with something other than gzip. My current best guess from digging around and looking at some source code is that older X games and tools used to use deflate through zlib, so my current theory is that the plugin manager is using deflate still. I am poking around looking for a working solution now.
3.4.2 is now up, and should support the X3 Plugin Manager generated pck file.
After spending some time going through x3 editor source code looking for how it handles decompression, and then the x2fd.dll source code (thanks doubleshadow for making source available), I found that the plugin manager appears to use an x2 pck format. (My theory about deflate instead of gzip was wrong.) Anyway, the customizer now tries x2 style decompression as a fallback when direct gzip fails.
After spending some time going through x3 editor source code looking for how it handles decompression, and then the x2fd.dll source code (thanks doubleshadow for making source available), I found that the plugin manager appears to use an x2 pck format. (My theory about deflate instead of gzip was wrong.) Anyway, the customizer now tries x2 style decompression as a fallback when direct gzip fails.
-
GageDragon
- Posts: 59
- Joined: Tue, 26. Apr 16, 01:21
Aha will try 3.4.2 then. Thanks for this.
Was just coming on to say I kinda found a work around for the gzip thing in 3.4.1 by deleting the file created by plugin manager then running customizer and then rerunning plugin manager so it recreates that file. No error then. It might even be better off this way if customizer makes any ware size changes then plugin manager would need to recreate the file anyhow I believe.
Also not sure if I did it right but I grabbed the source and input_script files from the master and just put those into my folder. If that was correct then it is still erroring when I try g:\x3xrm\addon. I do see an x3_customizer_logs folder in addon so I guess it is working correctly regardless.
As far as changes being permanent I dont mind. I plan on starting over once I get this working anyway
I noticed in my rummaging through authors_transforms some settings for restoring ammo based weapons. I look forward to breaking my game more trying to figure that out next because I miss ammo weapons lol...
edit: well it still errors with the plugin file in folder as well as the path error, that is of course if I am doing things correctly by grabbing from the master on git and adding the changed files into my folder. Docu and readme both have 3.4.2 in them
edit2: grabbed the compiled version and it runs with no gzip error now but g:\x3xrm\addon now gives error Exception: Path to the AP/addon folder appears invalid(path: G:\X3xrm\addon\addon).. kinda funny but no not really lol
edit3: it is renaming the file to TWareT.pck.x3c.bak, maybe it really should be left alone and just rerun plugin manager after customizer?
Was just coming on to say I kinda found a work around for the gzip thing in 3.4.1 by deleting the file created by plugin manager then running customizer and then rerunning plugin manager so it recreates that file. No error then. It might even be better off this way if customizer makes any ware size changes then plugin manager would need to recreate the file anyhow I believe.
Also not sure if I did it right but I grabbed the source and input_script files from the master and just put those into my folder. If that was correct then it is still erroring when I try g:\x3xrm\addon. I do see an x3_customizer_logs folder in addon so I guess it is working correctly regardless.
As far as changes being permanent I dont mind. I plan on starting over once I get this working anyway
I noticed in my rummaging through authors_transforms some settings for restoring ammo based weapons. I look forward to breaking my game more trying to figure that out next because I miss ammo weapons lol...
edit: well it still errors with the plugin file in folder as well as the path error, that is of course if I am doing things correctly by grabbing from the master on git and adding the changed files into my folder. Docu and readme both have 3.4.2 in them
edit2: grabbed the compiled version and it runs with no gzip error now but g:\x3xrm\addon now gives error Exception: Path to the AP/addon folder appears invalid(path: G:\X3xrm\addon\addon).. kinda funny but no not really lol
edit3: it is renaming the file to TWareT.pck.x3c.bak, maybe it really should be left alone and just rerun plugin manager after customizer?
If you want to run the python code directly, I won't worry too much about putting out new releases for minor changes.
3.4.3 source code is uploaded, and changes handling of path problems a little bit: it will now raise a hard exception if the x3/addon path looks wrong instead of attempting to still run (with a command line option to override this behavior, if desired). The error messages should show what paths are being used for the base x3 and addon folders, so hopefully that teases out whatever problem you are having.
Are you sure you aren't using something like:
path_to_x3_folder = r'G:\X3xrm\addon' ?
Because that will make the tool think the path is to the base x3 folder, and 'addon' will get appended again. (Perhaps I will add a check for addon\addon to catch this.)
For the plugin manage integration, the plugin manager should run first, though it may work in either order. The customizer will see the twaret.pck file and use it as the base for any edits, then will rename the pck file when outputting twaret.txt to avoid conflicts (x3 prioritizes pck files over unpacked txt or xml files). The plugin manager might do the same thing in reverse, loading the customizer txt file for editing, but I haven't tested that yet.
Edit:
The plugin manager does not capture changes in the twaret.txt file when generating its own, so it will need to be run first, with the customizer being run after the plugin manager is closed. It appears user edits to twaret.txt in general are ignored by the plugin manager.
3.4.3 source code is uploaded, and changes handling of path problems a little bit: it will now raise a hard exception if the x3/addon path looks wrong instead of attempting to still run (with a command line option to override this behavior, if desired). The error messages should show what paths are being used for the base x3 and addon folders, so hopefully that teases out whatever problem you are having.
Are you sure you aren't using something like:
path_to_x3_folder = r'G:\X3xrm\addon' ?
Because that will make the tool think the path is to the base x3 folder, and 'addon' will get appended again. (Perhaps I will add a check for addon\addon to catch this.)
For the plugin manage integration, the plugin manager should run first, though it may work in either order. The customizer will see the twaret.pck file and use it as the base for any edits, then will rename the pck file when outputting twaret.txt to avoid conflicts (x3 prioritizes pck files over unpacked txt or xml files). The plugin manager might do the same thing in reverse, loading the customizer txt file for editing, but I haven't tested that yet.
Edit:
The plugin manager does not capture changes in the twaret.txt file when generating its own, so it will need to be run first, with the customizer being run after the plugin manager is closed. It appears user edits to twaret.txt in general are ignored by the plugin manager.
-
GageDragon
- Posts: 59
- Joined: Tue, 26. Apr 16, 01:21
Yes path_to_x3_folder = r'G:\X3xrm\addon' was exactly what I was trying to use with authors_transform because that worked with example. That explains the problem at least
Good to know about plugin manager then thanks for the fixes.
I would prefer compiled but there is no need to release the smaller fixes imo.
Good to know about plugin manager then thanks for the fixes.
I would prefer compiled but there is no need to release the smaller fixes imo.
Updated to 3.5.2.
The big addition deals with excess spaceflies generated accidentally by some mods, which accumulate over time and slow down the game. Eg. 85% slowdown of Seta after 10 game days in my own case. Mods affected include Improved Races 2.0 v1.08, Salvage Command Software v4.11, possibly Gazz' Missile Defense Mk2, and likely others.
Kill_Spaceflies will clean out the accumulation by editing the KC to swap the "is disabled" script command (only ever used on spaceflies) to instead kill them after a 10 second delay. (Without this delay, some other X3 engine problems crop up when the mods try to access the dead spacefly.) This uses some hardcoded addresses and so it does not work on LU currently, but that support could be added easily if wanted.
Prevent_Accidental_Spacefly_Swarms will edit the KC such that the "create ship" command no longer performs an undocumented spawn of a spacefly swarm. This will cause it to behave the way the affected mods were expecting, putting a stop to spacefly accumulation. This one does work on LU, though I don't have enough experience with LU to know if spaceflies are a problem in any of those mods.
You can check for spacefly accumulation in the script editor, checking how many of the spacefly scripts (idle or follow) are active. It should generally be 0 up to a couple dozen or so (depending on sector asteroids); problematic mods build up hundreds or thousands.
Some non-spacefly changes are included as well.
Remove_Factory_Build_Cutscene will edit the time delays to 0 for the cutscene played when placing a factory. Future refinements may allow a user adjustable delay (eg. make it take 25% as long).
Keep_TLs_Hired_When_Empty will stop a hired TL from wondering off when you drop the last factory in its hold.
_Benchmark_Gate_Traversal_Time is a toy transform which removes the inter-sector warp loading delay in games with large complexes by cutting out a low level function call. Side effects are unexplored, so it is not part of the main documentation, and running it requires a full path: Transforms.T_Obj_Code._Benchmark_Gate_Traversal_Time().
Other than transforms, this update converts the python code fully into a multi-layer package, making it generally easier to work with and matching the style I use elsewhere in my larger programs. A by-product is that user input scripts need to swap from "from Transforms import *" to "from X3_Customizer import *".
The documentation generator now supports a BB code format, and the top post in this thread has been updated to be somewhat pleasant.
Edit:
I thought XRM had a spacefly problem, but was mistaken and the problem was in SCS.
The big addition deals with excess spaceflies generated accidentally by some mods, which accumulate over time and slow down the game. Eg. 85% slowdown of Seta after 10 game days in my own case. Mods affected include Improved Races 2.0 v1.08, Salvage Command Software v4.11, possibly Gazz' Missile Defense Mk2, and likely others.
Kill_Spaceflies will clean out the accumulation by editing the KC to swap the "is disabled" script command (only ever used on spaceflies) to instead kill them after a 10 second delay. (Without this delay, some other X3 engine problems crop up when the mods try to access the dead spacefly.) This uses some hardcoded addresses and so it does not work on LU currently, but that support could be added easily if wanted.
Prevent_Accidental_Spacefly_Swarms will edit the KC such that the "create ship" command no longer performs an undocumented spawn of a spacefly swarm. This will cause it to behave the way the affected mods were expecting, putting a stop to spacefly accumulation. This one does work on LU, though I don't have enough experience with LU to know if spaceflies are a problem in any of those mods.
You can check for spacefly accumulation in the script editor, checking how many of the spacefly scripts (idle or follow) are active. It should generally be 0 up to a couple dozen or so (depending on sector asteroids); problematic mods build up hundreds or thousands.
Some non-spacefly changes are included as well.
Remove_Factory_Build_Cutscene will edit the time delays to 0 for the cutscene played when placing a factory. Future refinements may allow a user adjustable delay (eg. make it take 25% as long).
Keep_TLs_Hired_When_Empty will stop a hired TL from wondering off when you drop the last factory in its hold.
_Benchmark_Gate_Traversal_Time is a toy transform which removes the inter-sector warp loading delay in games with large complexes by cutting out a low level function call. Side effects are unexplored, so it is not part of the main documentation, and running it requires a full path: Transforms.T_Obj_Code._Benchmark_Gate_Traversal_Time().
Other than transforms, this update converts the python code fully into a multi-layer package, making it generally easier to work with and matching the style I use elsewhere in my larger programs. A by-product is that user input scripts need to swap from "from Transforms import *" to "from X3_Customizer import *".
The documentation generator now supports a BB code format, and the top post in this thread has been updated to be somewhat pleasant.
Edit:
I thought XRM had a spacefly problem, but was mistaken and the problem was in SCS.
-
Aldebaran_Prime
- Posts: 1386
- Joined: Sat, 20. Feb 10, 17:47
I just tried version 3.05.1 to get running and to try your new Kill_Spaceflies() module.
I never used Python - so it's my first try.
I extracted your archive X3_Customizer_v3.05.1.zip to "C:\Users\Harald\Desktop\Neuer Ordner"
Then, I created a new transform-configuration "Spacefly_transform" in folder "input scripts" with the following content:
Then I opened a cmd window and changed the directory into the "...Neuer ordner" where I extracted your archive. There I tried to start the Transformation, but get a "Obj_Patch_Exception":
I never used Python - so it's my first try.
I extracted your archive X3_Customizer_v3.05.1.zip to "C:\Users\Harald\Desktop\Neuer Ordner"
Then, I created a new transform-configuration "Spacefly_transform" in folder "input scripts" with the following content:
Code: Select all
# Import all transform functions.
from X3_Customizer import *
Set_Path(
# Set the path to the X3 installation folder.
path_to_x3_folder = r'C:\Users\Harald\Desktop\AP QA',
)
# Kill spaceflies spawned before Prevent_Accidental_Spacefly_Swarms
# was added, lagging the game.
Kill_Spaceflies()Do I need to extract any obj.files from a .cat/dat before I start your converter?C:\Users\Harald\Desktop\Neuer Ordner>Launch_X3_Customizer.bat input_scripts\Spac
efly_transform.py
Attempting to run input_scripts\Spacefly_transform.py
Skipped Kill_Spaceflies due to a Obj_Patch_Exception exception.
Hm, do you know if you are running any mods which change x3story.obj, or perhaps have an obj file from before AP 3.3?
That exception is thrown when a search is made on the obj file for a reference code section (basically a series of bytes); if that section is not found, or an unexpected number of such sections is found, the tool throws an error and skips the transform.
Different versions of the obj file will change around internal addresses for function calls. Since I swap out some function calls for the spacefly killing, the transform will only work on the version of x3story.obj I made it for (vanilla AP 3.3). Creating versions for other obj files wouldn't be hard, as long as I can get a copy the file to pick out the different addresses.
That exception is thrown when a search is made on the obj file for a reference code section (basically a series of bytes); if that section is not found, or an unexpected number of such sections is found, the tool throws an error and skips the transform.
Different versions of the obj file will change around internal addresses for function calls. Since I swap out some function calls for the spacefly killing, the transform will only work on the version of x3story.obj I made it for (vanilla AP 3.3). Creating versions for other obj files wouldn't be hard, as long as I can get a copy the file to pick out the different addresses.
-
Aldebaran_Prime
- Posts: 1386
- Joined: Sat, 20. Feb 10, 17:47
yes, this was it - I stiil used a 04.cat/dat from X3AP 3.2, not from 3.3. Now I got "Successfully ran Kill_Spaceflies" and in the game-folder the directory "L" was created with a x3story.obj in it.
When I load my savegame, open the script-editor and the script-task statistics - it's great to see how the number of running spacefly scripts goes down from 5000 to 0 in some seconds!
THANK YOU VERY MUCH!!!
I think it is worth to mention, that, after this clean-up a new save needs to be done, then close X3AP and delete the just generated x3story.obj from L, but otherwise you never will find any spacefly in game any more, because they destroy themselves immediately as long your modified x3story.obj is used.
So after this cleanup the player either needs to generate a new modified version of the game with your modified create_ship command or with the bugfix for the installed mod-scrips like IR2.0
As you reported in the other thread, the increase of FPS in normal play ist not so big - I just got 2-3 FPS after deletion and reload. This is much less, when I deleted 5000 Spaceflys which are not in sector NULL, but in "normal sectors"
But the gameplay in sinza and during battles should have a benifit of this deletion.
When I load my savegame, open the script-editor and the script-task statistics - it's great to see how the number of running spacefly scripts goes down from 5000 to 0 in some seconds!
THANK YOU VERY MUCH!!!
I think it is worth to mention, that, after this clean-up a new save needs to be done, then close X3AP and delete the just generated x3story.obj from L, but otherwise you never will find any spacefly in game any more, because they destroy themselves immediately as long your modified x3story.obj is used.
So after this cleanup the player either needs to generate a new modified version of the game with your modified create_ship command or with the bugfix for the installed mod-scrips like IR2.0
As you reported in the other thread, the increase of FPS in normal play ist not so big - I just got 2-3 FPS after deletion and reload. This is much less, when I deleted 5000 Spaceflys which are not in sector NULL, but in "normal sectors"
But the gameplay in sinza and during battles should have a benifit of this deletion.
Good to hear it worked out.
You can also remove the changed file by rerunning the tool with Kill_Spaceflies removed (or commented out), or run the tool with the -clean flag. Each run creates a log of which files were created (with hashes) or renamed, so that a following run can clean out prior changes.
You can also remove the changed file by rerunning the tool with Kill_Spaceflies removed (or commented out), or run the tool with the -clean flag. Each run creates a log of which files were created (with hashes) or renamed, so that a following run can clean out prior changes.
Updated to 3.6, now with some support for TC. For the most part I just needed to tweak paths in two spots and the rest worked itself out (feels good when that happens), though I took some time to do light testing and update a couple transforms that weren't working.
In the Set_Path function, add "enable_TC_mode = True" as a parameter. Kill_Spaceflies should work for TC 3.4.
In the Set_Path function, add "enable_TC_mode = True" as a parameter. Kill_Spaceflies should work for TC 3.4.
![[ external image ]](http://download.urdenbach.de/TOTT-Banner1.jpg){kind=link}
I included the X3 Customizer and its features in my modding package.
https://forum.egosoft.com/viewtopic.php?t=399184
Its just a package with installer, not a mod by itself.
The X3 Customizer is one of the key parts and really shines - i think its the first real use like that.
The bat installer allows also a pretty fast test and run, in fact i was able to run like 6 different instances and sets of the types only with that batches.
You are also free of course to use what you want, like rip off the bat files for your own test package - its really useful.
I used the customizer beside it fixes also to generate for xrm and lucikes stuff the "make for ships missing variants". I did that for capital ships, extending XRM even further and its alot fun.
I did not included the Customizer bins or python files but you can find in
LxRM\_Tools\_X3_Customizer (modding)\
the transform file i used.
https://forum.egosoft.com/viewtopic.php?t=399184
Its just a package with installer, not a mod by itself.
The X3 Customizer is one of the key parts and really shines - i think its the first real use like that.
The bat installer allows also a pretty fast test and run, in fact i was able to run like 6 different instances and sets of the types only with that batches.
You are also free of course to use what you want, like rip off the bat files for your own test package - its really useful.
I used the customizer beside it fixes also to generate for xrm and lucikes stuff the "make for ships missing variants". I did that for capital ships, extending XRM even further and its alot fun.
I did not included the Customizer bins or python files but you can find in
LxRM\_Tools\_X3_Customizer (modding)\
the transform file i used.
Awesome; it is wonderful to see my tool getting used like this, though a bit scary if something breaks.
In other news, I have updated it to 3.7 with a couple minor changes.
Disable_Combat_Music now supports LU. This ended up only needing to wildcard the offset of a member variable.
For the brave, the experimental _Benchmark_Gate_Traversal_Time has been renamed to Remove_Complex_Related_Sector_Switch_Delay. This should only be used by people plagued by really long sector loading times due to having built large complexes in their game. It cuts out the call to SA_CleanUpObjects, and I really don't know what side effects might crop up. This transform makes me feel like that quack doctor from the Simpsons, pulling out someone's organ because he doesn't know what it does and it was in the way.
In other news, I have updated it to 3.7 with a couple minor changes.
Disable_Combat_Music now supports LU. This ended up only needing to wildcard the offset of a member variable.
For the brave, the experimental _Benchmark_Gate_Traversal_Time has been renamed to Remove_Complex_Related_Sector_Switch_Delay. This should only be used by people plagued by really long sector loading times due to having built large complexes in their game. It cuts out the call to SA_CleanUpObjects, and I really don't know what side effects might crop up. This transform makes me feel like that quack doctor from the Simpsons, pulling out someone's organ because he doesn't know what it does and it was in the way.
Updated to 3.8.
Non-script modified files will now be collected to an incrementally indexed cat/dat pair to reduce clutter. This behavior can be turned off using the no_catalog command line flag, or using "Settings.output_to_catalog = False" in the control script. If your game crashes after making this change, and was already open from before the change, a full game restart should clean up the problem. (It took me much longer than it should have to realize the source of my crashes during testing.)
The Add_Life_Support transform has been updated to now edit various mission director scripts which check for player ship life support. Those scripts have special checks for TP class ships, since the mission director does not support built-in ware checks; the locations I found will now include checks for the modified ship classes as well. Many of these changes are in plot scripts and are difficult to check, so there may still be issues (though things looked okay at a glance).
The Add_Ship_Variants transform will now update the Bounce mod wall file, if found, with the new variants. Note that this does not ensure the original wall file was up to date for the original TShips before variants were added.
Edit:
Updated to 3.9, adding Disable_Docking_Music.
Non-script modified files will now be collected to an incrementally indexed cat/dat pair to reduce clutter. This behavior can be turned off using the no_catalog command line flag, or using "Settings.output_to_catalog = False" in the control script. If your game crashes after making this change, and was already open from before the change, a full game restart should clean up the problem. (It took me much longer than it should have to realize the source of my crashes during testing.)
The Add_Life_Support transform has been updated to now edit various mission director scripts which check for player ship life support. Those scripts have special checks for TP class ships, since the mission director does not support built-in ware checks; the locations I found will now include checks for the modified ship classes as well. Many of these changes are in plot scripts and are difficult to check, so there may still be issues (though things looked okay at a glance).
The Add_Ship_Variants transform will now update the Bounce mod wall file, if found, with the new variants. Note that this does not ensure the original wall file was up to date for the original TShips before variants were added.
Edit:
Updated to 3.9, adding Disable_Docking_Music.
I hoped you like it - in fact the X3C gave me one of the main reason to make LxXRM. Its to good not to offer it connected with a setup.SirNukes wrote:Awesome; it is wonderful to see my tool getting used like this, though a bit scary if something breaks.
Btw, i had one idea which is perhaps something you find interesting too.
There is the problem with wrecks, ships and docks. Some scripts/mods can add alot, clustering the area.
The usual fix is to clean the wreck type files, but i think there is a better possible solution.
The npc bailing script comes with a timer for all its bailed ships, make them self destroying after some time. If that would be possible for wrecks, it would be the golden solution.
Is the wreck creation is hard coded or part of the scripts in the obj files?
In the last case it should be easy to add such timer script to wrecks.