Weird issue trying to load mods

[strask412]

Edit: Since this thread is now linked from the sticky, the actual answers begin in the reply below or click on this: viewtopic.php?f=181&t=417564#p4873566 .

Original question starts here:

Hi, I'm having an embarrassing (probably) issue attempting to get some mods loaded, I had the same set of mods (earlier versions of some) working back when I was playing X4 under Win7, but now I can't get them to load under linux. Here is what I tried first:

Code: Select all

$ cd
$ cd GOG\ Games/X4\ Foundations/game
$ ls
01.cat      02.cat      03.cat      04.cat      05.cat      06.cat      07.cat      08.cat      09.cat      alsoft.conf                lang.dat      support        testfailed  version.dat
01.dat      02.dat      03.dat      04.dat      05.dat      06.dat      07.dat      08.dat      09.dat      extensions                 lib           t              test_txc    videos
01_sig.cat  02_sig.cat  03_sig.cat  04_sig.cat  05_sig.cat  06_sig.cat  07_sig.cat  08_sig.cat  09_sig.cat  goggame-1588366064.hashdb  Licences.txt  testandlaunch  test_X4     X4
01_sig.dat  02_sig.dat  03_sig.dat  04_sig.dat  05_sig.dat  06_sig.dat  07_sig.dat  08_sig.dat  09_sig.dat  goggame-1588366064.info    mousecursors  testcommon     ui
$ ls -l extensions
total 28
drwxrwxr-x 3 trask trask 4096 Jun 20 22:43 'Average Prices Suffixes'
drwxrwxr-x 3 trask trask 4096 Jun 20 22:43  MassMoveMarines
drwxrwxr-x 3 trask trask 4096 Jun 20 22:43  MassMoveServicecrew
drwxrwxr-x 3 trask trask 4096 Jun 20 22:43  mej_disable_encounters
drwxrwxr-x 2 trask trask 4096 Jun 20 22:43  No_Dark_Ambient
drwxrwxr-x 3 trask trask 4096 Jun 20 22:43  RareModpartsSold
drwxrwxr-x 4 trask trask 4096 Jun 20 22:43  ShipTradersSellPaintMods
$ 

When I start X4, only the "mej_disable_encounters" mod shows up on the extensions option from the main menu. So I tried moving the entire extensions folder like this:

Code: Select all

$ mv extensions ~/.config/EgoSoft/X4/
$ ls ~/.config/EgoSoft/X4/extensions/
'Average Prices Suffixes'   MassMoveMarines   MassMoveServicecrew   mej_disable_encounters   No_Dark_Ambient   RareModpartsSold   ShipTradersSellPaintMods
$ ls ~/.config/EgoSoft/X4
auth.xml  config.xml  extensions  inputmap.xml  logos  pipelinecache.bin  progressbar.xml  save  uidata.xml
$ 

And that behaves exactly the same way -- when I go to the extensions option from the main menu, only the one mod shows up.

I examined the contents of each mod directory and they all seem to contain valid-looking contents.xml files, as well as other files appropriate to the nature of each mod. I checked the permissions of all involved files and directories, and everything is rwx to me and even r-x to world, just in case.

I am sure I'm missing something obvious (and likely well documented) but I just can't figure it out. If anyone has a suggestion for me, I'd appreciate it.

I modified the game start-up scripts to add the debug log option. Contents of the debug log look like this:

Code: Select all

Logfile started, time Fri Jun 21 17:37:29 2019
[General] 0.00 ======================================
[=ERROR=] 0.00 File I/O: Could not find file './etc\xml\catalog'
[General] 0.00 ======================================
[General] 0.00 ======================================
[General] 0.00 ======================================
[=ERROR=] 0.00 File I/O: Could not find file './etc\xml\catalog.sig'
[General] 0.00 ======================================
[=ERROR=] 0.00 FileIORequest::WaitForResult() Failed './etc\xml\catalog' with 4
[General] 0.00 ======================================
[General] 0.00 ======================================
[=ERROR=] 0.00 XLib::XMLInputOpenCallback(): Failed to open the file: etc/xml/catalog
[General] 0.00 ======================================
[General] 0.00 ======================================
[=ERROR=] 0.00 File I/O: Could not find file './extensions\mej_disable_encounters\content.xml.sig'
[General] 0.00 ======================================
[General] 0.00 'GeForce GTX 1070' NVidia-driver v418.56.0 (Vulkan 1.1.95) check=1

As you can see, only the one mod which is showing up has any messages printed about it. I don't know what's up with the could-not-find for etc/xml/catalog and such, but I see that in other people's log files who don't have my problem so I doubt it is related?

[steve_v]

Hi, I'm having an embarrassing (probably) issue attempting to get some mods loaded, I had the same set of mods (earlier versions of some) working back when I was playing X4 under Win7, but now I can't get them to load under linux.

The embarrassment is Egosoft's, not yours. The game can't deal with a case-sensitive filesystem properly, and nobody bothered to document it.

I am sure I'm missing something obvious (and likely well documented)

It's only obvious if you look at information for a different game. Seems nobody at ego thought it would be useful to sticky this in the X4 forum as well.

A quick and dirty recursive rename script might look something like this:

Code: Select all

#!/bin/bash
read -p "Rename files and directories to lower-case? (y/N)" x
if [ "$x" = "y" ]; then
    find . -type d | \
    while read oname; do
        nname=$(echo "${oname}" | tr [:upper:] [:lower:])
        mv -v "${oname}" "${nname}"
    done
    find . -type f | \
    while read oname; do
        nname=$(echo "${oname}" | tr [:upper:] [:lower:])
        mv -v "${oname}" "${nname}"
    done
fi

Or just remember to do it by hand whenever you install or update any mods.
Any mod that references it's own files from macros.xml will need the filenames fixed there too, but scripting that is left as an exercise for the reader.

As for mods that come packed (.cat/.dat pairs), I don't know since I don't run any. If they cause problems you might try unpacking them and renaming as above. The cat unpacker for rebirth works for X4.

[strask412]

Thank you so much for that information! And you went to the trouble of scripting it, which is going above and beyond.

We should probably have a duplicate of that post you linked stickied over here in the X4 forum, or at least linked from one of the existing stickies.

I should be able to play the way I want to now, so I am in your debt. Have a great day!

[steve_v]

Thank you so much for that information!

You are most welcome.

you went to the trouble of scripting it, which is going above and beyond.

No trouble, it's a pretty simple task and I keep snippets like that filed away for this kind of situation.
Incidentally, piping the output of find into a while loop like that is an easy way to avoid the issues you run into processing filenames with spaces and/or tabs in them, without messing with $IFS.

We should probably have a duplicate of that post you linked stickied over here in the X4 forum, or at least linked from one of the existing stickies.

Egosoft should fix their game engine so that it can load files properly on a case-sensitive filesystem, then we won't need all this horseshit. They knew about this problem with Rebirth, and that means it's been around for years. Why has it not been fixed?

Refusing to load files with upper-case in the names says to me that the file-loading code in X4 is making stupid assumptions, and assumptions like that will keep introducing bugs until they are removed. CamelCase is trendy at the moment and modders are going to keep doing it, because those filenames should load just fine if your game engine isn't full of windoze-isms.
It's only a matter of time until someone else, such as GOG or Egosoft themselves, ships a file that isn't lowercase in the Linux package and people start complaining. Fix it, fix it now. It will only create pain.

If you want more evidence of how infested with windoze-isms the GNU/Linux port still is, just look at that nasty mixing of '/' and '\' as directory delimiters in your log, it makes me cringe.

[strask412]

Incidentally, piping the output of find into a while loop like that is an easy way to avoid the issues you run into processing filenames with spaces and/or tabs in them, without messing with $IFS.

Heh, yeah. Shell quoting is a wonderful labyrinth of edge cases.

Egosoft should fix their game engine so that it can load files properly on a case-sensitive filesystem
...
Refusing to load files with upper-case in the names says to me that the file-loading code in X4 is making stupid assumptions, and assumptions like that will keep introducing bugs until they are removed.

That is true, it shouldn't have been an issue in the first place and it isn't correct in the current form. I do hope they fix it soon. Of course, EgoSoft's list of "things to fix" is probably pretty long, so I just try to stay happy with whatever they /do/ prioritize.

If you want more evidence of how infested with windoze-isms the GNU/Linux port still is, just look at that nasty mixing of '/' and '\' as directory delimiters in your log, it makes me cringe.

While it does look ugly, I have to give them a pass on that one. Obviously the strings for the filenames are being assembled from parts, and the parts were written by two different people or something, and most APIs now do the practical thing and just treat / and \ interchangeably in pathnames so I doubt that will result in bugs. It would be nicer if everything was consistent, and yes the code quality increase / bug risk reduction might be non-zero, but it would still be pretty small.

[steve_v]

Not my experience as at least for me patching MD files works flawlessly when naming my diff files exactly like the source files (keeping capitalisation) as one would expect it

In the GOG release all the source files are lower-case. Perhaps something is different with the steam release or the libraries it uses.
Out of interest, can you load mods in camelcase directories? How about with STEAM_RUNTIME=0?