Cat/Dat vs types installation of mods. General discussion. (from another thread)

[apricotslice]

The following is a quote of 2 sides of an argument in another thread.

Essentially the question is :

Should we as a modding community have a standard about how we distribute mods. In particular, should mods always be in cat/dat form ?

I've always considered that mods should have their own cat/dat. The exception being small changes such as the tweaks to hq.xml, and MD changes.

It was a bit of a shock to find that some mods go directly into the types folder.

I wondered what everyone thought about this practice. Pros and cons.

Can we come to a concensus on standards ?

Edit : Please note that Roger L.S. Griffiths has taken this as a personal attack, when for me this is nothing personal. As an issue of standards and policy as a community, I would be bringing this up for discussion no matter who brought it to my attention. It just happened to be Roger in this case. I am sorry if it has cause offense, but I deem it subject of sufficient importance to be discussed by everyone in its own thread, especially as it was highjacking the thread it came up in.

It is even simpler than that... AWRM does not use CAT/DAT files and instead installs direct into the types folder.

I have to say I disagree with that practice on a fundamental level.

Its ok for individuals who are modding for themselves, but not for a distributed mod.

The way people stack up mods these days, installing directly into types could be causing all sorts of issues that are attributed to someone elses' mod.

Matter of opinion... and on the whole if mods are changing the same files the end-user is borked anyway regardless of how the files are changed. Direct installation into the types folder is one easy way of detecting such conflicts at time of installation. Fake patches are fine to a point, but it is too easy to muck up the game. At least with extracted installations it is easy to tell what has changed. With simple weapon/ship mods which just change the Types files it is easier to maintain and easier to track down issues too.

Now when someone brings me a problem with my mods, I have to ask them "do you have awrm installed ?" If so, go talk to Roger first. Not acceptable for me to either have to remember to do that, or have to pass them on to you first.

Why???? Installing direct in the Types folder is only the same as using the highest CAT/DAT number except it is far less complicated for installation.

In any case, if a user is using multiple mods it is ultimatly their responsibility to do a round robin of the mod authors (or at least the mod support threads) to check for compatability issues. So there is absolutly no reason to criticise how ANY mod author chooses to package THEIR works, since YOU are only responsible for YOUR works and the compatability concerns there in.

Those concerns should be merely limited to maintaining a list of what files you change, what compatability patches you choose to support, and what mods you know are compatable with your works.

Makes me wonder if a few odd problems that were not solved recently are exactly this scenario.

With all my works (AWRM, CPLS, PMTS, Frontier Wars) included in the startup scripts are welcome messages reporting versions, detection/reporting of upgrades, plus some basic form of in-game information about the works via the Encyclopedia. If more scripts/mods did something similar rather than sitting silently in the background leaving the user to wonder if the mod/script is working or not then perhaps debugging issues would be ALOT easier.

I can see your points, but my approach if your mod is considered to be necessarily always the last one, is to enforce it being loaded as a mod, and never as a false patch. "Its loaded as 15". "Load it as the mod as instructed and come back if the problem still exists".

The whole unpacking into types idea is fine for debugging a mod, but what about the users who have no idea what they are doing during installation and when the unpack asks if they should overwrite existing files, what do they say ? Yes and hope it works ? No and hope it works ? Huh? and cry to the last mod author that it generated an error message on install ?

Can of worms.

Issues either way, but at least the cat/dat system keeps each mod completely separate.

[Litcube]

Cats encapsulate related modifications into manageable parts for exactly the same function.

If every mod used loose files without cats, we'd have a massive debugging headache with the average user on this forum (which I suspect to be a drunk 7 year-old with reading disabilities). I'm already experiencing God's test with the users on this forum. Loose files? No thanks.

[apricotslice]

Please note my edit in the OP.

[Sam L.R. Griffiths]

Merging of multiple mods is a pain in the rear at the best of times, but to my mind installing in the clear highlights any potential incompatabilities to the end-user who then needs to look at what mods they have installed and then make a choice of either:-

1. approach all the authors as a collective team and treat the resolution of compatability issues as a community effort
2. Approach one of the authors who is willing to shoulder the compatability burdens of their mod
3. Give up on using the mods together

In either case CAT/DAT or in the clear, the issues are still going to be there and it does not change the nature of the resolution of issues: The conflicting files still need to be adjusted (possibly with other changes as well) in order to resolve the issues.

Taking my AWRM mod as an example, I clearly state in my support thread what files constitute my mod and highlight those vanilla ones that I have changed. Now I do not have a problem with anyone else deploying mods in the clear so to speak, but in cases where there is LOTS of custom content it does make sense to package as a CAT/DAT pair, but in cases where there is only a couple of files it makes more sense to install in the clear as there is little benefit to packaging in a CAT/DAT and actually probably makes merging of simple content alot less hassle.

[apricotslice]

In either case CAT/DAT or in the clear, the issues are still going to be there and it does not change the nature of the resolution of issues: The conflicting files still need to be adjusted (possibly with other changes as well) in order to resolve the issues.

I dont disagree.

But if 2 mods are unpacked into types that conflict, and the user does not abort the unpack process, then you end up with a giant mess, and no way of easily extracting one of the mods. Most users would end up deleting and reinstalling their game, rather than just deleting one set of cat/dats.

For most users, merging of mods is not an issue because its simply beyond them to even contemplate it.

The most they can do is delete a mod that they decide is the conflicting one they can do without. They cant do that if mods are mixed up in the types folder.

Nor at a later date can they delete a mod when they want to start a new game differently. Again because they cannot differentiate as to what files belong to what mod. Listing them on your thread is fine, but how many people are actually going to go back there and look before they try to delete a mod ? And why should they when deleting the cat/dat is much much easier to do, with no consequences of missing a file.

[Sam L.R. Griffiths]

Edit : Please note that Roger L.S. Griffiths has taken this as a personal attack, when for me this is nothing personal. As an issue of standards and policy as a community, I would be bringing this up for discussion no matter who brought it to my attention. It just happened to be Roger in this case. I am sorry if it has cause offense, but I deem it subject of sufficient importance to be discussed by everyone in its own thread, especially as it was highjacking the thread it came up in.

With regards to this, it was more the way the issue was raised and initially discused rather than the topic in question. My views on this issue are pragmatic and ambivelant but ultimatly the underlying issue of resolving mod conflicts needs to be addressed. It is just a shame that such discussions can often be seen as mud slinging contests due to linguistic nuances in the majority of cases.

On the whole, CAT/DAT/ZIP is not the ideal distribution format as it often leaves the end-user with issues of the order of installation and as I mention in my post above, the issues of conflicts always need to be identified and resolved.

This is easier for an end-user to identify during installation if the changed vanilla files are kept in the clear, and in some cases (such as XML MD files) it makes it easier for them to identify and execute simple merging scenarios (such as additional starts). If such files are hidden away in CAT/DAT pairs identification of such conflicts becomes more esoteric and obscure (after all how many users actually read manuals/posts to find out such details - if they are asked to overwrite a file then they are more likely to ask questions before a problem is potentially encountered). In addition, if files are kept in the clear, it is very likely that installation BAT/EXE files could be used to flag conflicts during installation and back out partial installations.

I thought that SPK files were supposed to help with identifying such issues, but I have personally found them to be of less use since the SPK manager was revised for X3:TC. In X3:R, it worked a charm, but on the whole I have erred towards manual installations because I have found v1.20 too buggy (perhaps v1.30 is better but I can not comment).

In either case, the key point is that it is EVERY modders responsibility to identify and highlight potential compatability issues to their user base and how a mod is distributed/installed is to my mind a moot point.

[Nafensoriel]

I like patches and model files stashed away in nice cat/dat format.. beyond that they just get in the way of my tinkerlust. While yes it is a pain in the royal arse to merge things together especially when trying to mix xtc items, x3tc ship packs, my own models, etc etc the process is NOT that hard. Just time consuming. There are plenty of resources on this forum to use if you just use that lovely little search function.

I don't think it really matters what method a mod uses because in the end it is just user control. If the user is not checking what files the mod changes and the user is not educating themselves in HOW the files work.. then there will be problems with EITHER method. Of course then there is always the rule of backup.. If you change it.. BACK IT UP FIRST. Copy/paste the entire bloody x3tc directory if you have to.

Side note.. might make things more mod friendly to newbies if there were conflict detectors. Isn't always a good thing when they find out something is broken because an invisible ship flies by.

[Sam L.R. Griffiths]

But if 2 mods are unpacked into types that conflict, and the user does not abort the unpack process, then you end up with a giant mess, and no way of easily extracting one of the mods. Most users would end up deleting and reinstalling their game, rather than just deleting one set of cat/dats.

For most users, merging of mods is not an issue because its simply beyond them to even contemplate it.

The most they can do is delete a mod that they decide is the conflicting one they can do without. They cant do that if mods are mixed up in the types folder.

Nor at a later date can they delete a mod when they want to start a new game differently. Again because they cannot differentiate as to what files belong to what mod. Listing them on your thread is fine, but how many people are actually going to go back there and look before they try to delete a mod ? And why should they when deleting the cat/dat is much much easier to do, with no consequences of missing a file.

Ok, this reasoning all hinges on users extracting mods direct into the game directory.

Application of CAT/DAT files is equally flawed as the scripts are typically kept in the clear. I do not even know if scripts will run from the CAT/DAT domain but as Egosoft do not put them in CAT/DAT files, we must presume that they will not. Thus residue files from CAT/DAT + Script installations is probably inevitable.

The simple answer to the partial installation argument is that it is VERY easy to write a BAT/EXE file that will install and un-install files and even identify conflicts and recover from them (by automatically backing out the partial installations and creating uninstall configurations for fully installed mods).

Of course, all of this debate about installation mechanisms would disappear if the community as a whole fealt that they could trust the implementation of the SPK Mod Manager.

[apricotslice]

I dont use SPK for mods. I want direct control over where everything is put, and I had an unfortunate experience with an early version that put me right off it. I prefer my mods be run as a mod, not as a patch.

Scripts wont run out of a cat/dat.

Removal of scripts is usually easy. Delete the setup scripts, delete the AL scripts. And if the script writer uses a common name in all scripts, delete all scripts with the common name in them.

As long as the setup and AL starting scripts are deleted, the rest pose no problems.

[Sam L.R. Griffiths]

I dont use SPK for mods. I want direct control over where everything is put, and I had an unfortunate experience with an early version that put me right off it. I prefer my mods be run as a mod, not as a patch.

Scripts wont run out of a cat/dat.

Removal of scripts is usually easy. Delete the setup scripts, delete the AL scripts. And if the script writer uses a common name in all scripts, delete all scripts with the common name in them.

As long as the setup and AL starting scripts are deleted, the rest pose no problems.

But to follow your argument against distributing files in the clear, there is still a risk of residual files (or updated vanilla script files) being left behind from a manual installation.

The problem with MODs that run as MODs and not fake patches is that even if they are compatable the end user can only run one at a time. Thus the underlying issue of installing multiple mods and the risk of residual files (scripts or files in the clear) being left behind does not go away.

WRT Mods being installed as fake numeric CAT/DAT patches, if this is left to manual installation then it becomes problematic remembering what CAT/DAT fake patch correlates to what mod.

As I have some time on my hands this week, I will look into producing a mod/script install/uninstall helper program and post it here when I have a releasable version that I am happy with (ETC Sa-19-Mar-2011 but hopefully sooner).

[apricotslice]

then it becomes problematic remembering what CAT/DAT fake patch correlates to what mod.

Easy. 15.cat 15.dat 15-no-fog-mod.txt You just open notepad and save a file called whatever name starting with the number. Doesnt even need anything in it.

As I have some time on my hands this week, I will look into producing a mod/script install/uninstall helper program and post it here when I have a releasable version that I am happy with (ETC Sa-19-Mar-2011 but hopefully sooner).

Great idea, especially if it can read the contents of a zip and rar, and store it so that each mod is viewable, conflicts identified, and uninstall is one click. It could then work with both cat/dats and types.

An alternate version or feature would be to be able to look at all cat/dats, mod and types/director/etc, and report on conflicts.

An installation feature could be the saving of vanilla scripts, so that they can be auto-replaced if uninstalled.

Thats worth its own thread to see what people would prefer to see it do.

All respect to Cycrow, but the whole SPK thing doesnt work well enough for my liking (I understand why), and never was suited to mods (imo). I use it for script packs, but wont use it for mods. I release all mods as zips.

[Observe]

This topic touches on part of the reason we use an Installer program for Transcend mod.

The Installer keeps track of existing game files, and renames any we overwrite. It also inspects standard directories (types, director, maps, mov, etc) for conflicts. Then the Installer can switch back and forth depending on selection.

We use our own mod cat/dat selectable at game startup - rather than false patch. You have already outlined the problems with false patches.

Our method is not perfect, but probably as good as we can do within reason.

That's how I suggest when possible.

[JrK]

2 cents:
.SPK - rather not, but with plugin manager you can easily unpack and then install.

.cat/dat - yes, if a clear install order is given, makes it easier to have multiple files installed BUT ->

loose mod files - only if it is a file that would go in the highest cat/dat regardless of circumstances, in that case it is a good idea and doesn't need cat/dat imo.

Note that the unknowing user wouldn't even realise that those files were actually equivalent to mods.

[Sam L.R. Griffiths]

then it becomes problematic remembering what CAT/DAT fake patch correlates to what mod.

Easy. 15.cat 15.dat 15-no-fog-mod.txt You just open notepad and save a file called whatever name starting with the number. Doesnt even need anything in it.

That requires end-user intervention/fore-thought which can not be taken for granted. An appropriate alternate would be for a read-me to be included with the CAT/DAT pair with the same name as the CAT/DAT files which you expect to be installed alongside the CAT/DAT and renamed at the same time.

As I have some time on my hands this week, I will look into producing a mod/script install/uninstall helper program and post it here when I have a releasable version that I am happy with (ETC Sa-19-Mar-2011 but hopefully sooner).

Great idea, especially if it can read the contents of a zip and rar, and store it so that each mod is viewable, conflicts identified, and uninstall is one click. It could then work with both cat/dats and types.

An alternate version or feature would be to be able to look at all cat/dats, mod and types/director/etc, and report on conflicts.

An installation feature could be the saving of vanilla scripts, so that they can be auto-replaced if uninstalled.

Thats worth its own thread to see what people would prefer to see it do.

My initial offering is likely to be just a file installer/uninstaller with automatic backup of changed files but I will be looking at making it able to read from archives directly via use of 7-zip. When I have got my initial works off the ground I will create a thread for it.

All respect to Cycrow, but the whole SPK thing doesnt work well enough for my liking (I understand why), and never was suited to mods (imo). I use it for script packs, but wont use it for mods. I release all mods as zips.

Agreed wrt v1.20 and X3:TC, I should highlight that the SPK manager is still being developed. But it can and did work well for mods with the legcy X3:R version (c/f XTM for X3:R). I would like to emphasise that my helper tool is not intended to be in direct competition to the SPK manager but rather act as supplemental/temporary alternate option.

[Sam L.R. Griffiths]

This topic touches on part of the reason we use an Installer program for Transcend mod.

The Installer keeps track of existing game files, and renames any we overwrite. It also inspects standard directories (types, director, maps, mov, etc) for conflicts. Then the Installer can switch back and forth depending on selection.

We use our own mod cat/dat selectable at game startup - rather than false patch. You have already outlined the problems with false patches.

Our method is not perfect, but probably as good as we can do within reason.

That's how I suggest when possible.

Yeah, Installer development can be a pain with lots of files (I know that from personal professional experience). On this note, I wish to point out that there is a great free and open source cross-platform Installer generator called InstallJammer.

[Sam L.R. Griffiths]

...
Note that the unknowing user wouldn't even realise that those files were actually equivalent to mods.

(EDIT: Agreed) An average end-user probably does not really care how the mod/script is distributed, all they really want is to play the game with their selected mod content.

[JrK]

Kinda my point. For that purpose sometimes the loose files will work better. I'm, for instance, thinking of the EES and SRM jobs files.

[Observe]

Here are advantages and disadvantages of existing options (in no particular order):

- False patch
- Cycrow SPK
- Selectable mod (in mods folder)
- External "loose" files
- Custom installer (keeps track of overwrites)

If you want to distribute a mod, you have to use one or a combination of the above. Each has its limitations. I already described which method we use for Transcend.

False patch: At this stage of game maturity (no more patches?), false patch is not a bad method. It suffers from overwrite by others using false patch, and also from those using separate mod.

SPK: Cycrow designed this solution to address the topic being discussed. Personally, I never use this method because I'm always trying to push the game to the limit by trying new things, and I need to work directly with every detail in the mod.

Selectable mod: This avoids the problem of false patches being overwritten by official patches or by other mods using the same method. The problem is only one mod can be selected - not so good for small add-on's, but good for "major" mods.

External ("loose") files: Good for personal mods, or during development. Probably the worst solution for distributed mods, because external files take priority over those contained within cat/dat. In other words, external files can easily "clobber" those contained within cat/dat, and these files can get overwritten by other mods using this same method. The result can be a "basket-case rats nest".

Custom Installer: Best method, but not everyone can or has time to develop a custom installer for their mod.

So, there you have it. Pick your poison!

[Sam L.R. Griffiths]

External ("loose") files: Good for personal mods, or during development. Probably the worst solution for distributed mods, because external files take priority over those contained within cat/dat. In other words, external files can easily "clobber" those contained within cat/dat, and these files can get overwritten by other mods using this same method. The result can be a "basket-case rats nest".

True, to a degree, but at the end of the day it should not matter either way because the end solution for any given conflicts will be either (a) Re-install the over-ridden mod thus reverting the affected files OR (b) Application of a merge/compatability patch OR (c) clean install without one or other of the mods.

The latter option is easy to beat by keeping a backup of the Vanilla game state... in addition there are some mods/scripts such as EES, CODEA, and MEFOS that overwrite vanilla script files and thus are vulnerable to the "basket-case rats nest" regardless of how other mod material is distributed.

On the subject of clobbering it is no different than the case where the mod-author states that the CAT/DAT file must be the highest numerical value in the sequence. But, where multiple notionally compatable mods state this as a requirement it generates confusion for the user.

In any case, ultimately the main issue is of resolution of mod compatability (regardless of distributed form) and it is not the mod author that necessarily has to sift through the data files on the end-user's machine (even the end-user probably will not be doing that). The solution will probably end up being developed off-line and then distributed as a compatability patch in the most appropriate form (c/f my CMOD compatability patch for AWRM where I changed certain features of AWRM kit to better suit the CMOD balance).

[Saetan]

I think, best would be if EGOSOFT could be convinced to re-work their start dialog.

."/mods" then would be the only place then, where any mod ... any script ... and any mission-director-files could be placed and also they could keep their own names.

The dialog just would be to be re-written, so that more than one .cat/.dat files can be selected AND get their own priority-setting, which would do the same thing as the actual false-patch placement.
As EGOSOFT should have existing routines to see the contents in the .cat/.dat files already, it should even be possible to give a warning, if mods consist of the same files, and therefor would potentially conflict.
E.g., this could be done by "virtually" unpacking any .cat/.dat inside the mods-directory from priority "lowest" to "highest", automatically overwriting old files and repack them in a new "virtual" .cat/.dat which would be loaded.

As scripts and mission-director files can also be distributed as .cat/.dat files, even them could be easily installed and removed if don't liked anymore, as long as a (in-game) uninstall-option is kept in mind and used if necessary.

With such a re-done dialog, even the saving of different profiles ... that means different activation/deactivation-settings of different mods/scripts/md-files could be implemented, so no multiple installs would be necessary anymore and there's still an easy way to play vanilla.