Corrupt/very large (0.5Gb) save game files - fixed in 3.60 beta 1

[Mike Roberts]

My save games are growing very large very quickly to the point they fail to load.
File sizes were around 98Mb and growing slowly. Then grew to 140Mb then 250Mb and 500Mb on successive saves. I have around 10days progress, currently using smalltalk to upgrade skills for staff on capships I have taken over. I am using the standard game, no mods, current update (3.53). Is there some way to clean these files? I can't open them as they are too big! Mike

[CBJ]

Without knowing what's causing the problem it's hard to say. I know they're big, but can you upload one somewhere so that someone can take a look and find out what's causing them to grow?

[UniTrader]

hint: compress them before uploading - since Saves are uncompressed xml this reduces their Size dramatically (about 95% smaller for regular saves, could be even more in your case)

[Mike Roberts]

Uploaded to: https://dl.dropboxusercontent.com/u/682 ... cksave.zip

I went back to a save a couple before this started. I then had around 6 saves at size 140Mb but these then grew again to 240Mb and then 420Mb attached.

Thank you for looking at this. Mike

[donzi]

wow!

Code: Select all

<connection connection="drops">
<component class="collectableammo" macro="props_sm_container_xs_ammo_macro" connection="space" id="[0x34122]">
<events>

..with 7,137,143 lines of

Code: Select all

<event event="killcheck" time="962822.109"/>

..following and the final line (improperly terminated xml I might add..) having some 825,000 null bytes trailing it..

I see you have the teladi outpost DLC installed.. I guess that creates a bit more size on the saves.

Mine with vanilla 3.53 peak out and hover around 60 MiB for a completed campaign with some 8 stations, 15 cap ships and 30 or so small/medium ships.

Not certain of the total impact, but if you want I'll strip out the redundant lines and the null bytes so you can load and edit the file.

I don't have the DLC installed so I won't be able to test it. At least you'll have a starting point which can be loaded and edited if you want to fiddle with it yourself.

[donzi]

Off hand I think you may want to try and locate a very early save of the game your playing and see if it 'converts' to v3.53 okay and doesn't go out of control in size like what you've been seeing.

I trimmed out just the zillion lines of the final event in your save and the file results in .xml which is just under 27 MiB -- there are probably a good chunk of missing data which was lost somewhere during your playing, xml conversion, crash, etc.

Briefly comparing the structure of you file with one of my v3.50 converted to 3.53 saves (despite the teladi DLC in your game) it seems apparent the save is probably missing data that should be there.

I could add on any missing closing tags to the file and you could load it and see what the lay of the land is.. Might not be any worse than what you have now.. and maybe it will not go out of control.

fwiw..didn't pay attention before, but the save originally being v3.50. There is apparently the possibility that migration(s) create a slightly(?) different output .xml save than a purely v3.53 save.

[Mike Roberts]

Thank you for finding what was in the large files and offer of help.

First I am concerned that there is a BUG that is duplicating these 'event killcheck' lines. If this is not sorted I will have to edit the duplicates out every hour or so of play.

I don't think the file I posted is worth working on as it does not load and is probably corrupt/ missing correct xml termination.

I went back to a couple of earlier saves of sizes 75MB and 99MB. Removing the duplicate line you found reduced the sizes to just under 52MB and just over 53MB respectively. In the latter case I also found duplicates of the same line but with a different time. The sections of code after removing duplicate line are:

Code: Select all

-<connection connection="drops">
-<component connection="space" id="[0x8615e2]" macro="props_sm_container_xs_ammo_macro" class="collectableammo">
-<events>
<event time="957722.109" event="killcheck"/>
</events>
-<movement class="lineardeceleration">
<deceleration time="6" keepangular="1"/>
-<velocity>
<linear z="-5.9481" y="6.78779" x="0.316165"/>
<angular rz="-4.28167" ry="12.0438" rx="-21.7643"/>
</velocity>
<time start="942722.109"/>
-<offset>
<position z="105294.234" y="19943.174" x="313078"/>
<rotation roll="83.66483" pitch="62.47755" yaw="-122.4996"/>
</offset>
</movement>
-<offset>
<position z="229224.938" y="-313560.031" x="-36924.691"/>
<rotation roll="-49.42799" pitch="25.36599" yaw="176.13802"/>
</offset>
<source class="drop"/>
<ammunition macro="missile_player_dumbfire_macro" amount="6"/>
<connections/>
</component>
</connection> 

and

Code: Select all

-<connection connection="drops">
-<component connection="space" id="[0x8615d7]" macro="props_sm_container_xs_ammo_macro" class="collectableammo">
-<events>
<event time="957892.682" event="killcheck"/>
</events>
-<movement class="lineardeceleration">
<deceleration time="6" keepangular="1"/>
-<velocity>
<linear z="4.50659" y="-0.234347" x="0.436473"/>
<angular rz="-2.22211" ry="0.273409" rx="11.1058"/>
</velocity>
<time start="952492.682"/>
-<offset>
<position z="122639.797" y="-28536.998" x="-172317.516"/>
<rotation roll="73.69717" pitch="26.16624" yaw="-128.26575"/>
</offset>
</movement>
-<offset>
<position z="189749.734" y="171833.234" x="-8712.597"/>
<rotation roll="-83.24318" pitch="43.89509" yaw="-167.09753"/>
</offset>
<source class="drop"/>
<ammunition macro="missile_player_guided_heavy_macro" amount="2"/>
<connections/>
</component>
</connection>

I will have a go playing from this point. However I am very concerned that this looks like treating the symptom not the cause.

[donzi]

I read on the tentative update to address another problem (sort of similar to this one actually) where it's stated:

"Please note that when the fix is released, it will take about an hour of play before the affected savegame is back to its normal size and the game exits again at a more normal speed."

It implys that the game needs to continue to run for a time without interruption for the update to make the required changes and ultimately save files become v3.54 or whatever it'll be.

Anyhow, some chance there was a update (v3.50 - v3.53) queued in your game and XR didn't run long enough to complete things?

Seems like one of the few possibilities what could make some sense.. Especially since it's vanilla + official DLC.

I began playing at 3.50 in March and continued through to v3.53 with that game -- however I never added the DLC.

Progression across that game has 'game version' in the saves (below is from 4 separate save files) of:

Code: Select all

<game version="350" build="192864" time="91748.811" start="ep1"/>
..
<game version="351" build="193058" time="230629.191" original="350" start="ep1"/>
..
<game version="352" build="193370" time="531293.871" original="350" start="ep1"/>
..
<game version="353" build="193580" time="1179140.507" original="350" start="ep1"/>

..might be worth noting as you pick through saves -- the version info is near the top of the data. It might help to get a 'clean' save to resume with if you pick one that is an eariler version (in case some auto-update to the saves is the culpret).

Seems like if the DLC was somehow related though, it would have produced more complaints of huge save files.

Just out of curiosity, did your game begin with the DLC or added after starting the game?

Anyhow, I hope that you locate a reasonable save to regress your game with that eliminates the problem.

Worth a shot too -- maybe run the validate game files feature from the steam -> game properties area.

[Mike Roberts]

This is clearly a bug and is continuing. I have had good periods of several hours play and 20 or so saves all at the 50Mb size. Then it started growing, in jumps of the same large order. It was interesting to see the number of duplicates. The last one I fixed had 3 cases of duplicates. All were of the form shown on the code posted earlier. In each of the three cases the number of duplicates was 1048576 - that is 2^20. Might this help work out the cause?
The DLC was included from the start (bought together). The version data is:

Code: Select all

<game version="353" build="193580" time="1055626.279" original="350" start="ep1"/>

.. I don't have any save games from earlier versions. In any case this would be as good as scrapping all I have done and starting again. I don't think I am prepared to start again as nothing will have been done to the game code so same chance the problem will reoccur.

[donzi]

hi Mike,
too bad one of the saves didn't stabilize things. I can certainly understand the motivation killing effect of the issue.

I don't think I am prepared to start again as nothing will have been done to the game code so same chance the problem will reoccur.

Perhaps you're right.. It could be though that a purely v3.53 game will not have the same problem.

I can at least attest that a fresh vanilla v3.53 or my vanilla v3.50 - v3.53 games (neither using the DLC) have not exhibited any run-away save file growth.

I'm not intentionally pointing a finger at the DLC (I get the impression many are using it w/o problems) it's just the fact of my games so far.

I hope egosoft makes some use of your save and the next game update will salvage your existing work.

[Mike Roberts]

This problem seems to have gone away. Now have a months play without this reoccurring. So guess this will be one of life's mysteries! Thanks everyone for your help. If it does come back I will edit the file again. Not a problem if it is infrequent. Mike

[UniTrader]

i think this is resolved in the latest beta

Fixed memory leak in upkeep missions causing large savegames and long game shutdown times.