2.50HF1 possible map corruption issues? (Updated: dev comment would be nice at this point)

[strask412]

Hi all, I don't have enough data yet to properly report this as a bug but it would be helpful if others could reply as to if they have experienced similar issues or not since the update to 2.50hf1.

I've restarted my game 3 times now since updating to 2.50hf1 and have encountered this issue each time, the first two times it got bad enough that I felt forced to re-start my game from the beginning, and it's just now started to get ugly in my 3rd playthrough.

Symptoms include:

1. The little yellow person-head icon above destination ships/stations for passenger transport missions fail to get removed when the mission is completed.
2. Fog of war fails to clear when visiting new parts of space (as in, when viewing the map area around the ship I am currently piloting, the fog of war shows as if the area is unvisited / never explored, even though it is my current position).
3. Resource probes fail to detect ANY ore, even when deployed within 2-3 meters of a high-yield ore 'roid.

There are other symptoms that I'm failing to recall right now, but they all seem to have something to do with galaxy data as displayed on the map not getting updated.

I'm currently playing with a few mods:

1. Average Prices Suffixes by Sensei27
2. Improved Explore Command Behavior by MegaJohnny
3. Disable Encounters by MegaJohnny
4. Mass Move Marines by Tyrant597
5. Mass Move Service Crew by Drewgamer
6. Traders sell modification parts by t12121970
7. Ship Traders Sell Paint Mods by Kay Ence

Obviously if I have to restart a third time, which it looks like I will, I'll try a vanilla playthrough and see if that helps. But if someone else is experiencing this problem it would be good to know if you are using any of those mods.

[radcapricorn]

Even though mods may be affecting the magnitude of these issues, they're all vanilla, I can attest to that.

Usually though, the person icon does disappear after a few seconds. Resource probes do indeed sometimes find zilch in what appears to be rich environment. As for fog of war, there are various problems with it. For example, this. Or that the sensors don't update in a timely manner, making NPC ships appear over fog of war even though you're not supposed to see them. Also, there's this. Also, guidance lines on map sometimes fail. Also, I likely forget something as well...

[strask412]

Even though mods may be affecting the magnitude of these issues, they're all vanilla, I can attest to that.

Thank you for the reassurance that I'm not crazy.

I am now playing a vanilla game, and still encountering some issues although I'm not as far into the game as the time when they started to get intolerable before, so I still have some science to do.

My perception at the moment, however, is that my mods (in particular, the improved explore command and the disable-encounters) accelerated my detection of the bugs by simply accelerating my overall progress in the game.

[strask412]

While not conclusive, I thought it best to add a note at this time that my game has reached a similar stage to the first three attempts where I quit in frustration, and this time it still feels very playable. So some possibilities:

1. The seemingly-benign mods were potentially to blame for some of the issues, perhaps in some combination where any one of them wouldn't be an issue. This is the possibility that requires the most work to investigate.
2. The problems that were frustrating me the most (and which have not yet happened in my current unmodified playthrough) are sensitive to whatever randomization occurs at galaxy generation time, and the RNG was nice to me on this occasion.
3. I'm sure there are other options but I'm tired right now so I won't try to think of them all.

I think the best way forward for me right now is to start re-adding my mods, one at a time, and playing with each one for a few hours before adding the next, in case I can detect any kind of change tied to a certain mod.

[strask412]

Ok, so I decided to persist a bit and try another unmodified game start to experiment with the save file system. I won't explain why at this moment, suffice it to say that I am chasing down a probably-incorrect suspicion about something that the game might or might not do correctly.

I've run into an issue that I have trouble understanding. My procedure was:

1. First, reset all my logging and completely remove the save directory from my user data, in preparation for fresh game start.
2. Second, start a fresh game.
3. Immediately save the game as soon as I have control.
4. Fly around looking for the Boso Ta signal.
5. Find it, muck around a bit more, land on the station, walk over to pick it up, but do not pick it up yet.
6. IMPORTANT --- I PAUSE THE GAME AT THIS POINT, using the pause key in the upper right portion of my keyboard.
7. Now I hit shift-s to save the game, press enter to confirm the file name (maybe that's pressing enter twice, can't recall), and wait for the game to finish saving.
8. While the save is ongoing, and probably completely irrelevant, I sometimes (but not other times) switched out to my text editor to type something.
9. Then I repeated the "shift-s enter [ enter ] /* wait */" sequence five more times. (Edit: Oops, it's "shift-s down-arrow enter enter" )

At this point I have seven save files, and I quit to desktop.

save_001.xml is at zero minutes into the game, while the other six are after about 13 minutes of flying.

One would expect, since saves 2-7 were all taken in sequence while the game was paused, that they would be largely identical.

However, this is not the case. In the explanation below I'm going to paste some diffs, all taken with gnu diff options "-s --unified=10", and unless the forum has some handy feature that someone is about to tell me about, I strongly recommend cut-n-pasting these segments into your favourite code editor for syntax highlighting if you want to examine in any depth. I'll try to restrict the snippets to the very most useful bits up front, and then at the end I'll spoiler-tag the uncensored versions.

First up in our save file diff survey (not the same sequence I experienced it in) are the differences between saves 2 and 3.

Code: Select all

--- save_002.xml	2019-07-04 01:23:30.698296554 -0700
+++ save_003.xml	2019-07-04 01:23:31.282295617 -0700
@@ -1,14 +1,14 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <savegame>
 <info>
-<save name="#002" date="1562227371"/>
+<save name="#003" date="1562227409"/>
 <game id="X4" version="250" build="347849" time="835.602" code="21723" original="250" start="x4ep1_gamestart_discover" seed="1942228095" guid="1AE4069B-BFF0-4F20-826B-8FC0AD00CE8D"/>
 <player name="Selaia Tarren" location="{20004,230011}" money="5000"/>
 </info>
 <universe>
 <factions>
 <faction id="khaak">
 <relations locked="1">
 <relation faction="player" relation="-1"/>
 <relation faction="alliance" relation="-1"/>
 <relation faction="antigone" relation="-1"/>

Up at the top, minimal issues. Save file "name" (actually slot number) is recorded in the file, along with a timestamp (real time) for when the save occurred. Notably and reassuringly (I think) the <game time="835.602"> bit didn't change.

But then further on, I start seeing blocks like this:

Code: Select all

@@ -12225,20 +12225,23 @@
 <primary active="0"/>
 <secondary active="0"/>
 </weapongroups>
 <trade>
 <prices globalpricemodifier="1" buildpricefactor="1"/>
 </trade>
 <connections>
 <connection connection="con_storage01" macro="con_storage01">
 <component class="storage" macro="storage_gen_buildstorage_01_macro" connection="con_storage" id="[0x9894]">
 <offset default="1"/>
+<render>
+<parameter value="assets\fx\gui\textures\factions\faction_teladi" name="diffuse_map" type="texture"/>
+</render>
 <connections>
 <connection connection="con_cargobay_042" macro="con_cargobay_042">
 <component class="cargobay" macro="cargo_gen_buildstorage_01_macro" connection="con_cargobay_001" id="[0x9895]">
 <offset default="1"/>
 </component>
 </connection>
 <connection connection="con_cargobay_041" macro="con_cargobay_041">
 <component class="cargobay" macro="cargo_gen_buildstorage_01_macro" connection="con_cargobay_001" id="[0x9896]">
 <offset default="1"/>
 </component>

It's just the addition of something that seems to specify rendering of a teladi diffuse map texture on a station storage container? I'm not real clear on anything graphics-wise, couldn't even tell you what a diffuse map is, much less whether it should be something saved in a save file, but I couldn't help but wonder why it was in one of these save files, but not the other.

So then I ran a diff on save 2 as compared to save 7 -- again, both saves occurred during the same pause in game play, so should be fairly identical. But since 2 and 3 were different, I wondered what else I might find. Lo and behold:

Code: Select all

@@ -52005,22 +52133,22 @@
 <movement class="undock">
 <endoffset>
 <position x="0.001675" y="101.187" z="11.5"/>
 </endoffset>
 <time start="824.264"/>
 <offset>
 <position x="0.001675" y="12.875" z="11.5"/>
 </offset>
 </movement>
 <offset>
-<position x="3115.197" y="-3755.367" z="-4237.784"/>
-<rotation yaw="-36.24553" pitch="36.19342"/>
+<position x="3115.031" y="-3755.186" z="-4237.558"/>
+<rotation yaw="-36.24495" pitch="36.1932" roll="-0.000156449"/>
 </offset>
 <render>
 <parameter value="4278260500" name="letter_color" type="color"/>
 <parameter value="4280424994" name="number_color" type="color"/>
 <parameter value="assets\fx\gui\textures\factions\faction_teladi" name="diffuse_map" type="texture"/>
 </render>
 <drop ref="ship_medium_civilian"/>
 <source job="teladi_tech_trader_m" seed="1454440583582995944" zone="[0x8fc]" commandeerable="1" class="job"/>
 <modification>
 <paint ware="paintmod_0004" generated="1"/>
@@ -63536,22 +63664,22 @@
 <velocity>
 <linear x="200.961" y="-33.1347" z="-74.1646"/>
 </velocity>
 <time start="807.47"/>
 <offset>
 <position x="-533.054" y="188.846" z="214.842"/>
 <rotation yaw="110.25652" pitch="-8.79301"/>
 </offset>
 </movement>
 <offset>
-<position x="-528.563" y="188.106" z="213.185"/>
-<rotation yaw="110.25521" pitch="-8.79266" roll="0.000112925"/>
+<position x="-533.054" y="188.846" z="214.842"/>
+<rotation yaw="110.25653" pitch="-8.79301"/>
 </offset>
 <render>
 <parameter value="4279108101" name="letter_color" type="color"/>
 <parameter value="4280490787" name="number_color" type="color"/>
 <parameter value="assets\fx\gui\textures\factions\faction_paranid" name="diffuse_map" type="texture"/>
 </render>
 <drop ref="ship_medium_civilian"/>
 <source job="paranid_bio_trader_m" seed="4391467443826932285" zone="[0xc19]" commandeerable="1" class="job"/>
 <modification>
 <paint ware="paintmod_0038" generated="1"/>

I am most definitely not an expert in X4 internals, X4 scripting, or the X4 save file format. I'm just muddling about at this point. But it sure seems to me that objects are changing position and orientation while the game is paused, and that doesn't make me very optimistic. And looking back, I think the game playthroughs where I was taking my time, considering my decisions, and just pausing a lot were more error prone than the playthroughs when I was just trying to reproduce the errors as quickly as possible without the luxury of my mods, and therefore not pausing very much. So, part of me is hopeful that this is as alarming as I am alarmed.

Edit: sorry forgot this bit at first: the diff from save 6 to save 7 is potentially very instructive as well, I'll post it here in its entirety:

Code: Select all

--- save_006.xml	2019-07-04 01:23:32.970292910 -0700
+++ save_007.xml	2019-07-04 01:23:33.550291980 -0700
@@ -1,14 +1,14 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <savegame>
 <info>
-<save name="#006" date="1562227469"/>
+<save name="#007" date="1562227493"/>
 <game id="X4" version="250" build="347849" time="835.602" code="21723" original="250" start="x4ep1_gamestart_discover" seed="1942228095" guid="1AE4069B-BFF0-4F20-826B-8FC0AD00CE8D"/>
 <player name="Selaia Tarren" location="{20004,230011}" money="5000"/>
 </info>
 <universe>
 <factions>
 <faction id="khaak">
 <relations locked="1">
 <relation faction="player" relation="-1"/>
 <relation faction="alliance" relation="-1"/>
 <relation faction="antigone" relation="-1"/>
@@ -3589160,20 +3589160,21 @@
 <ref id="1214" string="$IsPirateBase"/>
 <ref id="1215" string="$LastKnownTime"/>
 <ref id="1216" string="$FirstKnownTime"/>
 <ref id="1229" string="$Yaw"/>
 <ref id="1255" string="unknown"/>
 <ref id="1280" string="preparing"/>
 <ref id="1296" string="$AssignedShips"/>
 <ref id="1360" string="$ReportCue"/>
 <ref id="1450" string="$weight"/>
 <ref id="1458" string="$chance"/>
+<ref id="9581" string="save"/>
 <ref id="1581" string="$ID"/>
 <ref id="1582" string="$MISSION_GENERATED"/>
 <ref id="1583" string="$MISSION_NO_VARIANT"/>
 <ref id="1598" string="signalleak"/>
 <ref id="1600" string="objectbased"/>
 <ref id="1606" string="$accepted_offer"/>
 <ref id="1607" string="$type"/>
 <ref id="1608" string="$text"/>
 <ref id="1609" string="Mission accepted"/>
 <ref id="1610" string="$DECLINED_OFFER"/>
@@ -3589514,21 +3589515,20 @@
 <ref id="9421" string="holyorder_scout_patrol_s"/>
 <ref id="9478" string="$performing"/>
 <ref id="9188" string="Attack"/>
 <ref id="9552" string="$preparing"/>
 <ref id="9521" string="$performing"/>
 <ref id="3245" string="$location"/>
 <ref id="3247" string="$lasttime"/>
 <ref id="9337" string="Cannot See, Cannot Scan"/>
 <ref id="9461" string="$performing"/>
 <ref id="9382" string="Attack"/>
-<ref id="9581" string="save"/>
 <ref id="3390" string="$name"/>
 <ref id="9329" string="(xenon) Defend area: Hatikvah's Choice I 0xb9a - 2 positions - "/>
 <ref id="3400" string="md_relation_discount_1"/>
 <ref id="3401" string="md_relation_discount_2"/>
 <ref id="3402" string="md_relation_discount_3"/>
 <ref id="9340" string="A Lucrative Opportunity"/>
 <ref id="9281" string="(holyorder) Defend area: Faulty Logic I 0xac0 - 2 positions - "/>
 <ref id="9223" string="[\012]Engineering: 2[\012]Morale: 2"/>
 <ref id="9175" string="(holyorder) Build station: station_gen_factory_base_01_macro in Holy Vision 0xb03 - "/>
 <ref id="9176" string="(holyorder) Recon: 0xd983 Empty Space - Sector: Holy Vision - Cluster: Holy Vision - "/>

That's really the entire thing. Pretty much how I expected ALL of the diffs to come out. (As a footnote, I'm pretty sure I scored a new point on the Unix purity test today by actually running diff on the outputs of two other diff commands, ) So I'm wondering if there might be some part of the save system that says "Hey everybody, shut up and wait I'm going to save the game!" and there are a couple bits that don't get the message and keep going, but then they run out of work before too long so eventually things stabilize?

I'm still investigating that other thing that inspired me to dig into the save files, but this thing with the apparent movement during pause is going to make investigating the other thing a lot tougher.

[strask412]

Possibly related, possibly not:

I have a habit, since loading from a save file is often just long enough to take a sip of a beverage, a puff on a vaporizer, or pet a cat, of hitting the pause key after telling it to load the save (after it says "initializing" and gives the quote of the day -- once the spinner starts counting percentage-loaded points). That way if my planned very brief time away from the keyboard turns into something longer accidentally (I spill my beverage, vaporizer is out of batteries/liquid, cat is more needy than expected) I don't have to be distracted (from the new distraction) by worrying about what the game is doing.

Today I've been doing even more saving and loading than usual, and during one load I used the save trick described above and when I un-paused after the load completed I noticed a bunch of items (entire stations) missing icons on the map, although they still appeared to be floading there in space and in one case at least was easily targetable (just not from the map). I immediately reacted by saving the game in a new slot, then attempting the same load of the previous save file again, this time without the pause. It loaded without any apparent error and the icons were present on the map as they should have been all along.

That made me think, combined with the possible partially-paused seeming stuff I described in an earlier post, that maybe the save/load system as a whole isn't respecting the paused/not paused state of the game very well? Is that a known thing? Should I avoid saving while paused, or pausing while loading?