[BUG][3.0-7.0][all DLC] (fixing in 7.50 beta) Black Marketeers may sometimes not spawn on non-DLC sectors

[Vectorial1024]

Environment info:

• Version: 7.0; however, since black market traders did not get much dev attention in recent years, suffice to say the bug existed over several game versions at least since v3.0
• Modified: yes; but should not matter since I am just reading the vanilla MD files and doing static analysis
• Game start: (open universe); but should not matter since black marketeer spawning does not really depend on the game start as revealed from said static analysis

(Edit: original post was titled "Several Potential Problems with Black Marketeer Spawning"; for clarity as recommended by others, I have narrowed the scope of the discussion to Problem 3.)

---------

Over the past few days doing sanity checks on my black marketeer (shadyguy) setup by re-reading md/npc_shadyguy.xml several times over and doing debug printing with my mod Market Discovery, I found several potential problems with their spawning. This basically matches some complaints seen in the past several years, and it is suprising these problems are left unresolved.

Problem 1: Not checking all the illegal drugs

(Edit: refer to viewtopic.php?t=464246)

Problem 2: Not checking for tradestations

(Edit: refer to viewtopic.php?t=464247)

Problem 3: Strange interaction with DLCs

The plug-and-play DLC policy means that late-joining DLCs may initialize their contents as the game is loaded/started. This includes setting up the shadyguys in the DLCs' space.

However, because cues are asynchronous in nature, there is very likely a problem if the DLCs are enabled when a new game is started. The exact mechanism is unclear, but my understanding is that, there may be a race condition.

Supposedly, the timeline should be the following:

• The player elects to start a new game with some DLCs enabled
• The md.NPC_ShadyGuy.Init cue is fired; this triggers md.NPC_ShadyGuy.EvaluateSectors
• By the cue md.NPC_ShadyGuy.EvaluateSectors, all sectors (including DLC sectors) are given black marketeers
• The md.NPC_ShadyGuy.Init enters "completed" state
• DLC shadyguy init scripts are fired, and check that md.NPC_ShadyGuy.Init should be "waiting", fails the check, and does nothing
• Black marketeers are available basically everywhere in the galaxy

However, the actual timeline is likely this:

• The player elects to start a new game with some DLCs enabled
• DLC shadyguy init scripts are fired, and check that md.NPC_ShadyGuy.Init should be "waiting", passes the check (because md.NPC_ShadyGuy.Init not fired yet!), and spawns black marketeers in their own sectors
• The md.NPC_ShadyGuy.Init cue is fired (late); it checks that there should be no spawned black marketeers, fails the check (DLCs already spawned their black marketeers), and cancels itself
• Setup complete; black marketeers are only found in DLC sectors

This deduced result matches with a lot of online posts (e.g. on Reddit and here) complaining about being unable to find black marketeers in the game, especially in non-DLC sectors. A trivial online search would yield some:

• viewtopic.php?t=426497 (2020)
• viewtopic.php?t=447260 (2022)
• https://www.reddit.com/r/X4Foundations/ ... space_why/ (2022)
• viewtopic.php?t=463749 (2024)

In this case, comments about "find another station", "retry on the same stations" etc are utterly useless because the black marketeers simply do not exist, and there are no way for them to be spawned after the game has started other than the DLC init scripts.

-----

There are now a lot of rumors and misinformation surrounding black marketeer spawning; to name a few:

• It is tied with faction rep (false)
• It is tied with the gamestart (technically true; Timelines DLC missions may have their own spawning logic)
• It is tied with "sector security" and "lawlessness" (false; not even quantifiable)
• Reports about "black marketeers not spawning" must be fake because the OP is delusional (false; I shall demonstrate in this bug report)

This has created too much confusion for too long, and should be dealt with.

[scryp]

Hey, I can also contribute something to this bug.


I started a new game with 7.0

Teladi start (entrepreneur again will)
Language: German
DLC: All
Modified: No

Problem:
In all original sectors (the ones without DLC) there are no black market traders, only in the DLC sectors they appear

After I scanned about 50 stations for black market traders and found none, I became a little skeptical and ran the tool from "foxxble" see viewtopic.php?p=5238278#p5238278 over it.

This confirms the absence of traders in the old sectors

Savegame: https://uploadnow.io/f/7MCswbr

[Varek Raith]

I added a Delay to the shadyguy cue call in all DLC setup mds and black marketeers now spawn everywhere in the galaxy as they should. Unsure of any unintended side effects of such a delay.

[Mookau]

• It is tied with the gamestart (technically true; Timelines DLC missions may have their own spawning logic)

I'm not sure why this one is in your list of misinformation/misconceptions. You say it's technically true, and then don't really offer anything to expand upon that. Did you just forget to elaborate on it?
The only exceptions that come to mind are for the black market Curs mission, or if DLC is enabled after game start. You could argue black marketeers moving after their station is destroyed is spawning, but that's probably not what people are referring to when they mention this.


But I agree that the issue is likely a race condition.
When a DLC is enabled it checks if the 'EvaluateSectors' cue is in the 'waiting' state, and if so it then spawns shady guys in the relevant DLC sectors.
The base game doesn't have a check to see if it's waiting (as it's supposed to fire first), it only checks if any shady guys have already been spawned, and if so, it doesn't spawn any more.

So if a DLC script fires first, then it sees the 'EvaluateSectors' cue is waiting, spawns it's shady guys, which increases the global shadyguy count. Because the count isn't 0 anymore, it then prevents the script for the base game from spawning them when it eventually fires.

I added a Delay to the shadyguy cue call in all DLC setup mds and shady guys now spawn everywhere in the galaxy as they should. Unsure of any unintended side effects of such a delay.

I don't foresee it being an issue given how most of the functions are called when DLC is enabled, or when a station with a shady guy is destroyed.

Though instead of a delay, you could probably change:

Code: Select all

<check_value value="md.$ShadyGuyMap.keys.count == 0"/>

To be in line with the DLC scripts:

Code: Select all

<cue_is_waiting cue="md.NPC_ShadyGuy.EvaluateSectors"/>

That way it wouldn't rely on timing anymore, and would simply add shady guys to the base game sectors on gamestart/init in the same way the DLC does. (Honestly, I'm not even sure why it checks that there are 0 shady guys.)

[db48x]

• It is tied with the gamestart (technically true; Timelines DLC missions may have their own spawning logic)

I'm not sure why this one is in your list of misinformation/misconceptions. You say it's technically true, and then don't really offer anything to expand upon that. Did you just forget to elaborate on it?
The only exceptions that come to mind are for the black market Curs mission, or if DLC is enabled after game start. You could argue black marketeers moving after their station is destroyed is spawning, but that's probably not what people are referring to when they mention this.

He’s not talking about which DLC you have, but which starting scenario you choose. The shady guys should be the same regardless of whether you start as a Terran Cadet or as a Spear of the Patriarch or as an Unworthy Entrepreneur.

[Vectorial1024]

• It is tied with the gamestart (technically true; Timelines DLC missions may have their own spawning logic)

I'm not sure why this one is in your list of misinformation/misconceptions. You say it's technically true, and then don't really offer anything to expand upon that. Did you just forget to elaborate on it?

I did not "forget" to elaborate it because the full explanation would be too long; here is a full explanation with minor spoilers to Timelines DLC:

Gamestarts include many information, e.g. which galaxy to start the game in. For most players and instances, since they are playing in open universe games, no matter which "gamestart", they all start in the default "xu_ep2" universe, where you will find e.g. Argon Prime, Trinity Sanctum, Ianamaus Zura, etc in your game, along with all your DLC sectors.

However, there are other technical gamestarts that are gamestarts but players don't think of them as proper "gamestarts". These are tutorial scenarios, and most importantly, every "map" (the hall, each nodus, each terminus) in the Timelines DLC (each "map" is a separate galaxy in the game's technical terms).

The starting scenario of a nodus mission in the Timelines DLC (you may use this to infer the intended 5-star solution to the mission)

Show

Specifically in e.g. A Merchant's Mind nodus, you have strong vision to the stations in the lone sandboxed sector Black Hole Sun V, but there is consistently only 1 station that has a shadyguy. This violates the vanilla init spawning logic where a <set_value min="1" max="(station count)" profile="increasing" /> statement results in (on average) more than half of the stations in the sector getting a shadyguy. Clearly, there is a different spawning logic to this nodus gamestart.

This situation technically fulfills the "depends on gamestart" description, but is entirely distracting in our discussion of black marketeer spawning; clearly, the original "depends on gamestart" statement tries to allude to e.g. "Terran Cadet will have more/less shadyguys than Fires of Defeat", which is simply false as elaborated by me.

[Mookau]

He’s not talking about which DLC you have, but which starting scenario you choose. The shady guys should be the same regardless of whether you start as a Terran Cadet or as a Spear of the Patriarch or as an Unworthy Entrepreneur.

That.. makes much more sense.

I know the game refers to a new game scenario as a "game start", but I failed to make that connection when I read the list. The main discussion I've read is usually about how shady guys are only determined on when you first start a new game/save, as opposed to the scenario picked. Given that shady guys are determined when you start a new game, and as already mentioned - is a large part of why this bug is an issue, you can probably see where my confusion came from.

is entirely distracting in our discussion of black marketeer spawning

Sorry for putting you on a tangent due to my misunderstanding.

Though I suppose it is possible that the scenario picked actually does affect shady guy spawns simply because of this bug. As load times/order might be affected by what is initialized by the scenario.

It would help the bug fixing process if it can be consistently reproduced. I'd like to find a way to reproduce it on my setup, as I haven't noticed it in my own games.
In that vein: Is there a particular game start/scenario where this race condition is known to happen more often, or are they all more or less the same?

I see the save that scryp uploaded is the: "The Unworthy Entrepreneur" scenario, which I haven't done myself. So I might start with that. But it would be nice to have more data points to work with.

[Vectorial1024]

It would help the bug fixing process if it can be consistently reproduced. I'd like to find a way to reproduce it on my setup, as I haven't noticed it in my own games.

I would do this from the theoretical/script-reading point of view because script cues are asynchronous in nature and cannot be "consistently" tested, but this bug should be observable by testing every gamestart:

Given an open world gamestart G, do the following at least 2 times:

1. Start game with G
2. Wait for completion
3. Save (you don't need to exit)
4. Pass the save file to viewtopic.php?p=5238278 (perhaps the timing of tool creation was a coincidence, but whatever)
5. Look at the distribution of shadyguys

Now, for the benefit of the doubt, perhaps we are too unlucky in the 1st run and all the shadyguys are in DLC space. But the probability of this happening a 2nd time should be quite low; if the 2nd run still sees all the shadyguys in DLC space, then the bug is basically confirmed.

You can go further by disabling DLCs from the extension menu and test e.g. 5-6 times. If execution order is random, then only having 1 DLC should see 50% chance of finding shadyguys in vanilla space. More DLC should see reduced chance of finding shadyguys in vanilla space.

[Mookau]

I've done a bunch of testing and debugging on this today and I believe I've locked onto the exact cause of the issue.
You were right on the money. It is a race condition due to 'EvaluateSectors' is being called by a DLC before 'Init' is called, causing 'Init' to fail the 0 shady guy check.

The reason it largely went unseen until 5.0 is because ToA's setup script calls 'EvaluateSectors' very early, where all other DLCs called it late in their setup scripts.
Due to this, having ToA enabled on a new game is fairly likely to cause this issue. Where it may be theoretically possible with the other DLCs, the chances are vanishingly small.

The bug is on the tracker now [#2573]. Hopefully it's an easy fix. Though fixing the broken state of existing saves is something that may need to be considered.

[Vectorial1024]

The bug is on the tracker now [#2573].

Is this tracker public? I can't seem to find it myself.

---

Note that there are 3 problems pointed out in this post, but good that EgoSoft recognized this as a problem.

[Mookau]

The bug is on the tracker now [#2573].

Is this tracker public? I can't seem to find it myself.

No. Sorry.

If you want bugs to be accepted, unless they are closely related (as to be the same cause) you should probably stick to one issue at a time.

[Vectorial1024]

The bug is on the tracker now [#2573].

Is this tracker public? I can't seem to find it myself.

No. Sorry.

If you want bugs to be accepted, unless they are closely related (as to be the same cause) you should probably stick to one issue at a time.

Ah ok... I will create 1 post for each of the remaining problem then.

[Vectorial1024]

It seems the 7.10 changenotes (beta 5 as of writing) did not mention anything about black marketeers so far?

[vvvvvvvv]

Per request of another user.

Here's a 7.1 unmodified save where this problem probably occurred.

https://www.dropbox.com/s/5huhrqjbjspdp ... ml.gz?dl=0

[vvvvvvvv]

Per request of another user.

Here's a 7.1 unmodified save where this problem probably occurred.

https://www.dropbox.com/s/5huhrqjbjspdp ... ml.gz?dl=0

---

I wasted several hours trying to locate a black market station in ANT, ARG and HAT. No luck. Probably scanned in ballpark of 40 leaks.

It is taken at the time I got fed up, and "Katana Explorer" is parked somewhere next to last candidate I tried. There's unfinished megafactory at HQ, and some oddly staffed ship, but, eh, nothing unusual. I believe the save is subjected to the bug, because the only way I could find a black market was using a mod, and that mod "rerolled" black market locations. For example, the mod pointed me at... eh... I think it was station HUU-581, and t here was a black market. Without mod there was nothing. Because of the inordinate amount of time search took, I assume this is subject to the bug.

[zakaluka]

may I ask, would these steps possibly be a reasonable workaround?

- disable all DLCs
- start a new game, save
- enable all DLCs and reload

would these steps force initialization in the correct order?

(I recently observed a couple new starts affected by this bug, but that's not the main point of my question - just concurring that this still happens occasionally)

[db48x]

Yes, I believe that would ensure that the bug didn’t happen.

[G315t]

Here is another unmodified savegame with the shady guys missing from basegame sectors for your reference (from my bug report in the German forum)

Savegame
https://www.file-upload.net/gal-345264/exz4d1/1.html

[zakaluka]

an interesting occurrence. I started six new games this morning. The first five all only spawned fewer than 350 black markets, very few in non-DLC sectors.

I got to thinking, why does this happen to me sometimes so consistently, and other times rarely if ever? I tried creating new games from the title menu, and from within a loaded game. No difference, again, first 5 games are all affected by this bug.

It occurred to me - my game had been left running overnight, paused. I know it's a bad habit, but I forgot. Upon exiting and re-launching the game, my first new game now has over 800 shady guys all distributed as I'd expect.

It's a large file but I have set aside all these save games. If it is relevant I can share them. I wonder if this is a clue or perhaps a link between reports?

[vvvvvvvv]

There's a tool called shady-search for finding traders:

https://github.com/foxxbl/shady-search

Judging by fixed bug report here:

https://github.com/foxxbl/shady-search/issues/20

Problem might be appearing when there's "cluster_black_sector01_macro" somewhere in the save. Because normal savegames without a bug do not trigger this problem.