[1.5][BUGS] Summary of issues found with trade.find.free affecting ships that use it - Dev input now.

[Vim Razz]

I've been buried in //aiscripts/trade.find.free for a few weeks now, and have been able to identify several bugs and other issues currently causing problems:

1) The $bestselloffers table is currently being populated by the worst sell offers from the $table_selloffers table, rather than the best.
The script treats the prioritized sell offers list as if the best deals are at the front and the worst are at the back, but the manner in which the table key used to sort them ($tradevalue) is generated puts the best deals for each ware towards the back and the worst towards the front. That is: -(relativeprice - 2) creates a larger number the lower (or further below zero) the relativeprice value is, and multiplying by the quantity expands on that, putting the best deals towards the end. This results in the the worst sell offers found being placed on the $bestselloffers list, rather than the best.

2) The scripts final decision making mechanism is a bit lacking.
The script's final decision making mechanism simply picks the first item from the $bestselloffers table it encounters that it happens to find a suitable buy offer for, and executes that trade. The primary problem with this is that the $bestsellofferslist list only prioritizes those trades alphabeticaly by ware name (searched in reverse order). Since it's just the the best sell offer found for each individual ware, the entry for any given ware on the list may be significantly better or worse than the entry for any other given ware.

3) The $tradevalue metric used to evaluate trades does reasonably well when comparing trades of an individual ware, but not so well comparing trades of different wares.
While $tradevalue is fine for populating the $bestselloffers table and the $bestbuyoffers table, it would be nice to see something more appropriate used if issue (2) is ever addressed.

4) Traders stop their searches as soon as any trade offers are found in any particular zone.
If this is the intended behavior of player-owned free traders, then it is not well communicated to the player.

5) Certain filters or adjustments don't make sense for player-owned ships.
The added prioritization to Build Storage units and the 75% chance of skipping over a trade if both buyer and seller are in the same zone make sense for NPC faction traders, but make little sense when applied to the player's ships.

EDIT: Number 6 is wrong, and should be disregarded.
6) Stray $selloffer values can "escape" and cause the trader to buy wares with no $buyoffer to sell them.
If a trade evaluation fails because either the buy offer price is lower than the sell offer price, or because of the 75% chance to skip trades in the same zone, and happens to be the last ware in the list to be evaluated, then the $selloffer value does not get removed before the search cycle goes into it's fail state and starts over. If it fails enough times to give up entirely, then the previously rejected $selloffer value will get passed back to the calling script with no matching $buyoffer, and the trade ship will buy wares with nowhere to sell them to.

7) Some ships attempting to sell current cargo do not reliably produce $bestbuyoffers tables, despite having found suitable trade offers.
Despite having found suitable buy offers from stations, and successfully generated both $table_buyoffers and $list_sortedbuyofferkeys with appropriate buy offers on them, some ships selling their current cargo due to the $sellcurrentcargo flag simply do not generate $bestbuyoffers tables and get locked in a perpetual state of searching for station buy offers over and over again. This appears to happen most often with NPC non-mining ships, and I have been unable to figure out why some ships behave this way while others do not.


Some other things I've noticed that are not directly related to trade.find.free, but that seem worthy of mention:
a) The $sellspaces lists that player-owned autominers are sending to trade.find.free always contain a list of all zones on the map, regardless of gate distance settings for selling their cargo. NPC-owned autominers do not have this problem.
b) If the relativeprice filter in trade.find.commander that prevents player-owned station traders from selling their homebase's products if the player has set the price at or above average is intentional, then it's purpose (or even the existence of such a restriction) is not well communicated to the player.


Anyway, hope you guys had a happy holidays and stuff. Also, -scriptlogfiles may be the greatest thing ever. Thanks!


Edit:
After further investigation and experimentation over the weekend, I'm pretty sure that my interpretation of what's happening related to issue (6) in the list above is wrong, and should be disregarded.

I am still comfortable with everything else on the list, though.

[j.harshaw]

1) The $bestselloffers table is currently being populated by the worst sell offers from the $table_selloffers table, rather than the best.
The script treats the prioritized sell offers list as if the best deals are at the front and the worst are at the back, but the manner in which the table key used to sort them ($tradevalue) is generated puts the best deals for each ware towards the back and the worst towards the front. That is: -(relativeprice - 2) creates a larger number the lower (or further below zero) the relativeprice value is, and multiplying by the quantity expands on that, putting the best deals towards the end. This results in the the worst sell offers found being placed on the $bestselloffers list, rather than the best.

Your analysis is correct, and thank you very much for pointing out the error! This results in free traders tending to prefer trades involving low-amount or high-price purchases and attempting to match those prospective purchases with sales. This results in traders either trading in low amounts or failing to find matching sales for a long period of time.

This has now been fixed.

2) The scripts final decision making mechanism is a bit lacking.
The script's final decision making mechanism simply picks the first item from the $bestselloffers table it encounters that it happens to find a suitable buy offer for, and executes that trade. The primary problem with this is that the $bestsellofferslist list only prioritizes those trades alphabeticaly by ware name (searched in reverse order). Since it's just the the best sell offer found for each individual ware, the entry for any given ware on the list may be significantly better or worse than the entry for any other given ware.

Unless i misunderstood your argument, this should be fine. The $bestselloffers table only contains one entry per ware. When looking for matching sales, it looks from the supplier's space outwards, stopping once it finds a sale to match one of the prospective purchases.

3) The $tradevalue metric used to evaluate trades does reasonably well when comparing trades of an individual ware, but not so well comparing trades of different wares.
While $tradevalue is fine for populating the $bestselloffers table and the $bestbuyoffers table, it would be nice to see something more appropriate used if issue (2) is ever addressed.

$tradevalue was actually designed precisely to compare prospective trades involving different wares. It was a function of relative price, amount, and distance. Distance was later dropped since it is already implicitly weighted in the logic. What, in your opinion, is wrong with it?

4) Traders stop their searches as soon as any trade offers are found in any particular zone.
If this is the intended behavior of player-owned free traders, then it is not well communicated to the player.

This was mentioned above, but i thought it best to reserve more detail until we get to this point. This was done very deliberately for a couple of reasons:

1. Performance. The various find_* actions are some of the most expensive functions in the game, and those finding trade offers are the most expensive of those. For this reason, we gain a significant performance improvement simply by limiting the number of times these functions are called.
2. Prefer short distances. One of the intangible costs of trading is travel time and, as such, trades involving short distances are preferred.

Note that this does result in free traders not finding the best trades available in their entire space from a monetary point of view. Rather, the logic is to find trade-pairs that are good enough, however "good enough" is defined, as close together as possible. Note that the objective of the logic is not to find the absolute best trades in the space defined, but to find acceptable trades and keep trading with minimum fuss.

Does this logic, in your opinion, result in them actually finding bad trades?

5) Certain filters or adjustments don't make sense for player-owned ships.
The added prioritization to Build Storage units and the 75% chance of skipping over a trade if both buyer and seller are in the same zone make sense for NPC faction traders, but make little sense when applied to the player's ships.

Good point. Set to only apply to NPC traders who do not belong to the player faction.

6) Stray $selloffer values can "escape" and cause the trader to buy wares with no $buyoffer to sell them.
If a trade evaluation fails because either the buy offer price is lower than the sell offer price, or because of the 75% chance to skip trades in the same zone, and happens to be the last ware in the list to be evaluated, then the $selloffer value does not get removed before the search cycle goes into it's fail state and starts over. If it fails enough times to give up entirely, then the previously rejected $selloffer value will get passed back to the calling script with no matching $buyoffer, and the trade ship will buy wares with nowhere to sell them to.

Investigating.

7) Some ships attempting to sell current cargo do not reliably produce $bestbuyoffers tables, despite having found suitable trade offers.
Despite having found suitable buy offers from stations, and successfully generated both $table_buyoffers and $list_sortedbuyofferkeys with appropriate buy offers on them, some ships selling their current cargo due to the $sellcurrentcargo flag simply do not generate $bestbuyoffers tables and get locked in a perpetual state of searching for station buy offers over and over again. This appears to happen most often with NPC non-mining ships, and I have been unable to figure out why some ships behave this way while others do not.

Investigating.

a) The $sellspaces lists that player-owned autominers are sending to trade.find.free always contain a list of all zones on the map, regardless of gate distance settings for selling their cargo. NPC-owned autominers do not have this problem.

Odd. That variable is initialized the same way in all cases. Will check.

b) If the relativeprice filter in trade.find.commander that prevents player-owned station traders from selling their homebase's products if the player has set the price at or above average is intentional, then it's purpose (or even the existence of such a restriction) is not well communicated to the player.

Good point, and obsolete as of 1.50. Restriction removed. Player-owned station-based traders should now sell at the prices the player sets at the station regardless of what those prices happen to be.

Fixes and changes mentioned above should be in a future update. Points marked "investigating" are being investigated now and should be acted upon within the day. Points marked "will check" are considered lower priority and will be investigated if time allows.

Thank you very much for the analysis and the detailed feedback!

edit: your edit regarding point 6 is noted. will downgrade to will check, time allowing. thanks again!

edit 2: noticed some questions in your mod thread. would you like me to answer there, or would you prefer to go through it on your own? would have to do it in my free time, though.

[j.harshaw]

Update: have not managed to reproduce points 6, 7, and a.

Note on point a: added error-checking and recovery if minbuy/sell is ever greater than maxbuy/sell or if min and max are equal positive integers. Can't see how it would lead to the situation described, but bullet-proofing was applied before test was done.

[magictrip]

Update: have not managed to reproduce points 6, 7, and a.

Note on point a: added error-checking and recovery if minbuy/sell is ever greater than maxbuy/sell or if min and max are equal positive integers. Can't see how it would lead to the situation described, but bullet-proofing was applied before test was done.

On the item 6, I don't know if it's the exact problem he mentioned but the symptom is the same, you can find a save with the problem occurring here:
viewtopic.php?f=180&t=411610

What happens is that traders not assigned to a station will sometimes buy stuff and just sit there doing nothing with the cargo full. As soon as you drop the cargo they resume the process. It looks like this issue is more common when you have player stations consuming the item that is been traded.

[Raevyan]

Does the logic of trader aims for the shortest acceptable trade also apply to station traders? If so why does a station trader from Holy Vision tries to sell to Antigone Memorial when there is an evenly good trade for the same ware with max cargo amount in the same sector? Kinda dumb that he picks 10 units and trying to sell across the universe

[magictrip]

Does the logic of trader aims for the shortest acceptable trade also apply to station traders? If so why does a station trader from Holy Vision tries to sell to Antigone Memorial when there is an evenly good trade for the same ware with max cargo amount in the same sector? Kinda dumb that he picks 10 units and trying to sell across the universe

I believe the reason is the item 1 of his list. They say that they fixed but I don't think they released the fix yet.

[Vim Razz]

Thank you for taking the time to reply. Please forgive me for replying to some things out of chronological order, and I'm going to need to reply in several installments due to limited time:

edit 2: noticed some questions in your mod thread. would you like me to answer there, or would you prefer to go through it on your own? would have to do it in my free time, though.

I'm happy to leave that to your discretion. I imagine you're got a lot on your plate already, and I'll be going through things there on my own regardless.

2) The scripts final decision making mechanism is a bit lacking.
The script's final decision making mechanism simply picks the first item from the $bestselloffers table it encounters that it happens to find a suitable buy offer for, and executes that trade. The primary problem with this is that the $bestsellofferslist list only prioritizes those trades alphabeticaly by ware name (searched in reverse order). Since it's just the the best sell offer found for each individual ware, the entry for any given ware on the list may be significantly better or worse than the entry for any other given ware.

Unless i misunderstood your argument, this should be fine. The $bestselloffers table only contains one entry per ware. When looking for matching sales, it looks from the supplier's space outwards, stopping once it finds a sale to match one of the prospective purchases.

My primary concern is it's extreme "reverse alphabetical bias", which I believe to be one of the primary causes of economic stagnation and shortages of particular wares. Even though there may be plenty at the factory, certain wares simply do not get moved as often as they need to be for the game's economy to function.

Consider an NPC high-tech trader who might be looking at best buy/sell pairs of: Quantum Tubes, Microchips, Hull Parts, and Engine Parts.

It will always choose the Quantum Tubes unless that trade is explicitly rejected, even if it's only trading the smallest possible cargo load, no matter how severe the need may be for Microchips, Hull Parts, or Engine Parts.

After a few hours of gameplay, a large proportion of trade ships are running around moving as little as they can get away with. They will not consider moving other wares until either their current preferred ware becomes unavailable at an acceptable price, or else the clients for their preferred ware become completely stagnant.

3) The $tradevalue metric used to evaluate trades does reasonably well when comparing trades of an individual ware, but not so well comparing trades of different wares.
While $tradevalue is fine for populating the $bestselloffers table and the $bestbuyoffers table, it would be nice to see something more appropriate used if issue (2) is ever addressed.

$tradevalue was actually designed precisely to compare prospective trades involving different wares. It was a function of relative price, amount, and distance. Distance was later dropped since it is already implicitly weighted in the logic. What, in your opinion, is wrong with it?

It has no consideration of profit margin (generally of interest to the player), and has an implicit bias against wares with large volume (because a larger number of smaller wares can fit into a container of a given size) which has a significant effect on the overall game economy if applied as a primary metric for comparing trades of different wares.

To illustrate this, I'm going to pull some numbers off of this trade decision record that I happened to post in the thread for my mod (and reformat things a bit so they're easier to read). The "sorted buy/sell offers lists" are the $table_selloffers and $table_buyoffers sorted by their $tradevalue/key value. Although the game is obviously modded, the $tradevalue calculations are not changed. The ship is a Demeter Vanguard with 7900 cargo space. These are all full loads of cargo being considered (the trade offer quantities are in parenthesis).

Sell offer $tradevalue ("key value") rankings:

Code: Select all

key value:   789,   	  margin: 31,560 Cr,   	relprice: -1,   	ware: Engine Parts,   	qty:  263 (8346),   	avg.price: 600 Cr,   u.price: 480 Cr
key value:   948,   	  margin: 32,548 Cr,   	relprice: -1,   	ware: Hull Parts,   	qty:  316 (11139),   	avg.price: 516 Cr,   u.price: 413 Cr
key value:  1077,   	  margin: 27,284 Cr,   	relprice: -1,   	ware: Quantum Tubes,   	qty:  359 (5889),   	avg.price: 511 Cr,   u.price: 435 Cr
key value:  1181.791,	  margin:  6,268 Cr,    relprice: -0.991875,   	ware: Graphene,   	qty:  395 (5894),   	avg.price: 166 Cr,   u.price: 150 Cr
key value:  1314,    	  margin:  9,198 Cr,    relprice: -1,  		ware: Antimatter Cells, qty:  438 (44539),   	avg.price: 202 Cr,   u.price: 181 Cr
key value:  1479,  	  margin:  9,860 Cr,    relprice: -1,   	ware: Teladianium,   	qty:  493 (54986),   	avg.price: 202 Cr,   u.price: 182 Cr
key value:  2188.265,	  margin: -4,257 Cr,   	relprice:  0.139231,   	ware: Sunrise Flowers,  qty: 1176 (1176),   	avg.price: 132 Cr,   u.price: 135 Cr
key value:  2632, 	  margin:      0 Cr,   	relprice:  0,   	ware: Water,   		qty: 1316 (33333),   	avg.price:  53 Cr,   u.price:  53 Cr
key value:  3948, 	  margin: 44,744 Cr,   	relprice: -1,   	ware: Swamp Plant,   	qty: 1316 (9802),   	avg.price: 138 Cr,   u.price: 104 Cr
key value:  7899, 	  margin: 21,064 Cr,   	relprice: -1,   	ware: Spices,   	qty: 2633 (39259),   	avg.price:  38 Cr,   u.price:  30 Cr
key value:  7900, 	  margin:      0 Cr,   	relprice:  0,   	ware: Medical Supplies, qty: 3950 (99883),   	avg.price: 115 Cr,   u.price: 115 Cr
key value: 15800, 	  margin:      0 Cr,   	relprice:  0,   	ware: Nostrop Oil,   	qty: 7900 (200000),   	avg.price:  60 Cr,   u.price:  60 Cr

Granted, not all of these wares are going to be in the same NPC trader basket, but even among those that are, the volume bias is pretty overwhelming. Even "bad" deals on small wares score significantly higher the the best possible deals on large wares.

Personally, I'm partial to ($buyoffer.relativeprive - $selloffer.relativeprice) * (total trade volume) for NPC traders, and profit margin for player-owned traders.


And I'm going to need to respond to the rest tomorrow. It's getting quite late. One quick last thing first, though:

b) If the relativeprice filter in trade.find.commander that prevents player-owned station traders from selling their homebase's products if the player has set the price at or above average is intentional, then it's purpose (or even the existence of such a restriction) is not well communicated to the player.

Good point, and obsolete as of 1.50. Restriction removed. Player-owned station-based traders should now sell at the prices the player sets at the station regardless of what those prices happen to be.

The filter at line 145 is still causing mischief. (Though I do greatly appreciate that the filters at lines 176 and 182 were changed in v1.30)


Thank you again for your time.

[Detritis Max]

Hi,

I am having the same issue as 7a, I have started a new game on 'The Young Gun' on 1.50 release and its unmodded.

I have only explored 3 systems so far, namely Black Hole Sun IV, Second Contact II Flashpoint & Argon Prime. I purchased a Drill Vanguard, standard mining configuration and hired a Captain (which ended up being a 4 Star Pilot).

I sent it off to Black Hole Sun IV to AutoMine for Ore. I have set its default behaviour to AutoMine 'Ore' and the 'Max Gate distance to gather resources' set to 0/26. Once it arrived in Black Hole Sun IV, I cancelled the 'Fly and Wait' order and let it sit idol.

After a few minutes it started on its AutoMine mission but immediately headed for the JumpGate and straight to Argon Prime. It proceeds to AutoMine happily in Argon Prime only. The behaviour is the same if I tell it to AutoMine 'Silicon' or 'Ore' in Second Contact II, it just heads to Argon Prime.

[Kryten42]

Update: have not managed to reproduce points 6, 7, and a.

Note on point a: added error-checking and recovery if minbuy/sell is ever greater than maxbuy/sell or if min and max are equal positive integers. Can't see how it would lead to the situation described, but bullet-proofing was applied before test was done.

On the item 6, I don't know if it's the exact problem he mentioned but the symptom is the same, you can find a save with the problem occurring here:
viewtopic.php?f=180&t=411610

What happens is that traders not assigned to a station will sometimes buy stuff and just sit there doing nothing with the cargo full. As soon as you drop the cargo they resume the process. It looks like this issue is more common when you have player stations consuming the item that is been traded.

I can confirm this. I have to check all my AutoTraders regularly to see if some got stuck with a full inventory. I think all they need is a check for cargo they could sell before they initiate a new deal.

[ubuntufreakdragon]

Balancing wise it might be a good idea to let NPCs only find trades with at least 0.2 difference in relative price to artificialy create holes for the player.

[sh1pman]

I don’t know if it’s caused by #6 or not, but in my game I often see NPC traders buying wares at my station and then dropping them into space. It happens if there are no buy offers for these wares in the nearby 3-5 zones.

For example, when the demand for Engine Parts in the nearby Wharf was satisfied, the NPC traders started to drop them shortly after buying. It happens fairly often, sometimes several traders at a time do this. I can provide a save file if needed.