BETA Release - Factory Loop Delivery Software - v0.12

[Aye Capn]

This is nuts: ASDS seems to strictly favor larger* loads first, followed by need only to break a tie. (Need indirectly becomes a factor when the destination fills up and the load size of that route is forced to shrink.)

Totally the opposite of FLDS.

I'm going to finish fixing FLDS to give both sides equal priority, and see how it works out. I suspect Ticaki knows something about this that I don't.


* "Larger" and "smaller" in the context of cargo size means "as a proportion of the factory's total warehouse capacity", so transferring a tiny cargo of swamp plants which represent 40% of a Dream Farm's warehouse capacity is valued the same as a gigantic 40% cargo of majaglit from a jewelry factory.

[TSM]

I assume you have actually contacted the Author of this script rather than just pull it to pieces on an open Forum?

[Aye Capn]

"Comments and bug reports are very very welcome."
-- Nazgutek

[Aye Capn]

I do wish he were here, as he could tell me what the consequences would be of taking out the critical section flag entirely.

ASDS doesn't appear to use critical sections at all, and I remember from the "Atomic Coding" chapter of the MSCI Handbook that task-switching only takes place at task-switching points (marked "@").

That's a lot like a critical section.

Maybe FLDS could take advantage of that? Maybe not. Only Nazgutek could really know for sure.

[Aye Capn]

The original fix I posted was correct. The odd behavior was a result of me misconfiguring my test bed.

Oops.

There is a problem with the original fix which I haven't "proven" in live testing but which is real: not all stations have the same inventory capacity for a given ware, so you can't add source and destination quantities and divide once.

You need two seperate divisions to build two separate ratings, then you can average (or bias) them and be comparing "apples to apples".

The variable "$Temp.Source.Amount.Maximum" just happens to have what I need and "$Temp.Ship.Amount.Free" is safe to use here as a temporary variable:

222 $Temp.Ship.Amount.Free = $Route.Delivery.Amount * 10000 / Temp.Source.Amount.Maximum
223 $Route.Priority = $Route.Destination -> get max amount of ware $Route.Ware that can be stored in cargo bay
224 $Route.Priority = $Route.Needed.Amount * 10000 / $Route.Priority
225 $Route.Priority = ( $Temp.Ship.Amount.Free + $Route.Priority ) / 2

This handles the case in which the destination is a high-end factory with extra large inventory capacity.

I also changed to "delivery amount" instead of "pickup amount" because we want to get rid of cargo already in the cargo bay.

[Aye Capn]

Ok, I have it! The perfected, flight-tested FLDS delivery priority formula!

New Delivery Rules:

1. Full sources and empty destinations have equal priority.
2. Full destinations (defined as less than the lesser of source inventory and a full ship cargo bay) revise the source priority down to the actual amount that can be delivered. (The original source inventory is still factored in so that even a route with a clipped source priority will pick the source with the largest inventory.)

Preserved Original Delivery Rules:

3. Transport docked at source gives route double priority.
4. Small loads of wares not in critical shortage are rejected.

Critical Section Tweaks (unwarranted paranoia officially cured):

1. Critical section check delay INCREASED to 1.2 seconds. Checking it 10 times per second spams the debugger log and needlessly burns CPU cycles. The ship's just sitting in port, anyway. A lazy check isn't going to kill anyone.

2. Between-job delay cut to 2 seconds. This compensates for the increased critical section delay. If we're going to wait it might as well be with an eye on the critical section flag.

***

On the topic of the critical section flag, the 100 ms delay was spamming the debugger log mercilessly, making it appear as if critical section blocking were a horrendous problem. It might become a problem if you had hundreds of ships running FLDS, but I'm fairly convinced I was making a big deal out of nothing.

The changes I made will cause less CPU time to be eaten up checking the critical section flag and eliminate the long delay between jobs, anticipating the "hundreds of ships" scenario in which 10 critical section blocks puts you at 14 seconds of total delay even with the shortened 2 second between-job delay.

***

The new priority scheme isn't perfect, but it's close. For instance, ideally highest priority should automatically go to the most-needed route which fills the ship's cargobay.

However ... the "equal priority to source stock and destination need" formula seems to mostly nail it. I've set up a large loop to push a single freighter to its limit, and it almost always makes smart choices about which cargo to move.

More importantly, a frieghter will more-or-less balance its efficiency to the demand placed on it. We actually want this, as freighter efficiency, which sounds good, is actually a tradeoff with inventory stockpile size.

What you'll notice in practice is that the harder you push your FLDS frieghters, the more "intermediate resources" will build up before delivery. This causes the end-stage fabs to shut down at first. That's the price you pay for using fewer ships.

Now ... about the revision code ... there are a lot of changes, so I'll post them in the next message. I won't distribute files -- only Nazgutek can do that -- but I've PM'd him with the message "come over here and check things out", so maybe we'll get word from the original author soon on whether or not he likes my proposed changes.

[Aye Capn]

The code revisions:

File "plugin.FLDS.Ship.Start.xml"
Change 072:

Code: Select all

072 @ = wait 2000 ms

File "plugin.FLDS.Ship.Main.xml"
Change 067:

Code: Select all

067 @ = wait 1200 ms

REPLACE lines 222-223 with the following:

Code: Select all

222 $Route.Priority = 0
223 $Temp.Ship.Amount.Free = $Route.Available.Amount
224 $Temp.Destination.Amount.Product = $Temp.Ship.Amount.Maximum
225 skip if $Temp.Ship.Amount.Maximum < $Route.Available.Amount
226  $Temp.Destination.Amount.Product = $Route.Available.Amount
227 if $Route.Needed.Amount < $Temp.Destination.Amount.Product
228  $Temp.Ship.Amount.Free = $Route.Needed.Amount
229  $Route.Priority = $Route.Available.Amount * 20 / $Temp.Source.Amount.Maximum
230 end
231 $Temp.Ship.Amount.Free = 10000 * $Temp.Ship.Amount.Free / $Temp.Source.Amount.Maximum
232 $Route.Priority = $Route.Priority + $Temp.Ship.Amount.Free
233 $Temp.Destination.Amount.Product = $Route.Destination -> get max amount of ware $Route.Ware that can be stored in cargo bay
234 $Temp.Ship.Amount.Free = 10000 * $Route.Needed.Amount / $Temp.Destination.Amount.Product
235 $Route.Priority = $Route.Priority + $Temp.Ship.Amount.Free

The variable names I used for temporary calculations are confusing and not self-descriptive. I did this so as to keep the changes as minimal as possible, including not declaring extra variables.

[Aye Capn]

Well you knew I couldn't leave it alone for long.

More tweaks:

1. Lowered the "bad job" threshold to 7000, based on no solid theory other than a gut feel for when my silicon distribution ships should move. 7000 seemed about right.

Code: Select all

216 $Temp.Critical.Limit = 7000

2. Added a 45s shutdown if no job was "good enough" to run. This saves CPU cycles and critical section usage.

Code: Select all

425 else if $Highest.Index < 0
426 @  = wait 45000 ms

That last bit doesn't have enough context. There's an error "if" block right at the end of the delivery module, and that "else if" tacks onto the end of the error "if" block: "[if error ...] else if no work was done, wait 45s". "$Highest.Index" is -1 if no suitable route was found.

I put the code for the 45s wait at the tail end of the function in case Nazgutek wanted to make a "No Suitable Route Found" return value to the command dispatcher module which would in turn handle the delay. That's where the between-job delay is, so it seems natural to have the "no job" delay there as well.

[Nath]

I've been using this script for awhile now, and one ship per loop makes me very happy

Thanks for taking it upon yourself to improve it

[Aye Capn]

No problem. Since I had never seen a line of MSCI code before FLDS Nazgutek's verbose style was a huge benefit.

I actually downloaded FLDS to solve a specific problem with high-yield silicon mines that SDS could only solve with a massive complex of overlapping orders.

FLDS should have solved that problem easily, but strangely it didn't, so I picked at it until it did. The "load balancing" algorithm isn't primarily a tweak to save on ships. It solves a specific problem.

I'm going to make one more change to FLDS: add rejection of invalid cargoes so I can make FLDS work with TP's. I want my Argon Express!

I wish Nazgutek were here. He seems to have left the X2 scripting scene, or at least taken an extended vacation.

[Nath]

I was starting to run into the same problem as you, just not on the same scale. So I'm glad it's fixed

[DeadlyDa]

Great work Aye Capn

I'm looking forward to the "reject invalid cargoes" fix.

[TheHammer]

Aye Capn,
I have been running FLDS since just before your fixes. I even incorporated your fixes (good job). Saturday however, I noticed a problem. All of the loops that were running FLDS were flashing yellow. When I checked, all transports were just sitting somewhere. Even though, some of the factories that they serve needed to be resupplied and there was energy available. I stopped and restarted FLDS without any luck. The only way that I was able to get it to run again was to stop all transports running FLDS then restart everyone. Earier you were say that there was a chance that one transport could hangup transports across the whole universe, is it possible that is what happened to me? Or did I get a random glitch or something?

[Aye Capn]

Yup, that's exactly what's happening, and I've fixed the problem.

Actually I've "postponed" the problem from a dozen ships running FLDS to maybe a hundred.

Cutting the critical section delay still further is possible but involves a lot more work, as I will have to re-write the route-finder to work mostly outside the critical section.

I will post the mod in the next reply. It works by moving a "wait" command from within a double loop to outside the inner loop. (I suppose I could make FLDS even faster by taking that wait out entirely -- since the algorithm is guaranteed to only run once per universe at any given time, cutting out that wait might be acceptable.)

The latest mod also makes FLDS compatible with all dockable ships. DO NOT run FLDS on a swarm of small, fast ships or it WILL go brain-dead even with the critical section fix. Nazgutek probably foresaw this and was his reason for the TS-only restriction.

I'll also bump the "bad route reject threshold" back up to 9000. Coupled with the 45 second "no suitable route found" delay that should ease critical section use by not-so-critical deliveries.

I've made a couple of other minor changes, too, to cut down a little on needless CPU usage. I don't really know how much time the "infinite loop detection" service eats up, but it isn't necessary here so I commented it out.

Edit: While posting the mods I noticed several places I could move "waits" outside of single loops and shave more time off the critical section without having a double-loop Bogart the CPU.

[Aye Capn]

Here are the mods:

Script "setup.plugins.FLDS":

CHANGE line 006 "class" to "Little Ship":

Code: Select all

006 global script map: set: key=COMMAND_FLDS_SHIP, class=Little Ship, race=Player, script='plugin.FLDS.Ship.Start', prio=0

CHANGE lines 020-022 to "Little Ship" & appropriate descriptions:

Code: Select all

020 * Check ship is small enough to dock and trade
021 if not [THIS] -> is of class Little Ship
022  $Message.Body = 'FLDS only works on small ships.'

Script "plugin.FLDS.Ship.Main"

COMMENT OUT "endless loop detect":

Code: Select all

028 * Original script enabled endless loop detect here

INSERT new line 082 and CHANGE line 083 (Yes, "Is.Invalid" really means "valid" here):

Code: Select all

081 $Route.Ware = $Route.Source -> get product ware
082 $Is.Invalid = [THIS] -> can transport ware $Route.Ware
083 if $Route.Ware AND $Is.Invalid

MOVE "wait" from 104 to 107:

Code: Select all

102      append $Route to array $List.Of.Routes
103     end
104    end
105   end
106  end
107 @  = wait 1 ms
108 end

CHANGE line 217 "Bad Route Threshold" from 7000 to 9000:

Code: Select all

214 if $Temp.Destination.Amount.Product > 0 AND $Temp.Destination.Amount.Product < 20
215  $Temp.Critical.Limit = 10000 - 10000 / $Temp.Destination.Amount.Product
216 else
217  $Temp.Critical.Limit = 9000
218 end

Script "plugin.FLDS.Ship.Command.Monitor"

CHANGE line 025 command monitor poll delay from 1000 to 2000:

Code: Select all

025 @  = wait 2000 ms

[simduck]

This sounds fantastic.

I have an idea, is it possible when a transporter gets destroyed to send the player a message asking them to automatically replace the transporter. If they choose yes, then the transporter is replaced and takes on exactly the destroyed ship's data, upgrades, shields, lasers & commands, Every thing. This would mean the player doesn't have to waste time buying another ship, allocating it to a homebase & then starting FLDS?

Or even an option in the hub central that allows the player to automatically replace a destroyed ship with all of the destroyed ships meta data & only a notification to the player that a ship has been destroyed?

Naturally the hub central would have to have the money. If it didn't then it would have to ask the player or take enough money from each linked factory.

If the player owns a trading dock that has all of the required shields & lasers, then maybe the replaced ship can take required goods from there?

Just a thought. It is frustrating having to manually replace ships & get upgrades for them all etc.....

[Aye Capn]

FLDS already makes it pretty darn easy to replace a destroyed ship, and whenever a ship is destroyed it is noted in the log. I don't see much benefit in trying to make it easier.

There does happen to be a script that allows you to order new ships. I think Red Spot wrote it. I've never used it, but it might make replacing FLDS ships easier.

Incidentally, I've stress-tested FLDS and it failed miserably until I cut out a bunch more of those "wait" commands.

"@ = wait 1 ms" is pretty much wishful thinking. Assume "wait 1 ms" will cost you at least 100.

[Burianek]

yep, fyi, no scripts should ever use a wait command of less than 100 ms. It has to do with how the computer distributes computing loads upon arriving at wait commands (and other breaks)
Cheers.

[Red Spot]

There does happen to be a script that allows you to order new ships. I think Red Spot wrote it. I've never used it, but it might make replacing FLDS ships easier.

it could .. and lozing the M1-delivery-system is kinda easy .. (read .. extremly easy .. )
if you'd want it .. I'd have a "no M1" version of the "main"-script for ya in min. ..

G

[Aye Capn]

Thanks for the tip, Burianek.

FLDS is loaded with 1 ms waits. Out they go.

I think it's time we start talking about hosting FLDS+LB. I hate to step on Nazgutek's toes, but he isn't here, there are some issues in the program logic, and the 1 ms waits represent bona fide bugs.

Incidentally I scribbled down a little plan for a new station delivery script, XDS, which calculates routes at a "Transport Dispatch Station" (think FLDS hub) and posts optimal routes as "jobs" for the transports. As long as there are jobs posted the transports fly without pause.

Since running FLDS' route-finder in bullet-time makes it work and doesn't seem to break anything, I don't really NEED to write XDS, so there's a good chance I won't finish it. Just so I'm clear about that ...

P.S. I need a host for Station Orientation and FLDS+LB. Anyone want to host a couple of scripts? Like maybe Red Spot?