many script instances slow down SETA (spaceflies, Anarkis/Pirate Guild)

The place to discuss scripting and game modifications for X³: Terran Conflict and X³: Albion Prelude.

Moderators: Moderators for English X Forum, Scripting / Modding Moderators

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Wed, 20. Jun 18, 00:50

Aldebaran_Prime wrote:But if somebody has some code snipplets for me to do it right, I will try it again.
I looked around and found a couple stock scripts that restart themselves on version change:
!job.special.tpliners
!station.shieldstock.terran

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Wed, 20. Jun 18, 06:15

I took another look at this, now that I have a bit more experience with the obj assembly. After running through a few ideas, one notable aspect of the spacefly scripts is they appear to be the only ones that use the 'is disabled' command (based on a quick grepping). So, I edited the jump table in the script execution code to go to the 'destruct' code when 'is disabled' is called.

In testing, it was very fun to see the 3k+ spacefly scripts (on a 10 day old game) die off in chunks. However, the list of save games got messed up somehow and appears to be listing the directory contents from when I made the save months ago, so something weird got messed up. Other bad news is that I didn't notice any notable framerate boost after the spacefly deaths.

Swapping to the Return command (instead of Destruct) was tried, but only works on idle spaceflies since the followers are running escort commands (and so never hit the 'is disabled' line).

Maybe it is unsafe to call destruct on [THIS] in a script? At any rate, I can play around a bit more and see if a solution works without side effects.

Cycrow
Moderator (Script&Mod)
Moderator (Script&Mod)
Posts: 22201
Joined: Sun, 14. Nov 04, 23:26
x4

Post by Cycrow » Wed, 20. Jun 18, 10:34

the "is disabled" command is mainly for when spaceflys have been stunned and can be collected

User avatar
Aldebaran_Prime
Posts: 1386
Joined: Sat, 20. Feb 10, 17:47
x4

Post by Aldebaran_Prime » Wed, 20. Jun 18, 14:13

SirNukes wrote:...
However, the list of save games got messed up somehow and appears to be listing the directory contents from when I made the save months ago, so something weird got messed up.
Yes, I recognized this effect, too. But just the actual savegame isn't shown anymore
SirNukes wrote:...
Other bad news is that I didn't notice any notable framerate boost after the spacefly deaths...
I got between 5 an 10 FPS more, if I deleted 12.000 Spaceflys
SirNukes wrote:...
Maybe it is unsafe to call destruct on [THIS] in a script? At any rate, I can play around a bit more and see if a solution works without side effects.
yes, the side effect, that the current savegane isn't shown anymore is strange - I' m curious, if you'll find something. In the meantime I'll try the correct script-update/restart mechanism

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Thu, 21. Jun 18, 02:19

Aldebaran_Prime wrote: I got between 5 an 10 FPS more, if I deleted 12.000 Spaceflys
This motivated me to recheck, this time measuring practical SETA speedup (game minutes passed during a couple real minutes).

Tested save:
XRM, 10 days in, 134 factories, 200 ships.
15k scripts active, 3.3k of them spaceflies (241 idle, rest followers).
Testing in Harmony of Perpetuity, far in the lower left pointing at the planet.
SETA set to 5000% (my normal 1500% was capping out) for 2 real minutes, starting from a new game minute.

With spacefly scripts running: 9 game minutes elapsed (450%).
With spacefly scripts killed off: 59 game minutes elapsed (2950%).

6.5x speedup from killing the spaceflies. Wow.

Here I was thinking your problem was for people with 10 year old saves, so I'm a little shocked at the results.


In other news, I tried a couple more tricks, refining the assembly edit to do a proper SelfDestruct inside the "is disabled" command handling code, followed by either passing to the "return" or "is docked" commands for cleanup (the latter fills in return args for the following conditional branch). These had the same effect as redirecting the jump table, though, with the saved game list getting messed up. If I only redirect to the "return" code this problem doesn't crop up.

There are vanilla scripts that call "[THIS]->destruct:..." followed by a return, for what that's worth. Maybe I need to look into where the saved game list is built to get an idea of what is going on.

But, if I patch the code and destroy the spaceflies, save the game into a known okay spot, remove the code patch, and reload, things seemed to be working fine again. I am just worried that there might be a subtle bug introduced somewhere.

Cycrow
Moderator (Script&Mod)
Moderator (Script&Mod)
Posts: 22201
Joined: Sun, 14. Nov 04, 23:26
x4

Post by Cycrow » Thu, 21. Jun 18, 10:34

SirNukes wrote: There are vanilla scripts that call "[THIS]->destruct:..." followed by a return, for what that's worth. Maybe I need to look into where the saved game list is built to get an idea of what is going on.
Generally anything after the destruct command will be ignored, the script editor does automatically add return statements at the end though, so all scripts have a return, even if they never get to it

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Fri, 22. Jun 18, 03:27

Thanks, good to know about the script engine.

I tried a couple other code edits, using JumpOutOfExistence instead of SelfDestruct to save on bytes and be able to slot in a spacefly class code check (in case "is disabled" was being called on anything else somehow), but with the same outcome (dead spaceflies, old save game list).

From those tests and Cycrow's tips, I figured the problem probably isn't me, it's somewhere else. For some reason, the save/load system seems to be making spaceflies and then not liking it when those spaceflies get killed off.

That led to discovering that new spacefly swarms are created whenever I load a game. Over multiple saves and loads, they accumulate.


Rolling the game back to plain vanilla doesn't see this problem. Further, applying my spacefly-killer patch doesn't seem to cause the vanilla save game list to go stale, either.

Testing some select mods, I have found so far that Improved Races 2.0 and probably XRM itself spawn these accumulating swarms. My main XRM game makes 2 swarms per load with IR, 1 swarm per load without IR, where the XRM swarm is added during that 1-2 second hiccup XRM has a few seconds after loading a game (running some delayed setup scripts, I guess).

It would be curious to track down where those mods are messing up, and if the problem is limited to spaceflies or if they are littering the null sector with other ships too.

User avatar
Aldebaran_Prime
Posts: 1386
Joined: Sat, 20. Feb 10, 17:47
x4

Post by Aldebaran_Prime » Fri, 22. Jun 18, 12:18

I just tested the "[THIS]-> destruct: show no explosion=0" instead of the "...destroy..." it works, too, but with the same side-effect on showing savegames.

I also use the Improved Races 2.0 in my MOD/Savegame - I try to have a look on this scripts, too.

I searched yesterday longer for a solution of the still non-deletable spaceflys. I found on page 72 of the MSCIHandbook the box "Technical Tidbit" - there is written, that each object should have a ID which is linked to the sector. Therefore the player-ship needs to be in this sector. This can be done by a loop and automatically placing the player ship in all existing sectors and then looping throug all sector objects.
And in this way, also the Zombie-spaceflys may be found.
I also found a old thread of a X2 SpaceFly detector, which uses this method and where are some code snipplets can be taken from:
https://forum.egosoft.com/viewtopic.php?p=887928#887928

Code: Select all

$this.sector = [THIS]-> get sector
$player.sector = [PLAYERSHIP]-> get sector

if $this.sector != $player.sector
return null
end

$sectorID = [PLAYERSHIP]-> get SectorObject ID

$flies = array alloc: size=0

while [TRUE]
dec $sectorID
$object = get object from SectorObject $sectorID
$class = $object-> get object class

* Break out of the loop if you encounter an existng object that is NOT a Space Fly
if $object != null AND $class != [Raumfliege]
break
end

if $object-> exists
append $object to array $flies
end

end

$size = size of array $flies

* Return null if no flies where found.
if $size == 0
return null
end

return $flies
but
$sectorID = [PLAYERSHIP]-> get SectorObject ID

and
$object = get object from SectorObject $sectorID

Does not work anymore in the X3 script engine
Are there newer functions, which do the same?
Last edited by Aldebaran_Prime on Fri, 22. Jun 18, 13:47, edited 1 time in total.

Cycrow
Moderator (Script&Mod)
Moderator (Script&Mod)
Posts: 22201
Joined: Sun, 14. Nov 04, 23:26
x4

Post by Cycrow » Fri, 22. Jun 18, 12:45

The get SectorObject ID command was replaced with, is in active sector.

The other sector object commands were removed due to issues with using those commands. They more often than not broke the game

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Fri, 22. Jun 18, 21:11

Well, I am happy to report that after looking into the problematic mods, things fell into place pretty quickly and the rest of my guesses were on spot.

In short, the fundamental problem is either bad documentation or bad behavior of the "create ship" command (depending on which angle you look at it). When a spacefly is requested, the script engine redirects from Create to CreateSwarm, so a whole flock of spaceflies are created and their scripts started. The documentation doesn't mention this, and modders do not expect this behavior.
X2-Illuminatus wrote:What could prove difficult is when the spaceflies were created in sector null. Some scripts use this area to create objects in a safe place, before they are used later, or to read their properties upfront for which you need the actual object. If, for example, a script was looping through all TShips entries and forgot to clean up afterwards, this may explain why there are Spaceflies you cannot find in the normal universe.
This is spot on, except that the mods do not exactly forget to clean up. They destroy the single spacefly they meant to create, but not the unexpected extra swarm.


Anyway, after noticing this and figuring the bad saved games list was due to a mod's spacefly getting killed off before it was finished analyzing it, I tweaked the "is disabled" autokill to have a 10 second delay. This allows the mod to do its business, avoiding the saved games list problem. It still doesn't make sense that a script trying to get information on a dead spacefly will mess up the saved games list, but maybe that can be written off as flakiness in the X3 engine.

To prevent new accumulation of spaceflies, I also edited the "create ship" command to behave the way modders would expect, making only a single spacefly with no running script. With this change, Improved Races and XRM no longer leave excess spaceflies lying around. This should also work for any other mod out there with the same problem (in vanilla, XRM, or LU).


These fixes are available in my customization tool (see sig), v3.5, under Kill_Spaceflies and Prevent_Accidental_Spacefly_Swarms. I will update that thread after adding a little polish (3.5.1 with documentation cleanup and such), but github was updated last night with the main changes if anyone wants to try it out.

The MSCI documentation and threads for affected mods should probably be updated, if somebody wants to take care of that. I feel spammy doing it myself.

User avatar
Aldebaran_Prime
Posts: 1386
Joined: Sat, 20. Feb 10, 17:47
x4

Post by Aldebaran_Prime » Sat, 23. Jun 18, 00:52

SirNukes wrote:...
In short, the fundamental problem is either bad documentation or bad behavior of the "create ship" command (depending on which angle you look at it). When a spacefly is requested, the script engine redirects from Create to CreateSwarm, so a whole flock of spaceflies are created and their scripts started. The documentation doesn't mention this, and modders do not expect this behavior.


... They destroy the single spacefly they meant to create, but not the unexpected extra swarm.


....
Very good, that you find a potential reason for the mess of spaceflys!
But maybe you can give us more details of the scripts and places of problematic code in this scripts/mods?
It sounds like, that the spaceflys are created as helper beacons in these cases - so alternatively to modify the create ship behavior in general, it may be another approach to modify the problematic code in these mods...

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Sat, 23. Jun 18, 12:13

Improved Races appears to have its main problem in xtl.lib.ship.choose.xml around line 49, though there are a couple other places it creates ships in the null sector that I didn't look at closely.

Grepping around also found some seemingly problematic code in the MARS setup.plugin.gz.missile.def.mk2.xml file, though that script appears to deactivate itself in AP (even though it is packaged with a version of Mars that works with AP).

I must apologize for thinking XRM was to blame for my second spacefly swarm; after applying some better care, it is actually Salvage Command Software that is the problem. Looking through the code, I can see potential spots where the bug might be cropping up, but there is no obvious 'loop over tships' code like in the above two mods. At any rate, SCS causes a hiccup shortly after loading a save that will pop out a swarm.


As for the style of the fix, my own preference is a clever catch-all solution. Finding and editing every script with this problem feels clumsy, both in carrying it out and sharing the edits. My gut feeling is that no one has ever used "create ship" to purposefully make a spacefly swarm, given the lack of of documentation of this behavior and lack of general interest in spacefly mods.

User avatar
Aldebaran_Prime
Posts: 1386
Joined: Sat, 20. Feb 10, 17:47
x4

Post by Aldebaran_Prime » Sat, 23. Jun 18, 23:26

Thank you! Yes, I found this:

Code: Select all

...
$WareSubType = get number of subtypes of maintype 7
while $WareSubType
dec $WareSubType
$cShipWare = get ware from maintype 7 and subtype $WareSubType
gosub CheckType:
if $EvalRet == [FALSE]
continue
end

$SpawnedShip = create ship: type=$cShipWare owner=[Unbekannt] addto=null x=0 y=0 z=0
...
I can see, that also spaceflys will created in sector NULL in this case and I believe you, that not only a singel Spacefly, but a swarm will be created, but not deleted afterwards.

additionally to patch the create ship command in the Egosoft obj-files, an other approach can be, in this special case, to correct the code of this IR2.0 script not to to create a spacefly in this loop. (I'm really impressed maximal, what knowlege you have! I will try your X3 Customizer, too)

I found, that the subroutine "CheckType:", which is called before the create ship statement, checks for several shiptypes and then not creates a ship in sector NULL:

Code: Select all

CheckType:
$EvalRet = [FALSE]
$eval = $cShipWare
if $eval == {Argonen Truelight Seeker}
endsub
else if $eval == {Terraner #deca.deaf}
endsub
else if $eval == {Terraner #deca.cefa}
endsub
I just would add a

Code: Select all

else if $eval == {Unbekannt Raumfliegen}
endsub
there and the problem is fixed, too.

- or do I think wrong?

If I'm right, we may post this research in the thread of IR2.0, too, to prevent other players to run into this problem.

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Sun, 24. Jun 18, 01:19

I agree, your fix would slot very nicely into that IR code.

There is another case of creating/destroying ships in xtl.xtlc.comm.order.checkout.xml, but it looks like that might only ever handle ships the player can order from a shipyard, so I'm thinking that won't ever make spaceflies.

Salvage Command Software took a while to track down the culprit, but it looks like it is in scs.library.xml around line 2618. (Annoyingly, grepWin wasn't finding this with "create.*ship.*addto.*null", even though notepad++ matches just fine. I ended up finding it by just reading through the code starting from setup.)

Side note: early on I looked into Zanzal's Ship Browser, since I knew that looped over ships and made them to record information. The interesting code is in plugin.zanzal.shipbrowser.menu.xml around line 422. That mod has no spacefly problem because it will pull the class of the ship subtype and compare it to the desired class (eg. M4) before creating it. Spaceflies have their own special class, and so are always skipped.

SCS, on the other hand, appears to think the ship has to be made first to be able to check its class, and so ends up making everything, spaceflies included. It does do some name checks for filtering out unwanted ships, so spaceflies could be added there. Though, now that I look at it, that name check (eg. "$tmp = find position of pattern 'Beacon' in $shp") wouldn't work on non-English versions, would it?

Cycrow
Moderator (Script&Mod)
Moderator (Script&Mod)
Posts: 22201
Joined: Sun, 14. Nov 04, 23:26
x4

Post by Cycrow » Sun, 24. Jun 18, 16:16

SirNukes wrote: SCS, on the other hand, appears to think the ship has to be made first to be able to check its class, and so ends up making everything, spaceflies included. It does do some name checks for filtering out unwanted ships, so spaceflies could be added there. Though, now that I look at it, that name check (eg. "$tmp = find position of pattern 'Beacon' in $shp") wouldn't work on non-English versions, would it?
my guess on this is that SCS is an older script, and was probably made before the command to check a types class was added.

alot of older scripts suffer from problems like this where they have to use hacks or workarounds due to the available commands not doing the job.

The cheats scripts is full of them as it was made before alot of the commands in TC were added (was made well before the release of TC).

the create ship command does have special handling of spaceflies where a swarm is automatically created, but only the "leader" is returned. And as there is no way to get spaceflies then the script cant destruct them

SirNukes
Posts: 546
Joined: Sat, 31. Mar 07, 23:44
x4

Post by SirNukes » Sun, 24. Jun 18, 19:36

Cycrow wrote:my guess on this is that SCS is an older script, and was probably made before the command to check a types class was added.
Ah, thanks for pointing that out. My apologies for throwing some shade on these older scripts.

User avatar
N8M4R3
Moderator (Script&Mod)
Moderator (Script&Mod)
Posts: 392
Joined: Fri, 24. Nov 06, 15:48
x4

Post by N8M4R3 » Sun, 24. Jun 18, 20:47

Hello Guys,

unfortunately, I can not contribute to the help from too little experience, but I am very happy to see and follow with the fact that here known and experienced people deal with the problem! :thumb_up: :thumb_up: :thumb_up: :)
Neue Erweiterung für X3 verfügbar: Farnham's Legacy | +Optional: weitere Verbesserungen im inoffiziellen Patch v1.3.14 *** Modified*** :khaak: :thumb_up:
Diese Woche im Angebot: HUD-GUI-Mix (FL) | Text-DB 0001-L049 (FL) | Textkorrekturen & Verbesserungen (FL)
Weitere Veröffentlichungen hier: N8workX
Nützliches Tool für nicht mehr vorhandene Downloads: web.archive.org
Externes Archiv für MOD/SCR Ressourcen: xdownloads.co.uk | code.google.com/archive/p/x3tcscripts/

User avatar
The Captain
Posts: 169
Joined: Tue, 7. Sep 10, 18:18

Re: many script instances slow down SETA (spaceflies, Anarkis/Pirate Guild)

Post by The Captain » Thu, 28. Oct 21, 04:07

I seem to have run into a similar issue as described in the first post in this thread, but I couldn't for the life of me follow what steps were taken to remedy the issue in the replies that were posted afterwards.

I did notice that one script I am running seems to have way higher numbers than all of the others (Pirate Guild), and it was soon after installing that script that I started to notice the slowdown. I've also noticed that if I engage SETA, it will run as expected for a random amount of seconds, then slow right back to what looks like normal operation until I cancel it. I can start it again immediately and it will speed up again, but continually tapping the j key is a little tedious.

Is there a fix for this?

Post Reply

Return to “X³: Terran Conflict / Albion Prelude - Scripts and Modding”