Expensive Vanilla Scripts

Vayde · Post by **Vayde** » Tue, 19. Jul 11, 10:24

Litcube wrote:
joelR wrote: is FPS always the best way to test if a game is running better or worse with different scripts or mods?

In a well run game, yes. There are cases where the script engine is skipping instructions, which is what happens in the 0/ms bug. During the incident in particular caused by ammo, it's interesting to note that if you fix your game to use no ammo on account of the 0m/s bug, you will experience slower frame rates in big battles. This is because in massive ship battles in sector, the 0m/s issue is actually causing the script to stall turret and movement instructions, thus letting up on the CPU. Sort of like how you can miraculously cure foot gout by chopping your legs off.

However, in normal circumstances, FPS tracking is the way to go, just like Glen's doing.

Thanks for reporting the plot issues, but I can't test them. I haven't done the plots and never will. This rewrite really shines in a heavier follower world, where each ship is escorted by a whole contingent; namely fully stocked carriers. If I introduce instructions/workarounds/exceptions for a plot sequence, we stray from the scope of this package.

Benchmark or no, when you have 3,000 ships sitting in hangars, each cycling a script every 6 seconds amounting to jack shit gameplay, it's going to be more expensive performance wise than when when those same ships aren't running scripts. Nevermind the additional 3,000 ships that aren't in hangars that also check for new changes when their supposed to be just following a leader.

Where I am right now: I stand by 0.33b.

Good enough for me. Thanks for all the hard work guys.

Halconnen · Post by **Halconnen** » Tue, 19. Jul 11, 19:37

Kadatherion here confirms my issue (top of post with pink pony.): http://forum.egosoft.com/viewtopic.php? ... &start=405

According to the discussion between Esgaro and Kadatherion on the page after, this seems to be linked to ships not moving when being told to follow a stationary target.

If that is the case, it sounds like a doable fix, even without plots.

Mizuchi · Post by **Mizuchi** » Tue, 19. Jul 11, 19:40

Halconnen · Post by **Halconnen** » Tue, 19. Jul 11, 19:46

So do I, in case that wasn't clear.

Some plot missions choking is a minor issue compared to the performance gained.

But it'd still be more convenient if I didn't have to keep swapping files around every time a mission chokes. Primarily because that results in ships with stock script spawning and bogging performance down noticably even after I place stuff back right after finishing the missions in question.

Kadatherion · Post by **Kadatherion** » Tue, 19. Jul 11, 20:02

Halconnen wrote:Kadatherion here confirms my issue (top of post with pink pony.): http://forum.egosoft.com/viewtopic.php? ... &start=405

According to the discussion between Esgaro and Kadatherion on the page after, this seems to be linked to ships not moving when being told to follow a stationary target.

If that is the case, it sounds like a doable fix, even without plots.

Oh, hello, brony passing by

Yeah, so, to summarize things without forcing Litcube to spot a couple of posts inbetween a quite busy thread, we could identify in this script the cause for certain ships stopping during plot missions (as stated there, the baldrics and the springblossom in two terran plot missions. Might be more, but I reverted to vanilla before investigating further. Should also be noted, that game was started with the modified follow script from the beginning, as it was suggested).
We also believe to have noticed some sporadic funkyness when issuing follow commands (Esgaro pointed out it seemed to happen mostly when the target of the follow command is an idling or stationary ship), but that is something I've definitely not tested thoroughly, and I couldn't really say your follow routine is at "fault".

A similar issue I felt could be related to this script, but again not tested enough to be confirmed and reported, was with your missile supply script. When I used it in SRM (which had jobs that enabled it) I kept finding the transports supposed to deliver the missiles to the cap ships stopping dead in space just like those plot ships. I noticed this especially in the Unknown sector north of A148, once a cap ship and bomber escorts are spawned during the Final Fury plot causing the transports to scramble to resupply them. I tried cheat-destroying those supply transports, but once more spawned and came there, the issue struck back shortly after they entered the sector. They were stockpiling around 5-10 km out of the gate, of the 3-4 transports that tried everytime to do their job, not one was ever able to avoid the stopping bug.

I then switched back mid game to the vanilla follow script, and I didn't see issues with those transports anymore, but as I didn't play long before restarting with XRM I couldn't really "prove" this was the culprit. Seems kinda likely, but I wouldn't want to waste your time pointing you in a wrong direction, so take this report with a grain of salt

Halconnen · Post by **Halconnen** » Tue, 19. Jul 11, 20:32

Kadatherion wrote:Oh, hello, brony passing by

You have -no- idea how much I wanted to write "(top of post with picture of Pinkie Pie)" in my post above, but I was afraid Litcube would just get confused.

>:

Litcube · Post by **Litcube** » Sun, 9. Oct 11, 22:27

Update.

Vanilla Rewrite 0.34b:
[ external image ]

- Jack08 has pointed out a bug and recommended a fix, which works. If a ship was set to follow a leader of which the ship was not of the same race, AND the leader had docking ports, the following ship would just sit at 0 and do nothing. This has been fixed, and could be the cause of some of the non-moving ships being reported.

MegaBurn · Post by **MegaBurn** » Mon, 10. Oct 11, 11:19

Litcube, I'm glad this thread popped up, I left the community for ~8 months and hadn't gotten around to picking through the backlog of S&M threads to mid january. Its kind of disturbing to think of how much stuff like this gets overlooked or forgotten if its not linked in a stickied thread... Anyway, few questions:

Back on page six you posted a list of your scripts (little long to quote), I'm only aware of about ten of those, though some others ring a bell, and some look like they are related to this Vanilla Rewrite project. Do you have a thread or page somewhere listing everything with details?

This project looks impressive, are there any other related threads or more to come?

I noticed most of your scripts are not on your download site and, from what I can tell, most are not bundled with other scripts/mods. How many of those scripts were not released due to the headaches of "publishing" stuff in this community? (sorry if its a loaded question, goes to the reasons why I left for a while)

Litcube · Post by **Litcube** » Mon, 10. Oct 11, 23:04

MegaBurn,

You're right. It's somewhat of a pain in the ass to publish a package. Not only do you have to isolate a package to run independent of any other package you currently, run, but the following question drives me nuts:

"Is this compatible with XYZ"?

When I get asked that, it makes me sad in my head.

This particular script will be updated. And if I do rewrites in the future it'll be included in this package. There's more optimizing to do with vanilla scripts.

MegaBurn · Post by **MegaBurn** » Tue, 11. Oct 11, 09:36

Yeah, lack of community infrastructure to ease such burdens, its turned into a sort of pet peeve of mine, but I digress...

Well, I'll happily test whatever you release against the 100+ script/mod build I'm working on, should have it ready in about a week.

joelR · Post by **joelR** » Mon, 19. Dec 11, 04:34

Litcube. Im usin this in AP and havent seen anything bad happen so far. Have you had a chance to check out AP yet with this?

I was in Treasure chest before I added this and could hardly move. After running this on the same save FPS shot through the roof. May be a coincidence, maybe not.

Litcube · Post by **Litcube** » Mon, 19. Dec 11, 05:45

Not yet. I'll be doing something with my scripts once I learn more about AP.

glenmcd · Post by **glenmcd** » Mon, 19. Dec 11, 11:54

joelR wrote:Litcube. Im usin this in AP and havent seen anything bad happen so far. Have you had a chance to check out AP yet with this?

I was in Treasure chest before I added this and could hardly move. After running this on the same save FPS shot through the roof. May be a coincidence, maybe not.

All the time while ANY game is running, it is fragmenting memory. Some games do it very slowly. TC (and I assume AP) does it relatively fast. Whenever you reload a saved game, all of the memory that was used to store the various things in the universe is freed, and then re-allocated as it re-creates the universe. In a reload, it all happens in one continous stream with virtually no freeing, thus avoiding/removing most of the fragmentation. In a mod I posted here a week ago (Fission Ships) I actually force the player to reload every so often as I found that when there are enemies all around you, memory fragmentation is greatly increased. In the early versions, I was experiencing double to triple the framerates after a reload if I hadn't reloaded in an hour or so. Since then I've recoded a few things to avoid this as much as possible. Measuring performance increases from script replacements is possible but not as straight forward in TC/AP as some may initially think. In TC or AP try buzzing around in a Xenon sector in a kestrel and dodging their missiles and bullets for an extended time. On some machines it's enough to cause a CTD. To look deeper into this issue if you are running multiple monitors, run "Resource monitor" (Win7) and look at the memory page while you're playing TC or AP. Even while just farting around in a fairly empty sector, the amount of memory in use by X3TC/X3AP will edge its way up. Enter a ware zone and it will happen faster. While Litcubes follow replacement may be significant, it may not actually be measurable simply because there's not a stable testing environment to be found here.

You've mentioned before joelR that you are using a ton of scripts/mods. A fair percentage of 3rd party scripts/mods are not written in a way that avoids memory fragmentation, and thus we're lead to believe that the more mods you use the slower your game gets. It can be true but if each were written with MF in mind, I don't believe that it would be noticed very much, apart from when one of them scans the universe for ships and/or stations (Universal Best Buys being a good example).

Jack08 · Post by **Jack08** » Mon, 19. Dec 11, 12:42

"Memory Fragmentation" isnt going to cause a significant slowdown by itself, it will only result in areas of memory being un-re-useable

The reason fragmentation is so performence degrading on media like a hard disk drive is because of the time it takes for the read/write heads to seek to the next sector, with ram there are no heads, and you can adress any location on the chip as fast as any other location

There is a issue with X3's memory management on some systems, witch causes it to fail to unload resources (most noticable on 64bit systems with between 2 and 4 gb's of ram).

Scripts use only a very small amount of memory (Test case: an infinate loop creating and storing arrays of 10 ships caused an increse of 1kb every 1-2 secconds, witch is an insane amount of data considering it has no wait timers)

And, X3's core game logic is run in a virtual stack machine, witch more then likely pre-allocates enough space for the game's core & scripts to run without issues, the only probelm area is meshes & textures (textures more then meshes)

glenmcd · Post by **glenmcd** » Mon, 19. Dec 11, 15:04

Jack08 wrote:"Memory Fragmentation" isnt going to cause a significant slowdown by itself, it will only result in areas of memory being un-re-useable

Memory fragmentation as I understand it is both a direct and indirect cause of slowdowns. Direct is rather complex and involves the algorithms used to determine which memory pool, a block should be allocated from. If the OS has to look through a million memory pools to find a suitable one to allocate from, it's going to take ten times as long as it would had it needed to look through only 100,000 memory pools. If the OS just looks at the largest blocks first, this exacerbates the MF problem by reducing the size of the largest chunks needlessly. If the smallest pools are checked first, then its highly dependant on how fragmented memory is as to how many too-small pools will need to be checked before finding a suitable one. No matter which algorithm is used, it always results in longer searches when there are more items to attempt to match to. To learn more about how memory fragmentation can significantly slow down computer systems, check out the wiki:
http://en.wikipedia.org/wiki/Memory_management
I was actually coding memory management software (in assembler) decades ago. I'm not familiar with the exact algorithms used in Windows 7 now, but judging by the Wiki the challenges faced haven't changed all that much. Yes, CPUs and memory speeds are much higher now, but then mem allocs and frees are happening more times per second also. I once profiled a computer system, timing every call to the OS from all running apps. Memory allocation of blocks slightly larger than the minimum (8 bytes at the time) was #1.

Jack08 wrote:it will only result in areas of memory being un-re-useable

This is misleading. In all memory management systems, there is a minimum allocatable (and thus free-able) block/chunk/whatever size. This is also the smallest size block/chunk/size that ends up in the middle of an otherwise large block. In the systems that I've worked on, there are more minumum size memory allocations than any other size. Thus, every part of memory can be allocated, provided that there are sufficient small allocation requests. The problem comes about with the slightly larger requests, that happen regularly but can't quite be satisfied by the hundreds/thousands/millions of small memory pools that may have to be scanned somehow, again depending on the actual memory management algorithm used. The very large alloc requests happen far less often and so don't figure so much.

----

The indirect slowdown results from more hard faults per second happening when the page file (or whatever it's called on your particular OS) is accessed more often.

Jack08 wrote:the only probelm area is meshes & textures (textures more then meshes)

MF is also caused by the very same things that most modders are encouraged to use - pauses. It's not the pause itself, but the opportunity it presents to add to MF when allocs are interleaved across different scripts. Allocating is no problem, but the resulting freeing being almost certain to not be done in reverse order of allocation, adds to MF.

Jack08, have you tried not saving for a few hours but otherwise actively playing/fighting and then saving/reloading to see what happens to your framerate? Have you tried using Resource monitor (memory page) + fraps while playing X3 with 4GB+ ? Other than the "last reload" instruction, scripts can't tell there has been any reloads at all, or whether the player has just done 100 reloads one after another. Something is happening on reload and in my opinion it's removal of memory fragmentation. Please share your research results with us, when you have some.

Jack08 · Post by **Jack08** » Mon, 19. Dec 11, 15:32

Ive been running a test script lateley to test the stability of my scripting, the script spesificly create and destroys a massive amount of objects acorss a few sectors, with arrays, variables and signale, this creates a massive ammount of memory freeing/injecting and ive had it running (today) for about 3-5 hours

Though this whole time my framerate has remained constant, i stopped the script and my framerate is still as high as when i began 3 hours prior.

were talking hundreds, if not thousands of objects per minute here, both in sector and out, loading and unloading.

Ive seen none of the effects that you have descibed that occur over time, and this test script should have accelerated them.

All of this while about 3 diffrent AL scripts were creating battles in sector and out - My game is extreamly overactive because of the testing im running for XTimelines

Litcube · Post by **Litcube** » Fri, 9. Mar 12, 16:16

Update.

Vanilla Rewrite 0.35b:
[ external image ]

- Fixed: If a ship was set to follow a carrier that was full, the ship could be stuck.

joelR · Post by **joelR** » Sat, 10. Mar 12, 00:58

Thanks Litcube. Im glad youre still working on this.

Litcube · Post by **Litcube** » Tue, 13. Mar 12, 14:44

Update.

Vanilla Rewrite 0.36b:
[ external image ]

- Fixed: In some cases, if wing ships were created within attack range of an enemy, the lead monitor would interrupt normal process. A small delay added to the lead monitor to remedy this. (Seen with traders ejecting fight drones, for example).

Mad_CatMk2 · Post by **Mad_CatMk2** » Wed, 14. Mar 12, 04:06

Litcube wrote:Update.

Vanilla Rewrite 0.36b:
[ external image ]

- Fixed: In some cases, if wing ships were created within attack range of an enemy, the lead monitor would interrupt normal process. A small delay added to the lead monitor to remedy this. (Seen with traders ejecting fight drones, for example).

I would assume that running this in AP is on a 'as is' basis?

Although I would tend to think that the scripts you're modifying would make a positive impact in both TC and AP. Great job otherwise!