Memory increses over time, Leak? New Game 6.0

[Metran]

If you have lots of Satelites scattered all over Windfall sectors you might want to try deactivating them when they're not needed - main reason for this is that VIG, the only faction that does this, has a tendency to have many dusins of fighters accompanying their destroyers on patrol, which only adds to the amount of objects that the game have to keep track of. This way you can sort of control the stress that's put on the CPU.

[badbane]

CPU is not being stressed at all. I don't have any framerate drops or anything. The only issue is committed memory. I mean I have to give EGO credit there there a ton of ships and my cpu isn't even stressing. I only have sattelites near stations in the early game until I can buy all of the trade licenses. Now laser turrets. There's a lot of them from the miners. It's how I countered the Kha'ak in mining sectors. The ships tend to mine in the same spaces so there is probly a hundred or more laser towers floating around.

[hardergamer]

It's nice to see my thread is still helpful to some, and back unlocked.

But sadly I was looked out of my own thread and had my most important posts removed as I made what seems a simple spelling mistake as I'm dyslexic and have learning difficulties, but made compete sense to others but not to a certain mod here, who locked me out of my own thread until now, after complaining I need my info in my posts, I got a link to some of my posts, but many of the most important posts in this thread which I was supplying to "Egosofts izalith" are missing, and most of are corresponding messages have been removed also, (about 15-18 msg) but as this memory leak is still ongoing with many talking about it on Steam, which is much more active than this forum, but mainly for uninformed players, I would like to see it fixed.

[Alan Phipps]

Do you mean this with just the two posts replying to other player suggestions, and both with messed up quotes such that your replies do not make any sense in relation to the quotes as given?

That is all that was removed and neither are to do with your reply to Izalith anyway. This main thread of yours was not locked at any time throughout - as you can plainly see by the continuing date progression of posts in it.

[badbane]

Just checking in to see if there is anything I can try in game to help you guys out and test?

[badbane]

Even with the last beta the game is unplayable if I jump into ships and actually fly around. I completed the plot for avarice and decided to try to capture the astrid. I queued up a nemesis to go after it and after less than an hour the memory was full. As much as I enjoy managing stations and such if I jump into a ship and try to do anything the paytime drops to like twenty to thirty minutes Unless I let the game fill my hard drive. I am on the same save game. I am on 6.20 patch 4. No mods nothing. I was better for a bit however it's still eating up 24 gb of memory. The actual memory utilization is still hovering around 5-6 gb.

[badbane]

I just relaunched the game and played it only ran for 15 minutes before it filled up 24 gb of memory.

[Daemonjax]

I'd like to see some screenshots showing _available memory_ being reduced that much, because it sounds like you guys are looking at _free_ memory (which includes the file cache... a better name for it would have been "unused memory").

On Win7 I get the exact opposite problem -- the game aggressively tries to use as little file cache as possible, even though I have 32gb ram installed -- leaving ~16gb "free" (so not used as cache). I'm loading up 16gb of the game into a ramdisk as a workaround.

I never seen the x4.exe process commit more than ~8gb (process starts at ~6gb). When I close and re-open the game it's back to ~6gb commit size.

I'm guessing you guys are all on win10, and so file caching in this game is (or at least WAS) working properly on that OS!?

[stevecs]

Well the ~8GB is not true for working memory as can be seen on my system here at more than double that (19GB). Even committed memory is of ~30GB is insane. I've seen committed leak up to 50+GB (I have 128GB of ram on this system currently).

https://postimg.cc/mzvZJgc4

This is process explorer against 6.10. There is definitely a leak here or game is NOT freeing up memory correctly.

[Scoob]

That's really weird. While the game certainly does bloat a little over a longer play session, this is fine as I have 32GB RAM and I'm happy for the game to use whatever it needs for a smooth experience. In my modified game, which of course is higher on RAM usage, I'll see a peak of about 20GB allocated to the game over a long session. Not had a memory-related crash for ages, and those weren't down to the game.

- There was a Windows 10 bug where the it was NOT freeing up Standby memory. I.e. I'd play modified Skyrim, then move to X4 but there was "not enough RAM" (out of memory) to run it. I found that my RAM was FULL of Skyrim data still in Standby, and Window was NOT freeing up this RAM as it was supposed to. This led to other issues. I temporarily used an app to force-free Standby memory, it's called "RAM Map".

- There is a Windows bug (but it doesn't seem as bad now) where X4 would page out as soon as it hit about 8GB used. Not a problem as such, except it was paging out data NEEDED for the current scene - leading to massive stuttering, literal seconds per frame, issues. To prove my point I DISABLED my swap file (as a test) and while Process Explorer still thought the game was paging stuff out, it was of course "paging" to RAM, as there was no page file. Game ran GREAT. Zero issues. However, other things want a page file (even if they don't strictly need it) and throw a hissy fit if they don't get one. The solution? Run X4 .exe has HIGH priority, makes it keep stuff in RAM more aggressively. I currently run with just a 256MB page file - prevents hissy fits - and X4.exe running as High Priority. Game runs great.

Why your game would be sucking up RAM like crazy is beyond me, but perhaps it's some weird Windows issue? The devs seem to have less control of the game's RAM use than you'd think. Windows is in charge.

Note: I'm running both a modified v610 game, and a totally vanilla v6.20 Beta 6 game, and have had no issues with RAM usage.

[stevecs]

I see this both under windows 10 & windows server 2022 as the OS.

As for swap/pagefile I have never used them on any of my builds I always add more memory instead (this is going back to the 1990's). Only used swap on older unix systems from the 70's and 80's when you just couldn't get enough ram at any cost to fit into a system.

And for sucking up ram like crazy it's an application issue, so would really be under the dev control. either they are NOT telling the OS to free unused memory or they are requesting more as in a leak. I haven't had a crash yet always caught it before it ballooned to 128GB. And am going to upgrade anyway to 256 or 512GB ram so would be interested to see if it grows beyond 128GB which used to be the limit in non-paged space as I believe W10 still limits to 128GB. or grows beyond that (windows server 2012R2 or later goes to into the TB range).

I haven't directly tied it but it appears it may have some relation to the act of flying through/going into/ new sectors. So perhaps not freeing up the information when leaving the previous sectors after a while?

[Daemonjax]

Well the ~8GB is not true for working memory as can be seen on my system here at more than double that (19GB). Even committed memory is of ~30GB is insane. I've seen committed leak up to 50+GB (I have 128GB of ram on this system currently).

https://postimg.cc/mzvZJgc4

This is process explorer against 6.10. There is definitely a leak here or game is NOT freeing up memory correctly.

Thanks for the screenshot.

Yeah, that's weird. I get nothing like that happening... on win7 tho. I get the opposite issue -- the game doesn't use enough ram because it keeps aggressively clearing my file cache... So I have most of the game files on a 16gb ramdisk.

[badbane]

Yea the game is unplayable on my current save and I have too much time invested to just start over for me anyway. They did apply a patch for 6.2 3 i think after that was applied. I initially got several hours of gameplay before I hit my artificial limit to prevent os corruption. IE my hard drive completely filling up and system file wirtes being afftected. I have 24 GB of virtual memory hard coded. I still think it is related to the new physics engine. I had to fly through a sector with VIG who has hundreds of ships right now. Doing it for the story plot. If I sit static all is well but the second I start flying around the gameplay goes from an hour to as little as fifteen minutes. Especially if I end up in my HQ sector or in that Vig sector. Ego has my save game where it is currently happening I'll submit it again as I am on the latest beta now to see if things improve. The game is completely vanilla too no mods.

Definitetly try the game on 6.20 patch 3 if you want to see what's happening after the "fix" was applied.

[badbane]

Tried again with the latest patch. The game session only ran for roughly 15 minutes before filling 24 GB of memory.

[stevecs]

Tried again with the latest patch. The game session only ran for roughly 15 minutes before filling 24 GB of memory.

Are you reporting this to the Dev's in the devnet channel? I'm not participating in the beta (can't stand the game vanilla) but from the beta log it *appears* they think they've solved the memory issue in 6.2 beta 2 so if it's still there then it's not been addressed.

[badbane]

I opened a support case. The issue was resolved briefly. Things got better for like twenty hours. I am pretty certain it has to do with entities in a sector. VIG spammed out over 100 ships and if fly through that sector I go from like 6 gb to 21 gb used in around fifteen minutes. They may have fixed something but there obviously something else going on.

[Daemonjax]

I just completed building a newish (10700k, 32gb ram) win10 gaming pc, and I get the same issue I had with win7 on my old pc (i5-3570k, 32gb ram) :

The game isn't making use of all my ram (32gb). The game aggressively clears whatever's in my filecache, constantly, leaving 16gb+ completely unused (free).

Except on win10 this game is even more aggressive -- I've never seen more than 2gb filecache used after a long gaming session... WAY too much free memory (like 22gb!). Free memory -- I'm not talking about Available memory.

If I try to circumvent the game's behavior by alt-tabbing (at any point after the game loads) and then cache ~16gb of the game's data files with my script, everything seems fine -- the data is cached as expected... until I alt-tab back into the game. It then aggressively wipes the filecache AGAIN back down to ~2gb.

To make full use of the ram I have, I'd need to make a ~16gb ramdisk to hold as much of the game's data as I can -- in ALL other games I can simply run a script to pre-cache whatever I want in the OS's filecache before running the game -- think of it as superfetch that isn't stupid. Sure, the ramdisk is more performant on the first read but pre-caching is way less fuss (double-click a game-specific vbs script -- it just works(tm)) and I don't get bad feelings about windows double-caching the data on the ramdisk. When using a ramdisk, I have to be vigilant to update the symlinks when steam overwrites them with a game file update. It's fussy. And windows will (very unfortunately) happily double-cache the data on the ramdisk -- except for THIS and ONLY this game lol. Normally I'd have to know in advance exactly how much ram the game needs, and then size the ramdisk perfectly to minimize the size of the windows filecache. Like I said, it's fussy.

All that being said, with this new PC's fast nvme drives AND enabling the nvidia option to make vulkan use the DXGI swap chain, this game is (amazingly) buttery smooth EVEN WITH 16gb (half!) of my ram going completely entirely utterly unused.

Which is the exact OPPOSITE of what everyone seems to be talking about in this thread.

So weird, right?

[stevecs]

@Daemonjax that sounds more like some external pressure (i.e. windows itself) forcing a memory cleanup event to the process due to resource constraints. Have you watched the process with process explorer as well as system's total resources?

In at least my case here which I just re-confirmed with the 6.20 release version still balloons (though a bit slower) and usually when entering/flying through new sectors over time but on mine I have a much larger surplus of resources (256GB ram, ~200GB free when starting X4) and I catch it before it gets to the point of forced cleanup as windows does that to *all* processes since it doesn't know a bad one from a good one which impacts other items that I have running.

Note, I am running Server 2021 (21h2) and windows 10 (21h2/LTSC) as a multi-boot system on server/workstation hardware (not that, that should make a difference here).

[MegaloManiac]

I had a similar problem. X4 used more cache RAM the more I travel through sectors and caused "X4 Panic, Failed to allocate more memory" crashes. I simply increased paging file up to 16 gigs. With 16 physical and 16 page file RAM game crashes no more.
Interestingly enough, both task manager and riva tuner show that RAM usage within normal parameters, however it is "allocated memory" usage that increased over time.