Why does X4 on Windows not cache data files?

[pref]

Will it potentially save more than 90 seconds of loading time and stutter time due to being sequential instead of random access on a mechanical drive? Currently this is not possible to test without resorting to some kind of third party drivers which RAM buffer content since the IO explicitly bypasses the OS file cache which can be used for such tests.

There is one way to see, but from what CBJ said there is another issue.

[pref]

If i get CBJ right, even on a ramdrive you would take long to load if the issue he mentions has nothing to do with caching..?

That's the way I understood it. And yet - game loads in around 21 seconds, saves load in around 24.

Load/save times are depending on asset count more then media imo, in a fresh game these times are much less then 50s for me too.
A real comparison would be to check the same save, which you can do if you want if you load one of my saves from here Picture album - my 5x shipyard support space city!.
A fresh start would be another option but there the differences would be insignificant and close to measurement imprecision.

[Jaga_Telesin]

Load/save times are depending on asset count more then media imo, in a fresh game these times are much less then 50s for me too.
A real comparison would be to check the same save, which you can do if you want if you load one of my saves from here Picture album - my 5x shipyard support space city!.
A fresh start would be another option but there the differences would be insignificant and close to measurement imprecision.

Sounds like worthwhile metrics testing, so I went and did it. I used the 3rd party caching software mentioned in the Steam topic to get drive throughput as fast as possible.

System in use: i7-6700k @4.8ghz, 64GB DDR4 RAM @3000mhz, W10 Pro x64, Samsung 970 Evo NVMe 1TB, 32GB L1 read/write cache (using 3rd party software to emulate RAMdrive speeds while still using original install/save location). In-game textures were set to High. I used your suggested savegames (v1.6) and converted them to 2.0 by re-saving with & without compression for the tests. Here are the results:

Load compressed save (from pref)
- Size: 49 MB (49,540 KB)
- Load from main menu: 42.0s, 42.3s, 43.0s
- Median compressed load time: 42.3s

Load uncompressed save (from pref)
- Size: 410 MB (410,699 KB)
- Load from main menu: 41.2s, 41.5s, 42.2s
- Median uncompressed load time: 41.5s

Game load from Desktop to main menu
- 19.6s, 20.2s, 20.3s
- Median game startup time: 20.2s

Create compressed save (using pref save data)
- 20.9s (size: 410 MB / 410,772 KB)

Create uncompressed save (using pref save data)
- 15.6s (size: 49 MB / 49,689 KB)

=========================================

Things I see from the results:
- With enough CPU speed, loading a compressed save results in negligible (~1s) increases in load speeds.
- Compression creates increased save times (around a 33% or higher increase).
- If you load a game in 50s, and I load the same in ~42s, then the caching software produces a decrease of about 20% in load times, even when your CPU is faster than mine.
- Saving is MASSIVELY faster than you're seeing. If you create a compressed save in 50s and I do it in 21s, then deferred write caching is over twice as fast.
- The game doesn't fully cleanup loaded data when exiting a game to the main menu. Loading was consistently faster when exiting the game completely, then re-launching and loading.

I know you're seeing zero stutter/hitching/pop-in when using the 3rd party software mentioned, which I can attest to. Planets/assets after a jump display immediately, etc.

It would be nice to see someone perform the tests with your save on an older spindle drive for comparison. I don't have any spares here that aren't in use. However your system using an NVMe, and mine using caching software (on top of an NVMe), appear to have some differences in times: 20% faster loading with caching, and 138% faster for saving with caching for my tests.

It would appear then, that caching has a marked effect on save load times and save creation times, at least compared to an NVMe. The differences would grow much more with older drives (a SSD or a HDD). And you are correct about overall # of assets in the game, and the size of the save taking longer to load. My original tests used new saves.

[pref]

- Saving is MASSIVELY faster than you're seeing. If you create a compressed save in 50s and I do it in 21s, then deferred write caching is over twice as fast.

I admit i had to check this. I just said it takes as long but that was just an impression. Never timed the save and perception of time is really subjective - especially since during gameplay i'm already in some activity and it might seem longer than at the loading screen when i'm just waiting for the game to continue.

So sorry about that, now i loaded my quicksave and pressed save again - like this it took 24s to save.

[Jaga_Telesin]

Ah, cool. So the speed gain from caching is still there on writes, but it more closely resembles the gains saw on reads. Good info.

[Imperial Good]

Unless they are explicitly flushing to disk, the file cache will buffer the writes and so allow the IO calls to return instantly. Save files do use the file cache, unlike the game data files, but I do not know if they explicitly flush the cache to reduce the chance for data loss or not.

Despite having 18 GB of memory, the game's internal cache does not seem to be doing a very good job. As an example when save scumming one makes a save, does an action and when the results are not desirable one reloads to try again. Under such work load there should be as good as no disk IO occurring since all assets should already be in the internal cache, and with 18GB of memory the internal cache could pretty much grow to hold all game data if required. This is not the case, as I can hear my hard disk head moving a lot (random access, slow) during such a cycle.

As such once again I have to put forward the argument that the no buffering flag be removed when the game opens the data files, or at least an option/command line flag to do so.

[blin25]

The whole game on the Ram disk is there any benefit. 1 To work 64bit OS, you need 4-5 GB plus the whole game with saving of about 13 GB. Also, when starting up, the game wants about 9 GB. Only 5 + 13 + 9 = 27 GB .This is only at the time of the beginning, and the game has a tendency to grow in memory very much.

[Jaga_Telesin]

The whole game on the Ram disk is there any benefit. 1 To work 64bit OS, you need 4-5 GB plus the whole game with saving of about 13 GB. Also, when starting up, the game wants about 9 GB. Only 5 + 13 + 9 = 27 GB .This is only at the time of the beginning, and the game has a tendency to grow in memory very much.

Use Primocache, it is disk caching software (the one we've been discussing). You don't need that much RAM to get good effect. My tests in this topic were done using it.

[pref]

As an example when save scumming one makes a save, does an action and when the results are not desirable one reloads to try again. Under such work load there should be as good as no disk IO occurring since all assets should already be in the internal cache, and with 18GB of memory the internal cache could pretty much grow to hold all game data if required.

Just in a second quite a large portion of the game object data changes, nearly all scripts' state etc. There isn't much you can reliably cache.
Do a compare on the uncompressed consecutive saves.

[Imperial Good]

As an example when save scumming one makes a save, does an action and when the results are not desirable one reloads to try again. Under such work load there should be as good as no disk IO occurring since all assets should already be in the internal cache, and with 18GB of memory the internal cache could pretty much grow to hold all game data if required.

Just in a second quite a large portion of the game object data changes, nearly all scripts' state etc. There isn't much you can reliably cache.
Do a compare on the uncompressed consecutive saves.

Your evidence that the data cannot be cached? Like have you done IO access pattern analysis and statistical processing to prove that data is accessed entirely randomly and distributed and hence caching has minimal effect?

If teleporting between a few sectors repeatedly there should be little to no IO as all assets should be cached internally, or in the file cache (if this was not explicitly bypassed). Instead one does get significant IO that on mechanical drives one can hear the seeking patterns. This contrasts a game like Heroes of the Storm where after several matches my HDD can even end up going to sleep due to such infrequent access pattern due to all the caching done from both tiers (in game and file cache).

Saves are file cached hence no difference. The bottleneck is not IO for writing and reading saves. The difference of compressed against uncompressed is that compression is resource intensive so to compress 300MB takes 5+ seconds. The difference in decompression is because decompression is still more resource intensive than no decompression but the difference is so small anyway (sub 1 second) that no one cares.

People are referring to using a tiered caching system, specifically a legacy concept one. Modern ones usually use a SSD to cache a mechanical drive, however in the old days before SSDs existed/were viable a RAM Disk was used to cache the mechanical drives. These smart caches act like a file cache but because they are part of the IO stack they bypass the "no buffering" flag which stops the standard windows file cache from working like X4 uses for accessing data files. As such they offer file cache like performance but for applications which do not allow the file cache to work, like X4. They are technically slightly better since they can persist the cache content between restarts unlike the file cache, however one can emulate this with the file cache for normal applications (not X4 since that bypasses it for data files) using a preload application to load chunks of the data into the file cache sequentially (fast, especially for mechanical drives) before launch.

The test one needs to do is checking load performance. Specifically testing both initial load (menu to loading a save) and repeated reloading of a save (loading a save which all assets have already be loaded). Especially the repeated reloading of a save should not cause any significant IO if an internal cache is used, and if it did then this is where a file cache would cause massive IO time savings but cannot because it is explicitly bypassed at the moment.

[pref]

As an example when save scumming one makes a save, does an action and when the results are not desirable one reloads to try again. Under such work load there should be as good as no disk IO occurring since all assets should already be in the internal cache, and with 18GB of memory the internal cache could pretty much grow to hold all game data if required.

Just in a second quite a large portion of the game object data changes, nearly all scripts' state etc. There isn't much you can reliably cache.
Do a compare on the uncompressed consecutive saves.

Your evidence that the data cannot be cached? Like have you done IO access pattern analysis and statistical processing to prove that data is accessed entirely randomly and distributed and hence caching has minimal effect?

I thought its about saving on several occasions and role of caching there.

With initial load the difference is 9 secs according to posts above for SSD vs RAM drive. 42 vs 50.
What's interesting is that the second load with SSD takes even longer (60 secs) - out of which 10-12 secs are spent on some clear up activity i think (that is how long the game stays in a frozen state after which the loading screen is displayed) and loading the same save within 10 secs of previous load it again takes 48-50 secs. With RAMdrive 2nd time also takes about the same time as initial loading.
So yes, no caching and SSD IO takes up about 20% of the loading time vs a RAMdrive - for HDD i can imagine it takes quite some time longer.
I only have 16g RAM, but 6G is still free - should be more then enough to cache the whole game data (which uses less then 6).

[Imperial Good]

I thought its about saving on several occasions and role of caching there.

Ah sorry I misunderstood.

The save files themselves are cached. Hence even placing the save files on a mechanical drive versus a RAM disk will make little difference to load times (1-2 seconds max). As good as all the loading time is a combination between state reconstruction (parsing the 300 MB or more of save data XML and creating a game state from it) and with loading the required assets for the player to play. What kills mechanical drive play performance is this loading of assets since it occurs painfully slowly due to the bad random access performance of mechanical drives. Once one gets to SSDs, especially NVMe ones, the difference between cached/RAM drive and uncached becomes very small in comparison, a few seconds at most and well within the range that different hardware can give.

[pref]

Once one gets to SSDs, especially NVMe ones, the difference between cached/RAM drive and uncached becomes very small in comparison, a few seconds at most and well within the range that different hardware can give.

Nope, that's what surprised me. Loading takes 20% longer on SSD vs RAMdrive. Quite a lot.. Okay i'm not saying that 9 secs has any kind of importance, but still way higher then i expected.

[Imperial Good]

Nope, that's what surprised me. Loading takes 20% longer on SSD vs RAMdrive. Quite a lot.. Okay i'm not saying that 9 secs has any kind of importance, but still way higher then i expected.

20% is still tiny compared with the 200% or more for a mechanical drive.

I wonder if this is caused by queue depth 1, or equally small numbers, access pattern. Especially with NVMe SSDs one is required to queue up many random IO operations at once to allow the SSD controller to be kept busy and provide maximum performance.

For example a Samsung 970 EVO Plus (modern NVMe drive, recently released high end) running with queue depth 1 can only manage a random access read speed of at most 19,000 IOPS of 4KB each which is just 76 MB/sec. This is best case as well, with actual performance likely being a lot lower. This is compared with its queue depth 32 performance of at best 600,000 IOPS of 4KB each or 2.4 GB/sec.

File cache or other RAM based file caching solutions can give random access performance in the order of gigabytes per second, even on older systems. They also do not really have a concept of queue depth in this case since the kernel can resolve the request instantly (no queue, limited by memory bandwidth and CPU time, both of which are very fast). Of course this only applies if the data is in the RAM cache, otherwise standard IO with an underlying mass storage is required.

Since X4 is reading in a random way, multiple IO read requests should be scheduled in parallel in other to maximize IO performance. It can potentially even help mechanical drives since they can queue multiple reads to schedule more efficient head movement, although this is nowhere near the gains that SSDs see.

[Kumba42]

We have our own caching mechanisms which are designed to work with our catalog system, and make the OS caching redundant in most cases.

The 75s startup-time mentioned by the OP is not related to file caching, and would not be helped by enabling it. It's a separate issue that only seems to affect certain people, and is something we'll be looking into when we can.

[snip]

The 75s startup time is because I am running off a big platter old mechanical drive, plain and simple. Other people report similar 60 second like times when using mechanical drives. Mechanical drives do not do random access well, especially big platter ones. People running off NVMe drives have loading times around 30 seconds, but I do not have one of those neither will I get one just to improve X4 loading performance. Likewise I will not buy into third party tiered caching solutions to bypass the no buffering flag with a hidden RAM cache and achieve something similar to what the windows file cache does by default, at least partly, within my use case. The tiered RAM cache approach is already a recommend way to improve X4 load performance, as the person above posted.

Running Windows Server 2016 here, 32GB system RAM, on an LSI 9750-4i hardware RAID-5, 512MB cache, w/ 3x WD "black" enterprise single-platter mechanical drives, ~1TB each. The game is installed to a ReFS-formatted partition about ~300GB in size, which is my "games" partition. Can report initial load time of save game from game start to be about ~75 seconds, give or take. Reloading a quicksave sometimes clocks in a bit less, but not by much (maybe ~10s less?). Depends on how long it's been since the last reload. Rapid reloads when testing something out are very fast, under ~30s, but I suspect that's my RAID cache coming into play there. After ~5min since the last reload, the long load time returns.

I sped up quicksaving games by disabling save game compression, as that appears to be using a single-threaded zlib library. Down to about ~15-20s to quick save. The autosave seems to take a bit longer for unknown reasons. Seriously am surprised that X4 doesn't use multi-threaded compression. It's got a number of open-source libs in it, so one more for multi-threaded compression doesn't seem like it'd make a difference.

Uncompressed games don't seem to load any faster. So I try to minimize how often I have to reload something. Normally, I'll save and close the game out, let Steam cloudsync, and then restart the game every few hours to avoid the obvious memory leak issues the game still has (very noticeable on the map becoming slogged down after a few hours).

In any event, I don't really mind the long load times. It's a chance to get up and stretch, walk around for a few seconds, etc. But it is something I hope Egosoft does eventually look into resolving somewhat in a future version.

[radcapricorn]

Gzipping a 221Mb save.xml into a 24Mb save.xml.gz here only takes 3 seconds. Saving takes several dozen seconds. Compression is not a significant factor here. Neither is writing a 24Mb blob to disk, or reading that 24Mb blob off disk. Or writing/reading a 200-300Mb blob for that matter.

[Imperial Good]

Seriously am surprised that X4 doesn't use multi-threaded compression. It's got a number of open-source libs in it, so one more for multi-threaded compression doesn't seem like it'd make a difference.

Since a single stream of data is being compressed, multi threaded compression might not yield much gains. Also depends on the compression algorithm used as some just do not scale well with multiple threads when compressing a single file.

Gzipping a 221Mb save.xml into a 24Mb save.xml.gz here only takes 3 seconds.

Depends on the compression quality/magnitude setting used. Low settings will take under 3 seconds while higher ones can take several minutes. Higher settings might have better gains but they come at a cost of longer compression times so it is a balancing act between good compression and longer compression times.

However what they can do is pipeline the entire save process so that as buffers of the XML file are being generated they get moved for compression on a separate thread and then written out asynchronously all in parallel. In this configuration the compression and IO time can be largely masked by the XML generation time, or whichever is slowest.

[radcapricorn]

Gzipping a 221Mb save.xml into a 24Mb save.xml.gz here only takes 3 seconds.

Depends on the compression quality/magnitude setting used. Low settings will take under 3 seconds while higher ones can take several minutes. Higher settings might have better gains but they come at a cost of longer compression times so it is a balancing act between good compression and longer compression times.

Thanks, captain

"Fun" fact: XML overhead severely outweighs actual useful data in the saves. Not to mention it inflates data size (try finding numbers that take up less than 4 bytes in it. Maybe there are a few "0"...). Basically, compression "solves" a problem that wouldn't have existed had they used a more efficient data format
But it also pretty much necessitates sequential saving and loading. Ugh...

[Tamina]

There is binary XML. Which "more efficient" data formats are there?

[radcapricorn]

There is binary XML. Which "more efficient" data formats are there?

In the wild? None. Binary XML isn't a solution either. Generic silver bullets do not exist It has to be tailored to data representation in game. If it isn't, you're wasting time on converting, and in this case, on slow sequential reconstruction without look-ahead.