Why does X4 on Windows not cache data files?

[Imperial Good]

What started as a topic on steam between many people trying to sell various tiered file caching solutions has lead me to this question.

Why does X4 on Windows bypass the file cache when accessing game data files?

For anyone unfamiliar with the Windows file cache...
https://docs.microsoft.com/en-us/window ... le-caching

After extensive testing I have concluded that X4 is explicitly and completely bypassing file caching when reading the main data files. The main data files will not have any presence in the file cache after running X4, and X4 will even remove them from the file cache if they were present before starting (eg as the result of a custom application to force them into it). This means that all game data IO literally has to be read directly from the underlying drive every time the data is requested. This certainly explains why people running off mechanical drives, such as myself, have some quite poor loading times since there is no caching occurring at all.

In my case it means it takes consistently ~75 seconds from pressing to run the game until main menu, even if the game is immediately closed and re-opened, as all IO operations have to be repeated. Now to put this in contrast a game like Heroes of the Strom which does allow data files to be cached by the file cache initially takes 60 seconds to open but subsequently reopens in under 30 seconds due to the caching of the data files in the file cache.

Now of course people running off a SSD will suffer from this considerably less since SSDs are orders of magnitude faster than mechanical drives. Additionally people utilizing a tiered storage solution to cache file data from a mechanical drive onto a small SSD or in RAM will also not suffer from it. However one could simply allow the automatic file cache of Windows to help out by opening the data files without the FILE_FLAG_NO_BUFFERING flag (guessed from Windows API documentation and behaviour). Now this will not be as good as a tiered cache solution or a SSD, but it could potentially be better for pure mechanical drives, especially if the system has a lot of free memory.

Unfortunately testing the difference caching the data files would make to loading performance is not easy. Specifically it would require a modified executable which permits the data files to be cached by the file cache. However there might be some technical or performance reason why the X4 data files are not allowed to be cached which would only be noticeable when one tries it. On the other hand all games from developers like Blizzard Entertainment have always used the file cache to cache data files and never had a problem with doing so. Hence the topic and question as I am genuinely curious as to this design decision.

If anyone is interested, reasons to disable file caching of a file would be when dealing with a massive or numerous files which are only read once or which has parts that are read so infrequently it is as good as once. Text book examples would be video files being opened to be read for encoding or anti virus scan checking files for viruses. In both these cases caching the file data would do nothing but flush much more useful file data from the file cache for no reason and hence decrease its performance. Game data files are not really applicable under this logic since technically they might be used again, eg if the application is restarted after a user taking a break, and usually games have a working set of data far smaller than their install size which all benefit from caching.

[comatose1]

I read that steam thread and checked myself to see what was being cached to do with the game and was very surprised to find nothing at all, not a single instance of any of the data files! Well that's not good

First my game is already installed on a 1TB SSD game only drive and in the back of my mind I did not really expect to see/feel much of a difference by running a cache system, how wrong i was.

Since then I have been using a Kernel level RAM cache system, and I have to say the difference is more than noticeable. It is not just the save game loading times but also the playing of the game, no more odd stutters while travelling about and the blank/black planets for a second or so when you jump into a system or any of the ships/stations texture assets loading which were causing bad/high frame time spikes (MSI Afterburner can show you this data) with the cache system on those frame time spikes are much much lower so they are there almost not noticeable. This has resulted in a much smoother game to play.

Just to make sure I was not fooling myself here, after a few days of playing with the cache system on I turned the cache system off and played the same save again, 10 minutes later i quickly quit the game and turned it all back on again. The cache system is ONLY being used on the 1TB SSD game drive, Win 10's own OS cache system is being left to do it own thing just missing 8GB of RAM that is being assigned to the cache system, I am sure it will cope without it while I am playing this game.

Why are you prohibiting the use of the OS cache system on the game files Egosoft? It is kind of hurting your great game.
Maybe you should do some testing amongst yourselves and find out what a difference caching the game files can make even on a HEDT machine listed below Egosoft.

Machine specs....
Asus Prime Z370 A with 32GB DDR4 3200 RAM.
i7 8700K with all cores boost to 4.9Ghz, water cooled with a 360 rad.
120GB Boot SSD, 1TB Game SSD and a 2x 8TB spinning rust HDD's for media storage etc.
Water cooled 1080 Ti driving a Predator 1440 Gsync monitor.

[Imperial Good]

According to various documentation from Microsoft, disabling buffering (do not use file cache) when opening a file for asynchronous operation maximizes asynchronous IO performance. The logic behind this is that the file cache manager requires synchronization and as such becomes a bottleneck in a highly parallel environment. That said I believe this is intended for server workloads where the file is located on a massively parallel and enterprise grade storage array with access patterns that gain very little from OS level caching algorithms. An example of such an application might be a database management systems operating on massive databases since queries may be executed in parallel on massive RAID storage arrays and the system is better off using its memory for user level caching of query results.

I do not think it is possible for consumer systems to run into this bottleneck. Even a high end consumer NVMe drive is only capable of around 2 GB/sec read speed for short periods with processor memory bandwidth being around 41.6 GB/sec (20 times the drive speed) and even then the cores potentially being data staved. With such a drive the synchronization bottleneck on IO performance is almost certainly trivial, and more than compensated by even just a few cache hits due to the reduced latency and kernel workload.

I would strongly recommend adding a setting to allow data files to use file caching and to have it active by default.

[CBJ]

We have our own caching mechanisms which are designed to work with our catalog system, and make the OS caching redundant in most cases.

The 75s startup-time mentioned by the OP is not related to file caching, and would not be helped by enabling it. It's a separate issue that only seems to affect certain people, and is something we'll be looking into when we can.

[Imperial Good]

We have our own caching mechanisms which are designed to work with our catalog system, and make the OS caching redundant in most cases.

The 75s startup-time mentioned by the OP is not related to file caching, and would not be helped by enabling it. It's a separate issue that only seems to affect certain people, and is something we'll be looking into when we can.

It should not make the OS cache redundant since that works in parallel and persists between sessions. Currently if you close X4 all caching is lost, which means the same poor initial load stutter when you next start it. Such double caching is not uncommon, with games from Blizzard doing it with both an internal asset cache while playing to improve loading performance during a session and using the file cache outside to allow some loading performance gains between sessions if memory is sufficient.

The 75s startup time is because I am running off a big platter old mechanical drive, plain and simple. Other people report similar 60 second like times when using mechanical drives. Mechanical drives do not do random access well, especially big platter ones. People running off NVMe drives have loading times around 30 seconds, but I do not have one of those neither will I get one just to improve X4 loading performance. Likewise I will not buy into third party tiered caching solutions to bypass the no buffering flag with a hidden RAM cache and achieve something similar to what the windows file cache does by default, at least partly, within my use case. The tiered RAM cache approach is already a recommend way to improve X4 load performance, as the person above posted.

File caching can help in this case. Since I have 18GB of memory, in worst case I could force all 8GB of game data files into the file cache in the background before playing to give faster than SSD load performance at least initially. Reading this 8GB sequentially is very efficient (>100MB/sec) as opposed to reading it via random access which I assume the game is doing by the low drive IO speed (<10 MB/sec). Of course this is largely speculation since there is no setting to allow one to test this.

Honestly I see no harm in such a setting letting the user choose. If one is low on memory then the current approach is probably better. However if one has a lot of memory (16-32GB) then there is no harm in letting the file cache do its stuff.

[Nafensoriel]

We have our own caching mechanisms which are designed to work with our catalog system, and make the OS caching redundant in most cases.

The 75s startup-time mentioned by the OP is not related to file caching, and would not be helped by enabling it. It's a separate issue that only seems to affect certain people, and is something we'll be looking into when we can.

It should not make the OS cache redundant since that works in parallel and persists between sessions. Currently if you close X4 all caching is lost, which means the same poor initial load stutter when you next start it. Such double caching is not uncommon, with games from Blizzard doing it with both an internal asset cache while playing to improve loading performance during a session and using the file cache outside to allow some loading performance gains between sessions if memory is sufficient.

The 75s startup time is because I am running off a big platter old mechanical drive, plain and simple. Other people report similar 60 second like times when using mechanical drives. Mechanical drives do not do random access well, especially big platter ones. People running off NVMe drives have loading times around 30 seconds, but I do not have one of those neither will I get one just to improve X4 loading performance. Likewise I will not buy into third party tiered caching solutions to bypass the no buffering flag with a hidden RAM cache and achieve something similar to what the windows file cache does by default, at least partly, within my use case. The tiered RAM cache approach is already a recommend way to improve X4 load performance, as the person above posted.

File caching can help in this case. Since I have 18GB of memory, in worst case I could force all 8GB of game data files into the file cache in the background before playing to give faster than SSD load performance at least initially. Reading this 8GB sequentially is very efficient (>100MB/sec) as opposed to reading it via random access which I assume the game is doing by the low drive IO speed (<10 MB/sec). Of course this is largely speculation since there is no setting to allow one to test this.

Honestly I see no harm in such a setting letting the user choose. If one is low on memory then the current approach is probably better. However if one has a lot of memory (16-32GB) then there is no harm in letting the file cache do its stuff.

I am curious to why you think a 75s startup time has anything to do with slow platters? Platter speed has little to do with first reads/writes. It only matters to total read/write speeds. I cant think of a physical reason for a drive to ever need 75s to begin to read ANYTHING.

Chances are you just won the Powerball lottery of bugs and legitimately have a hardware config outside the expected. It happens. They'll fix it.

[pref]

I cant think of a physical reason for a drive to ever need 75s to begin to read ANYTHING.

Exactly, it has most likely nothing to do with drive and caching - it's about iterating through all game assets.
I also have 30s+ load/save times on an NVMe drive.

[Imperial Good]

I am curious to why you think a 75s startup time has anything to do with slow platters? Platter speed has little to do with first reads/writes. It only matters to total read/write speeds. I cant think of a physical reason for a drive to ever need 75s to begin to read ANYTHING.

Chances are you just won the Powerball lottery of bugs and legitimately have a hardware config outside the expected. It happens. They'll fix it.

Random access latency. Since the drive has to wait for 1 full spin before it can read the next chunk of data. Result is that when starting X4 the read speed is as low as 1 MB/sec for some of those 75 seconds since it is having to read a lot of very small pieces of data. Smaller platter (faster spinning) drives help reduce this latency so can achieve higher read speeds with the same workload. Solid state drives are not at all effected by this since they have no mechanically moving parts and as such random access patterns are executed order of magnitudes faster than with mechanical drives.

Here are some diagrams of the IO being performed during the start-up sequence (links may be invalid in distant future).

This shows the detailed view of the IO with respect to time. Note that it was not stopped the instant loading finished so one must use the IO load as a reference.
https://www.dropbox.com/s/2iqsnxfz2ugh7 ... _35_06.png

This is a task manager view stopped more closely to the time the game finished loading. Time axis is 4 minutes for reference (slow sample speed).
https://www.dropbox.com/s/ymddgrxedtx8c ... _33_26.png

This shows the total IO metrics performed during the startup sequence.
https://www.dropbox.com/s/jgyoq0yolrloy ... _42_41.png

The start-up test was repeated and the graphs were approximately the same shape. As one can see about 450MB of data must be read to startup. This averages to around 6MB/second IO speed. Over half the time is spent with an IO speed in the range of 1-3MB/sec, almost certainly the result of random access patters on a mechanical drive reading lots of small data files. At other times higher IO speeds were achieved, likely the result of larger files being read which fall closer to sequential access speed limits. In theory if the file cache could be used then upon immediate restart that period of very slow IO should occur considerably faster, and load time would probably become bottlenecked by my CPU processing the data rather than the drive IO speed.

For reference here is an image of task manager showing IO speed when all game data files are read sequentially into the file cache by a custom written program.
https://www.dropbox.com/s/3esiq5pgtauf0 ... _49_05.png
An average speed of 100MB/sec was achieved. In roughly 90 seconds 9GB of data was read rather than 450MB over 75 seconds. Although meaningless, repeating the same test used to load the 9GB of data completed in under 10 seconds, likely limited by the how memory mapping large chunks is performed in Java however it is enough to prove the file data was cached and achieving RAM like speeds. This could be further verified by RAMMap.

I also have 30s+ load/save times on an NVMe drive.

Save is bottlenecked by CPU largely. Load time I am unsure as I have not profiled the IO load for that yet. Save files do use the file cache anyway and are large sequential files so NVMe or mechanical drive make as good as no difference for save time but a faster CPU core clock speed will. Loading requires loading assets so should run considerably faster on a SSD than a mechanical drive due to the above described random access pattern being executed better.

[radcapricorn]

...As one can see about 450MB of data must be read to startup. This averages to around 6MB/second IO speed. Over half the time is spent with an IO speed in the range of 1-3MB/sec, almost certainly the result of random access patters on a mechanical drive reading lots of small data files...

...For reference here is an image of task manager showing IO speed when all game data files are read sequentially into the file cache by a custom written program... ...An average speed of 100MB/sec was achieved...

I don't think your analysis is accurate. You're assuming that during startup the game does nothing but reads data off disk. And you demonstrate yourself that, if that was the case, read speeds would've been much much higher. The slowdowns you attribute solely to random access may as well be caused by the game's loading logic. A more accurate comparison would've been by analysing the game's I/O access while loading from a drive vs. a ramdisk.

[blin25]

Game saves are compressed by default. So the processor will be heavily used to unpack them.

[pref]

Loading requires loading assets so should run considerably faster on a SSD than a mechanical drive due to the above described random access pattern being executed better.

Loading also requires iterating through the same assets and creating them. That's the point, and that is why load takes so long on even a quick drive, not because NVMe takes so long to load a couple GBs.
Cache will not help it much.
I'm sure that launching the game takes less time for me then for someone with a HDD, but that time isn't much compared to loading/saving a game.
As CBJ mentioned some circumstances can make the startup time much longer, but that again is not the result of lack of caching because then it would affect everyone.

[radcapricorn]

Game saves are compressed by default. So the processor will be heavily used to unpack them.

That's not a significant contributing factor to load times at all. Sequential reconstruction of the universe (and perhaps reading required assets) is. Reading and gunzipping a single 30-50Mb archive containing a text file won't even take a second. And Imperial's looking, at least at the moment, at the game's startup time, not loading a save.

[Jaga_Telesin]

What started as a topic on steam between many people trying to sell various tiered file caching solutions has lead me to this question.

No one out there was trying to sell anything. It was a software product that works far better than Windows' file cache, and better than X4's internal caching mechanism, that was proposed as something that would take advantage of additional RAM and help justify that more memory can be helpful to the game. But nice attempt trying to downplay the advice given out there as a sales pitch.

For the record - using that caching product the game starts up in ~21 seconds for me, and saves load in ~24 seconds. 45 seconds total from the Windows Desktop (1 second less when disabling online features). Roughly 500MB is read during game start and save load from the product's cache, indicating it works as it should. Benchmarks (both random IO and sequential IO) are shown to be faster than an Optane, over 24GB/s. Machines using the software (especially a 5400 RPM spindle drive) will see speeds gains after data makes it's way into the cache.

Why you are discussing the Windows internal cache either place is beyond me. It was never a focus of that discussion, and as terrible as it is should never be considered a serious caching mechanism.

Additionally, the fact that the game discards it's own cache at exit time only lends support to a 3rd party caching solution as being helpful, and that 'more RAM' does indeed have it's uses where X4 is concerned. I won't repeat the product name here out of respect for the person who suggested it, and so that you can't claim anyone is trying to 'sell it'.

comatose1 has apparently confirmed that more RAM and that product (or a similar one) do indeed help tremendously. Load hitching and stuttering are eliminated, everywhere.

Your assumption that 'save is bottlenecked by CPU' is absolutely incorrect. Saves on my system (using the aforementioned product and deferred writes) take exactly 10 seconds. I use an i7-6700k, a 3-year old CPU with hyperthreading turned on. Hardly the latest and greatest, and if there was CPU bottlenecking going on for disk output I'd be seeing speeds similar to pref's, or slower.

The poster on Steam asked, and was given an answer, accurately. More than one person has tried the suggestion and found it worked. I'm not sure why you're still harping about the game using the Windows file cache, or discussing it's internal caching scheme. They are a moot point - more memory (along with a 3rd party product) can help the game.

[Nafensoriel]

/snip

Just to be slightly pedantic... an i7-6700 is by no means limited in any way and is considerably more powerful than most "budget" level processors. Including modern ones. Remember the IPC didn't really change much between skylake and coffee lake. All the newer generation has is more cores to work with. If you were using an I3 or a pre zen processor you would have a more persuasive argument.

Either way, CBJ confirmed its not their cache and from his wording it sounds like they have a general idea what's causing the issue. Third party solutions cause third party issues. YMMV in trying them.

[Imperial Good]

...As one can see about 450MB of data must be read to startup. This averages to around 6MB/second IO speed. Over half the time is spent with an IO speed in the range of 1-3MB/sec, almost certainly the result of random access patters on a mechanical drive reading lots of small data files...

...For reference here is an image of task manager showing IO speed when all game data files are read sequentially into the file cache by a custom written program... ...An average speed of 100MB/sec was achieved...

I don't think your analysis is accurate. You're assuming that during startup the game does nothing but reads data off disk. And you demonstrate yourself that, if that was the case, read speeds would've been much much higher. The slowdowns you attribute solely to random access may as well be caused by the game's loading logic. A more accurate comparison would've been by analysing the game's I/O access while loading from a drive vs. a ramdisk.

People have done this. Sub 30 seconds down from 60 when not. If the data files used the file cache I could test it that way but...

In any case the IO speed is accurate for random access. Since that is just what 10 year old 1TB big platter drives get from random access IO performance. I notice similar performance in other games and tests I run, hence why I assume it is performing random access during that time period.

Game saves are compressed by default. So the processor will be heavily used to unpack them.

No it will not be heavily used to extract them since decompressing is a lot faster than compressing. Parsing a save into DOM form in Java, including disk IO and decompression, takes about 3 seconds on my very old I7 920 stock clock speeds. Almost certainly most of the overhead is allocating the millions of DOM nodes.

Loading also requires iterating through the same assets and creating them. That's the point, and that is why load takes so long on even a quick drive, not because NVMe takes so long to load a couple GBs.
Cache will not help it much.

Faster IO will improve loading times because loading factors in asset loading to minimize pop in.

I'm sure that launching the game takes less time for me then for someone with a HDD, but that time isn't much compared to loading/saving a game.
As CBJ mentioned some circumstances can make the startup time much longer, but that again is not the result of lack of caching because then it would affect everyone.

The drive I am using is a worst case and not something I would recommend modern systems to use for their games. It is a very old (10 year) 1TB big platter mechanical drive so I can expect very bad loading performance from it. That said a lot of people say loading is an issue for this game and many people report massive improvements using hybrid RAM disk caches (tiered storage drives with a Tier 0 RAM cache) which bypass the NO_BUFFERING flag.

How long does it take for X4 to start for you off what sort of drive you have it installed on? Is it 30 seconds (SSD reported times) or closer to 75 seconds?

No one out there was trying to sell anything. It was a software product that works far better than Windows' file cache, and better than X4's internal caching mechanism, that was proposed as something that would take advantage of additional RAM and help justify that more memory can be helpful to the game. But nice attempt trying to downplay the advice given out there as a sales pitch.

Well obviously it works better than Windows in built file cache with X4 because X4 bypasses it entirely... One cannot even measure the difference. In a war of no caching against some caching obviously some caching will win, especially off a mechanical drive.

For the record - using that caching product the game starts up in ~21 seconds for me, and saves load in ~24 seconds. 45 seconds total from the Windows Desktop (1 second less when disabling online features). Roughly 500MB is read during game start and save load from the product's cache, indicating it works as it should. Benchmarks (both random IO and sequential IO) are shown to be faster than an Optane, over 24GB/s. Machines using the software (especially a 5400 RPM spindle drive) will see speeds gains after data makes it's way into the cache.

However this is a very one sided test since one cannot compare it with Windows file cache to see if that would make any difference. I cannot deny your results and as you prove, one should be getting better IO performance with sufficient memory, but one cannot claim it is better than windows file cache because there are no windows file cache measurements to test it against.

Why you are discussing the Windows internal cache either place is beyond me. It was never a focus of that discussion, and as terrible as it is should never be considered a serious caching mechanism.

It fulfils its purpose for most games. And for ones which are very small, eg old games, gives you the same RAM disk like performance with a little help of some simple preload applications to force file data into the cache. Did this 10 years ago for Warcraft III.

Additionally, the fact that the game discards it's own cache at exit time only lends support to a 3rd party caching solution as being helpful, and that 'more RAM' does indeed have it's uses where X4 is concerned. I won't repeat the product name here out of respect for the person who suggested it, and so that you can't claim anyone is trying to 'sell it'.

Again this is not a fair test because the game explicitly stops windows from caching the data files. Next time it starts it performs approximately as well as it did the first time because all data IO must begin anew. Any sort of caching wins, even if the caching is slower than what windows file cache could provide just there is no competition so it wins by default. This topic was created to point out that currently the only choice is to use such third party solutions because the native file cache is explicitly stopped from working. The only reason these third party caches work in the first place is they are effectively cheating around the no buffering flag, allowing data to be in memory already.

Your assumption that 'save is bottlenecked by CPU' is absolutely incorrect. Saves on my system (using the aforementioned product and deferred writes) take exactly 10 seconds. I use an i7-6700k, a 3-year old CPU with hyperthreading turned on. Hardly the latest and greatest, and if there was CPU bottlenecking going on for disk output I'd be seeing speeds similar to pref's, or slower.

Save files might be 50MB max. That is under 2 seconds write time for even an old mechanical drive and since they are file cached one can assume RAM speeds unless explicitly forced out. They are CPU bottlenecked because of compression, compressing 300+MB will take some time, especially to get a decent compression factor. Let us not forget the CPU resources needed to generate 300+MB of XML which is a textual format. You get 10 seconds because you have such a new and modern processor, my older I7 920 takes considerably longer (20+ seconds). That said I have noticed saving anomalies where some saves happen very fast (10-15 seconds) and others much slower (towards 30 seconds), so maybe there is some issue there but I do not have proof yet.

The poster on Steam asked, and was given an answer, accurately. More than one person has tried the suggestion and found it worked. I'm not sure why you're still harping about the game using the Windows file cache, or discussing it's internal caching scheme. They are a moot point - more memory (along with a 3rd party product) can help the game.

Which is true. However it is only true by default because X4 cannot use competing solutions like windows file cache since it is explicitly disabled. Some caching is obviously better than no caching.

Either way, CBJ confirmed its not their cache and from his wording it sounds like they have a general idea what's causing the issue.

Yeh it is not their cache since their cache only gets populated during play and is discarded on application close. This did not provide substantial proof that during a multiple start stop cycle that the windows file cache will not improve performance on systems with 16GB or more of memory.

I do not think an option to allow windows file cache to work, even if defaulted to off, would be a bad thing.

[radcapricorn]

I don't think your analysis is accurate. You're assuming that during startup the game does nothing but reads data off disk. And you demonstrate yourself that, if that was the case, read speeds would've been much much higher. The slowdowns you attribute solely to random access may as well be caused by the game's loading logic. A more accurate comparison would've been by analysing the game's I/O access while loading from a drive vs. a ramdisk.

People have done this. Sub 30 seconds down from 60 when not. If the data files used the file cache I could test it that way but...

In any case the IO speed is accurate for random access. Since that is just what 10 year old 1TB big platter drives get from random access IO performance. I notice similar performance in other games and tests I run, hence why I assume it is performing random access during that time period.

You misunderstood me. Obviously startup time from a ramdisk will be far superior, that is not my point. In your analysis you're comparing actual game's startup time against what I assume plain sequential reading of the same amount of data. Latter will be faster on any machine, and any drive (barring severe fragmentation). A more fair comparison would be to analyze I/O performed by game's startup on the same machine, but using different media. Because that will give you a better picture of the game's I/O access patterns vs. any operations it performs between reads.

[Jaga_Telesin]

I don't think your analysis is accurate. You're assuming that during startup the game does nothing but reads data off disk. And you demonstrate yourself that, if that was the case, read speeds would've been much much higher. The slowdowns you attribute solely to random access may as well be caused by the game's loading logic. A more accurate comparison would've been by analysing the game's I/O access while loading from a drive vs. a ramdisk.

People have done this. Sub 30 seconds down from 60 when not. If the data files used the file cache I could test it that way but...

In any case the IO speed is accurate for random access. Since that is just what 10 year old 1TB big platter drives get from random access IO performance. I notice similar performance in other games and tests I run, hence why I assume it is performing random access during that time period.

You misunderstood me. Obviously startup time from a ramdisk will be far superior, that is not my point. In your analysis you're comparing actual game's startup time against what I assume plain sequential reading of the same amount of data. Latter will be faster on any machine, and any drive (barring severe fragmentation). A more fair comparison would be to analyze I/O performed by game's startup on the same machine, but using different media. Because that will give you a better picture of the game's I/O access patterns vs. any operations it performs between reads.

And even a RAMdrive constitutes "different media". It emulates a NTFS volume and performs the same tasks as directed by I/O. To do the test adequately, you'd have to install the game to, launch it from, and save to (via a symbolic link mapping) a RAMdrive of sufficient size.

That, or test an old 5400 RPM spindle drive against a new NVMe drive. At least the two have enough of a speed difference to make a noticeable impact in results.

Given that I posted my times, was able to compare to pref's times (where he used a faster processor by far)... and mine were lower on both loading and saving, I'd say the media type makes a significant difference. My media for the purposes of comparison may as well have been a RAMdrive, since it both reads from and writes to memory first.

[pref]

The drive I am using is a worst case and not something I would recommend modern systems to use for their games. It is a very old (10 year) 1TB big platter mechanical drive so I can expect very bad loading performance from it. That said a lot of people say loading is an issue for this game and many people report massive improvements using hybrid RAM disk caches (tiered storage drives with a Tier 0 RAM cache) which bypass the NO_BUFFERING flag.

How long does it take for X4 to start for you off what sort of drive you have it installed on? Is it 30 seconds (SSD reported times) or closer to 75 seconds?

Starting the game off is 30 secs until the main menu is available if i don't cancel the intro movie. So maybe 20ish?

Loading is indeed slow just as saving - loading a game takes now with a more advanced save 50 secs.

Drive's a 970 EVO - 3,500MB/s sequential read in theory.

I think the real performance gain for a ramdrive is in game asset loading/pop in, i can imagine that can reduce some lag spikes. That might be the next best thing for this game after a strong CPU and acceptable GPU/vram as it streams everything, there are no loading screens at all. If you have enough memory to dedicate 10gigs to the game it might make things much smoother on sector changes or with a modded extreme travel speed ship.
With startup time i don't think there would be much more gain vs an SSD, i bet it won't load the full 10G at start (never had more then 8G mem usage i think?).
If i get CBJ right, even on a ramdrive you would take long to load if the issue he mentions has nothing to do with caching..?

[Jaga_Telesin]

If i get CBJ right, even on a ramdrive you would take long to load if the issue he mentions has nothing to do with caching..?

That's the way I understood it. And yet - game loads in around 21 seconds, saves load in around 24.

[Imperial Good]

I think the real performance gain for a ramdrive is in game asset loading/pop in, i can imagine that can reduce some lag spikes. That might be the next best thing for this game after a strong CPU and acceptable GPU/vram as it streams everything, there are no loading screens at all. If you have enough memory to dedicate 10gigs to the game it might make things much smoother on sector changes or with a modded extreme travel speed ship.
With startup time i don't think there would be much more gain vs an SSD, i bet it won't load the full 10G at start (never had more then 8G mem usage i think?).
If i get CBJ right, even on a ramdrive you would take long to load if the issue he mentions has nothing to do with caching..?

The test is not so much if file caching or tiered storage with RAM is faster than a SSD, which it is but likely not noticeable, but rather if it could potentially give improvements at some stage to the performance of a mechanical drive.

For example if I was to spend 90 seconds to cache in all 9GB of data files into my 18GB of memory before starting the client, how much better would startup, loading and even playing perform? Will it potentially save more than 90 seconds of loading time and stutter time due to being sequential instead of random access on a mechanical drive? Currently this is not possible to test without resorting to some kind of third party drivers which RAM buffer content since the IO explicitly bypasses the OS file cache which can be used for such tests.