Faster savegame

[Wispborn]

Compression and writing to disk are the only parts that could, in theory, be threaded and done once the game has been unpaused, but then you run the risk of finding out that the process failed after the game has carried on being played. That wouldn't be good!

It wouldn't be good, but it should be an edge case and a solution could simply be notifying the player, who would then need to save the game again and/or fix the problem, same as present-day. The game could even be automatically paused as soon as the process fails, so we'd be talking about 0 - 30 seconds of the player incorrectly acting as if the game were saved. Less than optimal for sure, but having that stage of saving fail should be an edge case.

For the > 99% of cases, it'll be a very welcome improvement to the saving experience.

[tomchk]

Compression and writing to disk are the only parts that could, in theory, be threaded and done once the game has been unpaused, but then you run the risk of finding out that the process failed after the game has carried on being played. That wouldn't be good!

It wouldn't be good, but it should be an edge case and a solution could simply be notifying the player, who would then need to save the game again and/or fix the problem, same as present-day. The game could even be automatically paused as soon as the process fails, so we'd be talking about 0 - 30 seconds of the player incorrectly acting as if the game were saved. Less than optimal for sure, but having that stage of saving fail should be an edge case.

For the > 99% of cases, it'll be a very welcome improvement to the saving experience.

*IF* this could be done in X4, I think it may be worth the risk! Just my two cents.

[Ranix]

not for a measely 4 seconds imho and I'd rather they work on other things

[radcapricorn]

Compression and writing to disk are the only parts...

...

For the > 99% of cases, it'll be a very welcome improvement to the saving experience.

It really wouldn't. Compression and writing to disk take only a small portion of time spent saving the game. Say your saves went from 25 seconds to 23. Would you really notice, or care?..

[SolidSnakeRUS]

Even less, save and compress takes about 1-2 second, as i mentioned before, most of time spent on serialization.
So, there is no need to spend time for implementation of background saving and compression until save data collector will be reworked, which is quite hard.

[tomchk]

Ah, that's right, never mind! Not worth the effort to save only 10% of the time.

[Shehriazad]

Putting an exception on the X4 savegame folder while having savegame compression disabled actually sped up the saving time for me remarkably.

It's always around 15 seconds now instead of my old 30-45 seconds.

My next step is trying to set up a symbolic link to move it from my normal SSD to my M.2 Drive to maybe shave off another second....

[Imperial Good]

It really wouldn't. Compression and writing to disk take only a small portion of time spent saving the game. Say your saves went from 25 seconds to 23. Would you really notice, or care?..

Assuming this does not happen in parallel already, in which case 0 time can be saved since compression and I/O is then not a bottleneck.

Some buffer of serialized data can be generated which then gets passed to another thread to compress with the compressed results then written out asynchronously. How fast the save can be done would be limited to whichever of those steps is the slowest assuming an appropriate buffer size is used. Sure there is still the pipeline latency added to the total time as it must be emptied but that is comparatively trivial.

My next step is trying to set up a symbolic link to move it from my normal SSD to my M.2 Drive to maybe shave off another second....

This should make no difference unless the game is writing through the OS file cache. Otherwise the system memory will absorb the I/O as good as instantly irrespective of if a fast M.2 or slow HDD is backing the file.

About 15-20 seconds is what it takes to save on a decent CPU such as a Zen2 with 3200 MHz memory.

[Shehriazad]

*Stuff*

Well doing it all DID however shorten my savegame times noticeable.

I actually use the save feature often enough now that I'll easily write 50gb of data a day.

I've got a Ryzen 3600 and VERY tight timed 3200mhz ram...so there's that. But I actually kind of measured the difference (I counted it on my fingers lmao) and it was noticeable enough so I thought I'd just share that at least for me it works.
The big one seemed to be adding an exception to your virus protection for the savegame folder when having compression disabled...Just did it because some dude on reddit told me to (lmao) and it worked like a charm.

[radcapricorn]

I counted it on my fingers lmao

Man, use a hourglass next time...

[Gregorovitch]

We had this topic quite a few times in the past already, and the reply is always the same. Quoting the (I hope) most recent reply from CBJ:

I'm afraid saving isn't likely to get much quicker, no matter how many people ask for it on a wish-list. The game universe is a complex structure with complex data, and it simply isn't possible to just take a "snapshot" of memory and save things from that in the background. The game needs to pause while the data is collected for saving.

I think Bernd spoke about this is in one of the XLog videos, too.

There is no doubt that X games, being "real" clockwork universe with all agents active all the time and all, are vastly more onerous to save than many, probably most, posssibly virtually all other games.

I am curious though why the game cannot transcribe the game's objects into XML on a separate thread in parallel behind the scenes so to speak?

[radcapricorn]

I am curious though why the game cannot transcribe the game's objects into XML on a separate thread in parallel behind the scenes so to speak?

Because those "objects" don't exist in a vacuum, so to speak (pun not intended). A save need to represent a singular moment in time, otherwise it will contain inconsistent data. You cannot be saving a live running state in the background, simulation has to be paused.

[SolidSnakeRUS]

Because, without pause, while you serialize one object, other objects will change. It's all about data consistency.

[Gregorovitch]

I am curious though why the game cannot transcribe the game's objects into XML on a separate thread in parallel behind the scenes so to speak?

Because those "objects" don't exist in a vacuum, so to speak (pun not intended). A save need to represent a singular moment in time, otherwise it will contain inconsistent data. You cannot be saving a live running state in the background, simulation has to be paused.

I have come across this arguement before in different contexts. I think the question should be asked if that level of consistency actually matters in this case. All it would seem to mean in practice is that, for example, the precise positions of all the ships would be slightly changed relative to each other on reload since their save points would be slightly temporally offset. It seems to me that would be a very small price to pay, inconsequential really, for avoiding the lengthy freeze on quicksave.

Of course there might be some deeper reasons, if so I'd be interested to know what they were.

[StoneLegionYT]

Not a programmer so sorry for a stupid question.

If a system had the memory. Could the information be replicated and dealt with separately without too much of a delay?

I assume a lot of the delay is the dealing with and compiling the data into the xml format, etc. Just wondering if it was replicated the raw data then used say separate task / thread in the background if it had even possible chance not even in X4 just overall if something like that was possible.

[radcapricorn]

Because those "objects" don't exist in a vacuum, so to speak (pun not intended). A save need to represent a singular moment in time, otherwise it will contain inconsistent data. You cannot be saving a live running state in the background, simulation has to be paused.

I have come across this arguement before in different contexts. I think the question should be asked if that level of consistency actually matters in this case. All it would seem to mean in practice is that, for example, the precise positions of all the ships would be slightly changed relative to each other on reload since their save points would be slightly temporally offset. It seems to me that would be a very small price to pay, inconsequential really, for avoiding the lengthy freeze on quicksave.

Of course there might be some deeper reasons, if so I'd be interested to know what they were.

That's gross, and I mean gross oversimplification. Coherent state matters, always. Say you fire a torpedo, and then the game starts saving. By chance, it first saves the position and velocity of torpedo, and only a few milliseconds later - the position and velocity of your ship. Which, in those few milliseconds, has moved, relative to that (now saved) position of torpedo. You play further, torpedo flies and hits its target. You'd expect the same happen after reloading. But now you reload such save, suddenly you get blasted by your own torpedo in the nether regions Or, you start saving just before a killshot reaches a carrier full of ships. The carrier is saved, but during that time in live universe it got destroyed, so it has no ships anymore. So those ships aren't saved. And by the time that killing bullet would've been saved, it's also not there anymore. So, in your live play, you hit "save" and witnessed destruction of a carrier. When you load, you will find the carrier alive, with no ships, and not blowing up anymore.

And these are just simple "harmless" examples. The game is not rigid. Things pop in and out of existence all the time. You'll get a save where things refer to other things that don't exist, or things that could not possibly exist do. In short, you'll get to a state that should be impossible to achieve by normal operation.

I guess a good analogy from the real world would be taking a long exposure photo of fast moving objects (saving), and then attempting to discern what the hell is going on in that image (loading).

[radcapricorn]

Not a programmer so sorry for a stupid question.

If a system had the memory. Could the information be replicated and dealt with separately without too much of a delay?

This question has already been answered in this very thread.

[StoneLegionYT]

Not a programmer so sorry for a stupid question.

If a system had the memory. Could the information be replicated and dealt with separately without too much of a delay?

This question has already been answered in this very thread.

Ahh Yeah just saw it the Snapshot part hmm. I guess because it can't just take raw binary data without actually handling it correctly. I assume the how compiling and moving it is what actually is most of the lag / work now so can't really do that then I guess haha.

So my next fix is why not allow us not to save scum and let us respawn when we die

[bobucles2]

One thing that I noticed is that saving and loading never exceeds a single thread. While concurrent programming for the actual data juggling is an incredibly difficult problem, that same challenge has already been solved for the compression algorithm itself. There are many concurrent compression programs, one quick search finds pigz. The compression ratio may take a bit of a hit, but as long as data in = data out, it's all good.

[Gregorovitch]

Or, you start saving just before a killshot reaches a carrier full of ships. The carrier is saved, but during that time in live universe it got destroyed, so it has no ships anymore. So those ships aren't saved.

Actually they may not be saved (depending on the object save order) but yeah, that one is a fair enough example, It would be impossible to avoid that possibility 100% and it's equally impossible to argue that it's inconsequential.