Faster savegame

[SolidSnakeRUS]

Hi!

What about some improvements to speed up savegame process?
Obviously serializing memory take some time, but how much?
What is the most time consuming stage (Serialize memory or compression and dumping to file)?

Is it possible to stop game only for time that required to take state snapshot?
Memory operations is extremely fast, so it should take less than half second.
Next, game can continue while save thread will do the job, like data to text serialization, compression and dumping to file.

[SolidSnakeRUS]

Where are you, dear developers?

https://steamcommunity.com/app/392160/d ... 1586377027

[X2-Illuminatus]

We had this topic quite a few times in the past already, and the reply is always the same. Quoting the (I hope) most recent reply from CBJ:

I'm afraid saving isn't likely to get much quicker, no matter how many people ask for it on a wish-list. The game universe is a complex structure with complex data, and it simply isn't possible to just take a "snapshot" of memory and save things from that in the background. The game needs to pause while the data is collected for saving.

I think Bernd spoke about this is in one of the XLog videos, too.

[SirNukes]

What is the most time consuming stage (Serialize memory or compression and dumping to file)?

The saves are gzipped with compression level 6. On a 29 MB save, compressing and saving the data (after separately decompressing to xml) takes 4.5 seconds for me, where the in-game save takes 22 seconds. So about 20% is compress and write.

I too wouldn't mind seeing saving sped up. But maybe egosoft worries about the player alt-F4ing out once the game is no longer paused, regardless of the save still completing in the background.

[StoneLegionYT]

What is the most time consuming stage (Serialize memory or compression and dumping to file)?

The saves are gzipped with compression level 6. On a 29 MB save, compressing and saving the data (after separately decompressing to xml) takes 4.5 seconds for me, where the in-game save takes 22 seconds. So about 20% is compress and write.

I too wouldn't mind seeing saving sped up. But maybe egosoft worries about the player alt-F4ing out once the game is no longer paused, regardless of the save still completing in the background.

Is it really that? If so add a option for it with a warning popup... Maybe even add a text bottom right like FPS showing saving status.

I had assumed it was more the stopping the simulation compiling it into memory then saving it. 500mb save on a m.2 drive is well 1/6th of a second lol.

If I'm wrong please correct me I assume it's more technical then that.


What I like to see is the ability for me to do less load / save and give me clone or a way to respawn without reloading the game. Make the game more immersive for me and offers another method then just reloading saves over and over.

[Imperial Good]

So about 20% is compress and write.

Assuming that the process is pipelined and run across many threads. In which case on a multi core CPU the compress and write stages are completely masked by the time taken to gather the data as they are not a bottleneck.

[SolidSnakeRUS]

We had this topic quite a few times in the past already, and the reply is always the same. Quoting the (I hope) most recent reply from CBJ:

I'm afraid saving isn't likely to get much quicker, no matter how many people ask for it on a wish-list. The game universe is a complex structure with complex data, and it simply isn't possible to just take a "snapshot" of memory and save things from that in the background. The game needs to pause while the data is collected for saving.

I think Bernd spoke about this is in one of the XLog videos, too.

The game universe is a complex structure with complex data, and it simply isn't possible to just take a "snapshot" of memory and save things from that in the background. The game needs to pause while the data is collected for saving.

Thant's true, game must be paused to collect data.
But memory operations is very fast, why saving take 15-30 seconds?

[radcapricorn]

But memory operations is very fast, why saving take 15-30 seconds?

Memory access is one of, if not the slowest thing that a modern CPU has to do. Raw memory copying benchmarks have little to do with a game - that's not what a game does.
To muse on CBJ's response quoted above: "The game universe is a complex structure with complex data, and it simply isn't possible to just take a "snapshot" of memory and save things from that in the background."
It can be possible, but for whatever reasons they didn't implement the game in such a way as to make it possible, and so making it possible would be a huge undertaking now. But let's not try to do backseat programming, and instead let's just assume that those were, hopefully, good reasons.

[Ranix]

We had this topic quite a few times in the past already, and the reply is always the same. Quoting the (I hope) most recent reply from CBJ:

I'm afraid saving isn't likely to get much quicker, no matter how many people ask for it on a wish-list. The game universe is a complex structure with complex data, and it simply isn't possible to just take a "snapshot" of memory and save things from that in the background. The game needs to pause while the data is collected for saving.

I think Bernd spoke about this is in one of the XLog videos, too.

the game needs to pause while the data is collected for saving, but not while the data is serialized into XML, compressed, and written to a file. We know save files are ~500MB uncompressed so we can be pretty sure the data being rendered is smaller than that. We have machines with 16, 32, or even 64GB of RAM - making a copy of the data that needs to be saved to the heap and then writing that in a background thread is not infeasible

I doubt it's the best use of developer time but it's not infeasible

[SolidSnakeRUS]

But let's not try to do backseat programming, and instead let's just assume that those were, hopefully, good reasons.

Common reason is timeline. Product must be available on market as soon as possible.
Other thing is existing code base. I suppose, some parts was reused, which is not bad at all.
Saving, seems, works in the same way as it was in X3 (or even X2?).

So if they find time to improve this part, next games will also benefit.
If there is absolutely no time, maybe, at least, the next X game will include re-implemented saving with focus on speed.

[radcapricorn]

We have machines with 16, 32, or even 64GB of RAM - making a copy of the data that needs to be saved to the heap and then writing that in a background thread is not infeasible

How much memory you have in your machine is, in this case, a secondary concern. If the game state is a giant graph with random pointers everywhere you're very likely to spend comparable amount of time "just" copying it, to traversing it for saving. You can't simply copy a blob of memory and call it done when all the contents that you've copied still refer to the original "live" version all over the place.

[Ranix]

We have machines with 16, 32, or even 64GB of RAM - making a copy of the data that needs to be saved to the heap and then writing that in a background thread is not infeasible

How much memory you have in your machine is, in this case, a secondary concern. If the game state is a giant graph with random pointers everywhere you're very likely to spend comparable amount of time "just" copying it, to traversing it for saving. You can't simply copy a blob of memory and call it done when all the contents that you've copied still refer to the original "live" version all over the place.

it was already pointed out upthread that just threading the compression and disk write is a 20% time savings. You know the game already has to traverse and deep copy that data anyway to make an XML file right?

[SolidSnakeRUS]

We have machines with 16, 32, or even 64GB of RAM - making a copy of the data that needs to be saved to the heap and then writing that in a background thread is not infeasible

How much memory you have in your machine is, in this case, a secondary concern. If the game state is a giant graph with random pointers everywhere you're very likely to spend comparable amount of time "just" copying it, to traversing it for saving. You can't simply copy a blob of memory and call it done when all the contents that you've copied still refer to the original "live" version all over the place.

it was already pointed out upthread that just threading the compression and disk write is a 20% time savings. You know the game already has to traverse and deep copy that data anyway to make an XML file right?

radcapricorn talking about algorithm that collecting data for further processing.
If they use single threaded data collection of huge graph with random pointers, then, data collection may take some noticeable time. Indeed.
But we don't know, is this assumption is correct or not. Even if so, there is a solution - optimized multi-threaded data collector. Not easy, but still possible.

[CBJ]

If the game state is a giant graph with random pointers everywhere you're very likely to spend comparable amount of time "just" copying it, to traversing it for saving. You can't simply copy a blob of memory and call it done when all the contents that you've copied still refer to the original "live" version all over the place.

This is basically it. The game's "live" state is, and needs to be, optimised for running the game, not for saving it, so there are lots and lots of pointers and other things that you can't just copy and expect to still work. Serialisation has to be done as part of a deep traversal of the live data, and doing that on multiple threads would be pretty impractical (at least for the bulk of the data). Compression and writing to disk are the only parts that could, in theory, be threaded and done once the game has been unpaused, but then you run the risk of finding out that the process failed after the game has carried on being played. That wouldn't be good!

[radcapricorn]

it was already pointed out upthread that just threading the compression and disk write is a 20% time savings.

Nope. What was upthread was SirNukes playing with gzip on his uncompressed save files and drawing arbitrary conclusions from that, and ImperialGood musing on how that might've worked. All of which, in the context of the proposed "grab a snapshot and do the work in background", is irrelevant anyway - first you need to create something to compress (or not), which is the bulk of work, and if we had saving that would've been performing that work in the background it wouldn't have mattered if it took 18 seconds or 22, now would it? Because, you know, it would've all been done in the background.

You know the game already has to traverse and deep copy that data anyway to make an XML file right?

Yes, that's what I'm saying. Unless you can efficiently make a copy of game state, which CBJ is saying that X4 can't do, then you won't get any practical gains from such an approach.

This is basically it. The game's "live" state is, and needs to be, optimised for running the game, not for saving it, so there are lots and lots of pointers and other things that you can't just copy and expect to still work. Serialisation has to be done as part of a deep traversal of the live data...

Which is not to say that this can't be done "at all", but retrofitting the engine for a different approach in storing (and therefore, traversing) the data would mean rewriting, like, "half" the game.

[CBJ]

Which is not to say that this can't be done "at all", but retrofitting the engine for a different approach in storing (and therefore, traversing) the data would mean rewriting, like, "half" the game.

Retrofitting to store data in a copyable form would almost certainly completely kill performance, so no, I don't consider it possible.

[SolidSnakeRUS]

If the game state is a giant graph with random pointers everywhere you're very likely to spend comparable amount of time "just" copying it, to traversing it for saving. You can't simply copy a blob of memory and call it done when all the contents that you've copied still refer to the original "live" version all over the place.

This is basically it. The game's "live" state is, and needs to be, optimised for running the game, not for saving it, so there are lots and lots of pointers and other things that you can't just copy and expect to still work. Serialisation has to be done as part of a deep traversal of the live data, and doing that on multiple threads would be pretty impractical (at least for the bulk of the data). Compression and writing to disk are the only parts that could, in theory, be threaded and done once the game has been unpaused, but then you run the risk of finding out that the process failed after the game has carried on being played. That wouldn't be good!

Ok, now we know what exactly require time. I also tested, dump to file and file compression performs pretty fast in comparison with data collection.
Is it really deep traversal operation take most of the time?
I'm sure, object serialization take most (90%) of the time.
If I am right, than there is big field for relatively easy optimization.

[CBJ]

If i'm correct, than there is big field for relatively easy optimization.

With all due respect, having spent a lot of time actually working on that code rather than just talking about it on forums, I can assure you that if there was "relatively easy optimisation" it would have been done a long time ago.

[SolidSnakeRUS]

Not saying that it easy. We all respect hard work of EGOSOFT team.
Game is really good. I'm buying every new game from EGOSOFT, even if i have no time to deeply dive in

relatively easy optimization.

Relatively to entire game rewrite. Please, don't take it wrong.

Thanks for clarification.

[Rei Ayanami]

The simple answer is : The more game objects and game objects parameters need to be saved, the longer it takes.

In rather simple games like first-person shooters, where you at most have only like 30 characters with no customization and perhaps 20 bullets flying around at any moment while the entire level is static, saving these few dozens of objects into a few bytes is super fast and simple.

On the other hand, having thousands of entities with tons of possibilities of what an entity could have (weapon loadouts, shields config, wares in cargo, damaged subsystems, info for the entire crew, shield status, hull status, current tasks and their settings, current target, commander, settings for turrets, settings for subordinates, station module type and position, station module loadout, station module hull integrity, etc and so on) on the other hand is a massive amount of data, not to mention game objects cross-referencing eachother that needs to be resolved, to make sure no relevant game object gets forgotten in the process.

We had this topic quite a few times in the past already, and the reply is always the same. Quoting the (I hope) most recent reply from CBJ:

I'm afraid saving isn't likely to get much quicker, no matter how many people ask for it on a wish-list. The game universe is a complex structure with complex data, and it simply isn't possible to just take a "snapshot" of memory and save things from that in the background. The game needs to pause while the data is collected for saving.

I think Bernd spoke about this is in one of the XLog videos, too.

the game needs to pause while the data is collected for saving, but not while the data is serialized into XML, compressed, and written to a file. We know save files are ~500MB uncompressed so we can be pretty sure the data being rendered is smaller than that. We have machines with 16, 32, or even 64GB of RAM - making a copy of the data that needs to be saved to the heap and then writing that in a background thread is not infeasible
I doubt it's the best use of developer time but it's not infeasible

So your idea is after the game collects all savegamedata in RAM, while it's writing the collected data to disk the game should continue playing?
I see a big problem with this:
If you allow your game to continue running unpaused while the savegame is being written to disk and a crash or program-close happens (alt+F4 or exit game command), you will corrupt both the current savegame and the savegame that was potentially overwritten.

Pausing the game for saving is not only good for taking a snapshot of the current state, but it also prevents the game from being aborted or going into a potentially unstable state before writing the snapshotted game state has finished.