Happy 2022! Now, about those save times...

[SirConnery]

This new? Or you talking about the uncompressed save which has mixed results (most people say it does not improve save speed by much)?

Uncompressed save is what I mean yes. For me it cut about a third of my save time. That was in 4.0, don't know how it's now since they already improved save speeds in 4.2 I think.

Also excluding your savegame folder from windows search indexation is gonna have a speed increase.

Takes like 15 seconds for me now to save the game. Which while longer than most games is bearable. In 4.0 before I did any of that the game was saving for a minute or so.

[surferx]

I've played other games that had quicker save times, but reloading the save was hit or miss, and this was significantly more frustrating than waiting a few more seconds for a save to load. So, I'm happy with having a save that I know is going to load correctly and I would hate for Egosoft to compromise the save operation for a few players who just don't have time to spare those few seconds of their busy lives.

[xixas]

Uncompressed save is what I mean yes. For me it cut about a third of my save time. ...
Takes like 15 seconds for me now to save the game. Which while longer than most games is bearable. In 4.0 before I did any of that the game was saving for a minute or so.

On Linux I'm averaging 49 seconds a save right now, but I've got 4 active monitors (playing across 3), so Xorg is always hogging a bit of CPU to keep up.
Still, the save and Xorg are running on different cores, so that shouldn't affect things much.
I'd rather not balloon the saves to 400+ MB on an SSD.

... I would hate for Egosoft to compromise the save operation for a few players who just don't have time to spare those few seconds of their busy lives.

If I'm sitting down to play X4, it's the weekend and I likely expect to spend the next 10 hours in strategy mode... it's not a matter of a busy life.
It's about breaking immersion. Anything above about 5 seconds is a problem -- 30+ seconds might as well be 3 minutes.

[rant]
If I'm saving because I'm about to attempt boarding and am severely out-classed, anticipation wanes.

If I'm saving because I just spent the last 5 hours assembling a fleet to take on a Xenon sector, and we're already in formation waiting on the go signal, I've got plans in mind; I've got a strategy in place; I know what I want 7 different wings to be doing 5 seconds from now; and I've got a properly zoomed map and both info panes open ready to start quick-clicking through things to make it all happen in the order I want to see it go down. I was probably going to issue all those initial commands while the game's paused anyway, so sure, I can go get a cup of coffee while I wait for a save. But in the mean time I lose that click sequence my brain had already worked out because I stopped looking at a screen I have no control over.

The latter of those 2 issues could be cured with command queues, but there's really no fixing the former.

That's doubly true with the quick save I might tap if I get into an accidental dog fight that seems like it might be fun but I really can't say if I'm going to win the day.
I'm mid flight, guns blazing, and I've already got a couple missile locks on me... it's significantly less fun after coming back from a save.
If I were coming in from a load, it'd probably be because I did not win the day, and my immediate action is to high tail it out of there.
"Coming back" from a save shouldn't feel like coming back from a load.
[/rant]

X4 is already highly CPU bottlenecked during gameplay.

I haven't noticed too many CPU bottlenecks playing on newer hardware, but yeah, gotta admit there's a good bit of lag/skipping on 200+ ship OOS battles on the map screen.

Except in theory process fork is a lot faster as it adds as good as no overhead outside of initial page copy that only occurs when a page is first written. ...
The only down side to this is that Windows does not support it.

Hah, I know it'd be greedy to say "well just make Linux faster then"
I'm well aware that parity between OS releases is preferable, and maintaining multiple code paths is a PITA.

While forking may look like a more appealing option than in-place locking code, if you're saying they're flat out off the table then there's no need to follow that train any further.
I do think, though, that entity or large-component level access abstraction is still likely a viable option.
Data offsets as previously described could be mapped to a pseudo-scratch space, with locking only required at a higher level for batch reads, and mapping only performed on initial write (like page copy).
Worst case it'd effectively suffer from the common cache-miss overhead, but that's the same underlying problem with most hashtable implementations and it certainly doesn't stop us from using them.

Don't suppose you could say which ECS X4 uses? If it's third-party that is. That might help narrow down snapshot/COW options.

Alternately, could we determine a ballpark on the maximum expected amount of RAM actually used for active/saveable data?
If the memory consumption of said packed structures is relatively low, maybe this could be approached by doubling the block size of saveable units.
In said scenario, accessors could read unions and then bitshift instead of dealing with cache misses -- e.g.

Code: Select all

union U { std::uint16_t b; std::uint8_t a; }; U u; 

...

Code: Select all

 std::uint8_t x = u.a || ((u.b & 0xff00) >> 8);

While it's not exactly as simple as that, I think it conveys the idea.

[surferx]

... I would hate for Egosoft to compromise the save operation for a few players who just don't have time to spare those few seconds of their busy lives.

If I'm sitting down to play X4, it's the weekend and I likely expect to spend the next 10 hours in strategy mode... it's not a matter of a busy life.
It's about breaking immersion. Anything above about 5 seconds is a problem -- 30+ seconds might as well be 3 minutes.

[rant]
If I'm saving because I'm about to attempt boarding and am severely out-classed, anticipation wanes.

If I'm saving because I just spent the last 5 hours assembling a fleet to take on a Xenon sector, and we're already in formation waiting on the go signal, I've got plans in mind; I've got a strategy in place; I know what I want 7 different wings to be doing 5 seconds from now; and I've got a properly zoomed map and both info panes open ready to start quick-clicking through things to make it all happen in the order I want to see it go down. I was probably going to issue all those initial commands while the game's paused anyway, so sure, I can go get a cup of coffee while I wait for a save. But in the mean time I lose that click sequence my brain had already worked out because I stopped looking at a screen I have no control over.

The latter of those 2 issues could be cured with command queues, but there's really no fixing the former.

That's doubly true with the quick save I might tap if I get into an accidental dog fight that seems like it might be fun but I really can't say if I'm going to win the day.
I'm mid flight, guns blazing, and I've already got a couple missile locks on me... it's significantly less fun after coming back from a save.
If I were coming in from a load, it'd probably be because I did not win the day, and my immediate action is to high tail it out of there.
"Coming back" from a save shouldn't feel like coming back from a load.
[/rant]

Suppose you spend 5 hours assembling a fleet and save , then find out you wasted your time because the save was corrupted because Egosoft tried to cut a few seconds off the saving process. That 5 hours might as well be 5 days.
I get the breaking immersion part, immersion is totally my reason for playing. But it does demand a little compromise sometimes. Well, cheers and Happy 2022 to you too!

[Imperial Good]

I haven't noticed too many CPU bottlenecks playing on newer hardware, but yeah, gotta admit there's a good bit of lag/skipping on 200+ ship OOS battles on the map screen.

It is entirely CPU bottlenecked. You can check out the other threads giving frame rate numbers. Both memory latency and CPU architecture improve performance significantly.

The skipping on the map is intentional and not related to performance. Low attention objects such as those far away from the player's ship update less frequently. This is how X4 can scale to tens of thousands of ships.

Don't suppose you could say which ECS X4 uses? If it's third-party that is. That might help narrow down snapshot/COW options.

As a moderator I am not involved with development at all. It clearly does use an entity component model though given the structure of game data, literally referring to "entity" and "component" types. Given that entity component models are fundamentally not that complex I have to guess that it is written in house, possibly during development of XR which is the engine X4 is based on and that was a large change from the X3 engine, which I think was also written in house.

In said scenario, accessors could read unions and then bitshift instead of dealing with cache misses -- e.g.

Any sort of hacky approach like this for something fundamental decreases maintainability. It is effectively coupling the code to architecture. Although currently X4 is x86-64, it is possible that ARM ports or even other ports could be desirable in the future.

[xixas]

Suppose you spend 5 hours assembling a fleet and save , then find out you wasted your time because the save was corrupted because Egosoft tried to cut a few seconds off the saving process.

No one's suggesting compromising save stability -- that's the whole point of an unalterable snapshot.
Same thing they're going for with a full freeze in the current flow.
I'm proposing that freeze can be backgrounded, and Imperial Good's doing their best to shoot holes in said proposal

As to the 5 hours -- there's at least 10 other saves in there.
Especially experimenting with new loadouts -- I'll quick save after queuing up each set of ships across various shipyards (early game that is, until I've built up shipyards/blueprints).

That's another downside.
I find I save a lot less in a game that's a couple hundred hours in than in a fresh roll where resources are still limited and small mistakes make for big setbacks.
In other words, one needs to save more when they're trying to make the quickest progress.
The long saves were admittedly far more frustrating as a new player. I pushed through because I really enjoy the overall experience.
But generally speaking, new players are far less forgiving than invested players, and dev teams spend copious amounts of time on the "first time user experience".

The skipping on the map is intentional and not related to performance. Low attention objects such as those far away from the player's ship update less frequently. This is how X4 can scale to tens of thousands of ships.

Good good, that's what I was saying I expected was the case with prioritized locality timing.

Any sort of hacky approach like this for something fundamental decreases maintainability. It is effectively coupling the code to architecture. Although currently X4 is x86-64, it is possible that ARM ports or even other ports could be desirable in the future.

What's hacky about allocating sequential blocks? That's not at all arch-dependent.
It's just allocating space for, say, two int values in a sequential block instead of one -- read pointers still point at the same block.
Then the offset descriptors are simply adjusted to account for the extra X bits, same as would be done if an extra int value were added into the binary block.
A standard int read would return the same active value it does now from the same position.
Only difference would be, if a save flag's set, then the second block's checked and the first one's loaded over it (if it's not defined) via an OR operation and a bitshift -- operations with near zero overhead (and certainly aren't arch-dependent).
When it's done with a type union, you don't even need to worry about big/little endianness.

[Imperial Good]

What's hacky about allocating sequential blocks? That's not at all arch-dependent.

Anything related to cache behaviour can be architecture dependent as not all cache schemes act the same.

[xixas]

Anything related to cache behaviour can be architecture dependent as not all cache schemes act the same.

We're not talking about a caching scheme, though, are we? We're talking about active, in-use, real-time data.
If the ECS depends on known virtual memory offsets for the sake of serialization/deserialization, then those blocks have to have hard definitions at runtime, same as a network packet.
Whether that structure's pre-defined manually or generated dynamically makes no difference -- saying "this is a saveable value, so it takes twice the storage" == saying "there are two ints here instead of one".

[shovelmonkey]

I understand very little of this discussion but, for some reason, find it immensely entertaining.

[iznogood]

I was saying that trying to multi-threaded serialisation is non-trivial due to the complexities of encoding object state in a consistent way. You already said this is not the intended case as you only intend for 1 thread to work on serialisation, although from your example it was not clear.

I agree creating a world snapshot, and serializing it to xml, while the game is running might be extremely hard BUT:
Currently the world is paused while this happens, and that might be acceptable if the game at least made proper use of the available cores.

The process of making a world snapshot, serializing to xml snippets, assembling and saving the resulting string can absolutely be done multithreaded, given that the game is paused while it happens.
Currently when i save my game i see around 13% cpu utilization, that's not very impressive for something that is wasting our time.

[Falcrack]

No one's suggesting compromising save stability -- that's the whole point of an unalterable snapshot.
Same thing they're going for with a full freeze in the current flow.
I'm proposing that freeze can be backgrounded, and Imperial Good's doing their best to shoot holes in said proposal

Imperial Good does his best to shoot holes in every proposal, no matter how sensible it may be, it's a recurring theme I've noticed, although he seems to prefer shooting holes in proposals that come from me!

[shovelmonkey]

No one's suggesting compromising save stability -- that's the whole point of an unalterable snapshot.
Same thing they're going for with a full freeze in the current flow.
I'm proposing that freeze can be backgrounded, and Imperial Good's doing their best to shoot holes in said proposal

Imperial Good does his best to shoot holes in every proposal, no matter how sensible it may be, it's a recurring theme I've noticed, although he seems to prefer shooting holes in proposals that come from me!

Most good debates benefit from a devil's advocate. Imperial Good's replies seem to be met with good natured responses from the OP. In fact, from the untrained eye it would seem to be quite the friendly battle. Something lacking these days. Probably why I'm enjoying it.

[xixas]

Most good debates benefit from a devil's advocate. Imperial Good's replies seem to be met with good natured responses from the OP. In fact, from the untrained eye it would seem to be quite the friendly battle. Something lacking these days. Probably why I'm enjoying it.

Hah, and here I thought I was the one playing Devil's Advocate by starting the thread
Anyway -- debate yes, battle no. We're on the same side here. We both want to see what's best for the game.

... Imperial Good's doing their best to shoot holes in said proposal

Imperial Good does his best to shoot holes in every proposal, no matter how sensible it may be, it's a recurring theme I've noticed, although he seems to prefer shooting holes in proposals that come from me!

This was, indeed, intended as a good-natured ribbing.
Moderators are the team's gatekeepers on the forum. It's their purview to catch and reign in the ridiculous before requests get out of hand or threads go off the rails.
If the conversation requires double Devil's Advocate, so be it

I agree creating a world snapshot, and serializing it to xml, while the game is running might be extremely hard BUT:
Currently the world is paused while this happens, and that might be acceptable if the game at least made proper use of the available cores.

So... I start with the big ask, and then you come in with the soft sell?
Suppose I could be on board with that strategy, lol.
Please pardon me if I don't give up just yet, though.

I do, however, believe the conversation's reached at least a temporary impasse.
Given enough time, I can cut a performant path around almost any obstacle, but I think we've gone full circle back to where the conversation began.
Neither of us have eyes on the code, so neither of us can define precisely what the obstacle is... so Imperial Good has no choice but to tow the proverbial line --
i.e. "it's too complicated to explain why, but it's just not possible"

And without further information, that's the safe default -- it gives deference to proven developers' rationale and protects their time to work on what they believe they can accomplish.
What it does not do, however, is bring us any closer to improving or explaining a long standing community grievance.
Improving would be preferable, while explaining would allow further community discussion and/or brainstorming.

Whether the team wants that kind of granular scrutiny at the moment is a separate question.
Community engagement and walled garden approaches are both viable ways to make a great game, and it depends heavily on strength of vision.
It doesn't seem like the team's lacking in product vision, or that they're floundering for new features. The release cycle is steady.
So community engagement's not necessarily the best thing right now, especially while they're pushing hard on 5.0.

In that light, if we're to discuss it, I'd like to take the pressure off and say I think the discussion should be had for discussion's sake -- not necessarily for the sake of getting something done right now.

--

So, someone correct me if I missed something, but I think here's what the community would like to see in regards to the save feature, in order of biggest to smallest ask:

1. Backgrounded saves that don't interrupt play -- the topic of this discussion
2. Action queues -- the ability to use the map and command screens to queue up additional actions as if the game were paused
3. Multi-threaded serialization -- shorten the save time by dividing serialization processing
4. Multi-threaded compression -- shorten compression time for compressed saves by dividing compression across multiple cores

I still firmly believe #1 is possible, but it may require someone semi-specialized in performant data packing, and would certainly be a time sink for both Dev and QA.

I haven't seen too much discussion on the second topic (please link if I missed that thread).
It might be a pain to queue up events like that, but if the UI's already written as an event messenger then it may only require a secondary UI state processor capable of reading and applying a message queue on top of the frozen data in real time.

The rationale against #3 (that it's non-trivial) is effectively the same as for #1, minus the potential performance implications and huge QA time sink.
It requires significant testing, but from QA's side that test is mainly saving and loading complex data sets on a loop -- non-trivial, but not quite "is the -entire game- still running correctly" like with #1.
If non-triviality is really all it is, then it'd be nice to see this scheduled as a future todo instead of writing it off as undoable.

I'm not sure why #4 isn't already a thing. Almost all modern compression libraries have an option to compress across multiple threads.
Do we happen to know what version of what library is used for save compression?

[Ehli]

While I can appreciate the attempt to help Ego, IMO the OP has way too little information of the X4 code to be giving out advise like that. You can't assume ECS, it's a custom in-house made engine. It also feels like a border-line insult to the Ego developers; it has been said many times by Ego they very much looked into ways to pull this off for their engine.

The suggested solution/github code is, IMO, a way too simplistic / is a blue-eyed approach and only shows a lack of understanding their core issue. All it contains is a CRTP base class wasting heap memory with the abhorrent new/delete in the implementation, a wasted CPU write cache by doing writes within the same L1 cache line reads occur (hello mutable base members), and the worst, a mutex eating well over 200 CPU cycles on a fine-grained level. Especially the getValue - even if you use a spinlock instead of a mutex you're performance is now guaranteed abysmal! There's just no way I would ever allow a merge request like that going to development. You don't build up your memory threading model by starting at the bottom/detailed level like that, it will never perform.

Sorry for sounding harsh, but this code entirely disproves commercial experience with high performance threading/memory engines. The problem isn't serialization, the problem is consistency. Besides the terrible memory and performance issues the example code suffers from it also does not tackle the consistency problem. Not a surprise, because that's NOT something you solve with an externally written trick. That's something you need to cater for in your engine design. Many engines are eventual consistent, which allows you do write "background save" algos. The X4 one isn't, check my post with Ego's reply: viewtopic.php?f=146&t=437717&p=5047384#p5047003

It'd also be insulting to Ego to claim their engine design is wrong: besides lacking the information to be able to come to that conclusion you also have to take into account their requirements are vastly different from let's say an unreal engine. Ironically enough their engine has more similarities with compute engine rather than a gaming one. Probably also why they ended up making their own.

[Panos]

This new? Or you talking about the uncompressed save which has mixed results (most people say it does not improve save speed by much)?

Uncompressed save is what I mean yes. For me it cut about a third of my save time. That was in 4.0, don't know how it's now since they already improved save speeds in 4.2 I think.

Also excluding your savegame folder from windows search indexation is gonna have a speed increase.

Takes like 15 seconds for me now to save the game. Which while longer than most games is bearable. In 4.0 before I did any of that the game was saving for a minute or so.

4.20 had a general optimization on game perf including saving. On a 5900X with MP600 nvme drive now takes around 5 seconds to save on Windows 10, compressed, even a 70 hours campaign. That wasn't the case pre 4.20 which needed much more time.

Similar perf when saving on a 970EVO+ on Linux with the same exact system.

Can clearly see this time 100% of the CPU been used to serialize and save the file.

[dmk]

4.20 had a general optimization on game perf including saving. On a 5900X with MP600 nvme drive now takes around 5 seconds to save on Windows 10, compressed, even a 70 hours campaign. That wasn't the case pre 4.20 which needed much more time.

Can clearly see this time 100% of the CPU been used to serialize and save the file.

on my old rig (still 4 real cores, 8 hyperthread one) it takes 1m30sec,
and hibernation is faster, hdd, but during save it is not used more than on 10%,
i.e. no multi-thread usage, and it was fast only in early game, still in late game uncompressed save 800 mb, compressed 95mb

[xixas]

4.20 had a general optimization on game perf including saving. On a 5900X with MP600 nvme drive now takes around 5 seconds to save on Windows 10, compressed, even a 70 hours campaign. That wasn't the case pre 4.20 which needed much more time.

I'm mainly playing on a 5900HS linux laptop these days, and I've encountered no such improvements.
The 49 second average I quoted was on 4.20, ~200 hrs on the current playthrough, ~90 MB compressed saves.
Looks like the last couple saves are in the realm of 5-8 MB smaller than 4.10 saves, but save duration's about the same.

I've got X4 saving to an external SSD atm, but disk I/O certainly doesn't seem to be the bottleneck.

Sorry for sounding harsh...

S'ok, no kid gloves requested.
Glad you got the indignity out on the first shot

But hey, it's not about me, and it's apparent I'm not insulting anyone or trivializing the effort involved.
I'm just making a good-faith effort here on behalf of the player base to get some tangible answers after 3 years and, hopefully, some eventual results.

1. IMO the OP has way too little information of the X4 code to be giving out advise like that.
2. You can't assume ECS, it's a custom in-house made engine.
3. The suggested solution/github code is, IMO, a way too simplistic / is a blue-eyed approach and only shows a lack of understanding their core issue.
4. ...a mutex eating well over 200 CPU cycles on a fine-grained level. Especially the getValue - even if you use a spinlock instead of a mutex you're performance is now guaranteed abysmal!
5. The problem isn't serialization, the problem is consistency.

That's a nice list of things we already covered in this thread.
Appreciate the recap, so I won't bother addressing most of it again, save to say about #3 and #4:

• It was intentionally simplistic for a novice-to-intermediate audience -- it's a general example designed to use as a talking point... and it doesn't mention X4 anywhere in it
• Sounds like in your haste to post you skipped the second code sample

But no matter.
It's common in every passionate forum that no example is ever good enough.
It's either too generic -- and apparently insulting to some -- or it's hyper-specific and shot down for not being reusable.

I believe the rest of your post could be summarized as "I'd rather feign indignation on behalf of EgoSoft than allow any code-oriented discussion," to which I'll simply say that no one's twisting your arm to talk.
You're more than welcome to snap those big boy suspenders you seem to be so proud of and tell us how it's done if you'd like to contribute.
Reiterating how it's not done, however, is not constructive without counter-suggestion.

Point is, the issue keeps coming up.
Maybe you don't like it. Maybe Imperial Good doesn't like it. Maybe EgoSoft doesn't like it.
Like it or not, the issue is still a problem to a lot of players -- and, frankly, most of them don't care how hard it is to fix.
Most also won't bother making yet another forum account to air their grievance -- they'll just stop playing and casually bash the game to their friends in private when asked why they gave up on it.

In game development, "It's too hard" (usually pronounced "It isn't feasible at this time") is a common delay tactic devs feed their PM to de-prioritize a nagging ticket.
But when Steam, or Apple, or whatever app store your game uses to generate the majority of its revenue says jump... well, impossibilities tend to become possible real quick when it affects the bottom line.

It could be naively argued that this doesn't affect the bottom line because it only affects players after they've made a purchase -- but we all know that's not how it works.
Unhappy players -- especially when they feel they're getting ignored or blown off -- affect sales through the grapevine. That's just the way it is.
But, for the sake of argument, let us say that EgoSoft is perfectly happy with their current quarterlies. That still doesn't change the root of things...

If it's going to keep coming up, we should discuss it constructively.

[Panos]

4.20 had a general optimization on game perf including saving. On a 5900X with MP600 nvme drive now takes around 5 seconds to save on Windows 10, compressed, even a 70 hours campaign. That wasn't the case pre 4.20 which needed much more time.

I'm mainly playing on a 5900HS linux laptop these days, and I've encountered no such improvements.
The 49 second average I quoted was on 4.20, ~200 hrs on the current playthrough, ~90 MB compressed saves.
Looks like the last couple saves are in the realm of 5-8 MB smaller than 4.10 saves, but save duration's about the same.

I've got X4 saving to an external SSD atm, but disk I/O certainly doesn't seem to be the bottleneck.

External SSD cannot be compared with PCIe 4.0 internal NVME drive. The former has 1000MB/s at USB 3.2 connection the other 5000MB/s
Also your laptop ram speed is slower than the desktop one, so is the latency. And RAM speed matters on X4. There is a difference even between 3600C16 to 3800C14.

[xixas]

External SSD cannot be compared with PCIe 4.0 internal NVME drive. The former has 1000MB/s at USB 3.2 connection the other 5000MB/s
Also your laptop ram speed is slower than the desktop one, so is the latency. And RAM speed matters on X4. There is a difference even between 3600C16 to 3800C14.

Agreed.
The intended point was that disk I/O shouldn't have any real play here (we're talking ~90 MB).
And that if improvements have been made, they may not impact the majority of players.
i.e. most players aren't running a 5900X/5950X

[Ehli]

I did see the second code link, but it's not better and I did not want to post two reviews here. People in this thread expressed finding it difficult to judge your code: a latin mass. I happen to earn my living with that kind of latin, so the very least I can do is shed another light on it.

Since you more or less asked about your second code link: still the write cache trashing, still the memory barrier costing you many wasted CPU cycles, still the heap allocation and overall poor memory usage, but this time with a costly vtable and a virtual call.

However, all of that does not matter: it's not solving the issue. It can't even serve as an example, sorry. I tried explaining the consistency issue together with a link with more info. I replied to you containing a technical answer, together with a link to a post (by me) which explains the issue and challenge - technically. It also explains what IS needed, architectually. Maybe you don't like the answer, but it is one: saying I didn't is just poor form.

I'd like to end this by stressing again that Ego already explained (too much and too often even) they seriously tried to make this better, spending quite some man-hours. They even improved it a bit recently. And you are a "game developer" - that's what you insinuated at least. Come on man, be nice: you should know better. Developers doing this for their job know you can't judge a book by its cover.. That's just insulting. Call it "feign indignation on behalf of EgoSoft" if you want - I think it's just bad manners. Especially the bitter last 3 paragaphs you just wrote..