Happy 2022! Now, about those save times...

[Matthew94]

While I can appreciate the attempt to help Ego, IMO the OP has way too little information of the X4 code to be giving out advise like that. You can't assume ECS, it's a custom in-house made engine. It also feels like a border-line insult to the Ego developers; it has been said many times by Ego they very much looked into ways to pull this off for their engine.

That doesn't mean there weren't valid approaches they didn't think of or that they dismissed without fully appreciating. You exemplify this strange ignorance I see online all the time: "these people work in field X so that means they're the best in their field (and everyone else is equally as good) and anything they do is an optimal solution. If you think they could have done better then you're wrong and clearly don't understand the situation". Believe it or not but it's possible for people to be shit at their job and I really doubt Egosoft have the kind of pull to be hiring the cream of the proverbial crop of programmers.

[Alan Phipps]

@ Matthew94: One of the nice things being repeatedly said in and about this thread is that the protagonists are being very respectful and polite about each others' views (and about the dev team) during their lively and *deeply* technical discussions. I don't think anyone here wants you interjecting with thinly-veiled insults about other posters and the devs while adding nothing to the thread topic, thank you.

[xixas]

That doesn't mean there weren't valid approaches they didn't think of or that they dismissed without fully appreciating.

I have no choice but to admit agreement with this initial statement, or my argument wouldn't have a leg to stand on.

Otherwise, however, I agree with Alan Phipps.
While I appreciate your defending my point of view, I see no reason to question anyone's development acumen.
I intended this thread to be a good-natured discussion between seasoned "greybeards" like myself. I didn't expect the attention it's received... and certainly not the negative bits.

If I didn't believe the dev team were a capable bunch I wouldn't have bothered to start the thread.
And if I didn't believe the X series draws an above-average-intelligence audience, I wouldn't have solicited assistance.

I'm here to address a problem I genuinely think is fixable -- or get solid answers as to why it isn't.
I'm not here to (nor will I attempt to) defend myself or my professional standing, and I'd prefer if no one else is made to feel that's a requirement.

Developers tend to be a... let's just go with "sometimes overly enthusiastic" bunch, and topics do get heated.
Let's just keep it hot for the right reasons

[vmo]

I didn't intend to stay engaged in this conversation, but since you all are still having a good time with it...

The difficult thing is almost never serializing and writing the data to disk. The difficulty is almost always that the runtime organization of the data looks completely different from the on-disk representation. I would bet that converting from the runtime data model to the on-disk representation is the bulk of the work, which is why saving is CPU bound. The reasons are often along the lines of 1. the on-disk representation would result in abysmal runtime performance and 2. the runtime representation would result in dramatically larger on-disk size and could actually take longer to serialize than the current save times. Ehli's post on the other thread is the explanation that IMO is most likely the closest to the actual situation.

We can only guess at the runtime data model, which is why I said earlier that it's unlikely that we can offer any useful "help" to Egosoft on this matter.

[xixas]

The difficult thing is almost never serializing and writing the data to disk. The difficulty is almost always that the runtime organization of the data looks completely different from the on-disk representation.

Indeed. But "my code is a mess, so it performs terribly" isn't an excuse I'd wager any of us are willing to pass up the chain to our superiors.
If that's the assumption -- and I'm not saying it is -- then there's no harm in us presuming to present a better model.

I would bet that converting from the runtime data model to the on-disk representation is the bulk of the work, which is why saving is CPU bound.

I, good sir, will gladly bet you a Coke.

Thing of it is, it doesn't really matter where the bottleneck lies -- only whether or not it's fixable.
We have a number of industry standards at our disposal, as does the EgoSoft team, and if we know how to wield them...
...well, perhaps we can't write a drop-in replacement, but we can illustrate potential in the hopes it reaches the right link in their proverbial chain.

Ehli's post on the other thread is the explanation that IMO is most likely the closest to the actual situation.

Aye... aye.
And if he or she had portrayed as much in the manner of portent to which they seem to believe to carry themselves, it might have been more well received.
But... frankly... it seemed like every time Ehli took a swing... Ehli punched Ehli in the nose... and I was more than happy to let that happen

Still, we're working in hypotheticals here.

For as much as I believe that Ehli knows the material, the assumption that I do not...
...or more to the salient point, that players have no right to question the current inner workings...
is equal parts hypothetical and ridiculous.

• Imperial Good estimated that there's an ECS (entity-component system) at play, to which we have no direct working knowledge -- simply based on savegame tag notation -- and the assumption at least feels likely, even if they don't consider it a wholesale "ECS" in-house... though if it was a wholesale ECS, then the runtime data model should be inherently easy to serialize.
• I estimated that libxml2 was the most likely serialization mechanism, and upon further inspection, was correct -- and I'm hoping I'm not the only one on this forum who knows how to use it... it's simple, but there are a number of potential performance hacks.
• We've already determined that multi-threading savegame compression pigz-style is at least a viable option given the current library in use (zlib) -- though we've yet to determine the beneficial magnitude of such a change.

I can't say we're certainly going in the right direction.
It's a series of stabs in the dark -- best guesses -- but those best guesses are made with a general assumption of developer skill level, a toe-deep knowledge of available tools, some background in at least tangentially related fields, an internet full of resources and best practices, and, hopefully, some good intentions (yeah, yeah... go ahead and tell me what's paved with what... my sentiment stands).

Everyone chiming in appears to have a different background, and that's great.
But no one person's background -- unless it's as a developer at EgoSoft -- will properly qualify any of us to make hard assertions.
...and even then... wouldn't that just be asking someone stuck in the middle of the Minotaur's Maze for the quickest way out? ...a fellow traveler mired in quicksand to throw us a rope?

No, I don't know that we're moving in the right direction.
I do know that I don't shy away from "impossible" problems...
...and here we've got significantly more than an army of one to work with.

Save times have been bugging players for years, and I never underestimate passionate players.

We can only guess at the runtime data model, which is why I said earlier that it's unlikely that we can offer any useful "help" to Egosoft on this matter.

All we have to do is provide reasonable, viable, and (most importantly) working models to prove that our goals can be accomplished, at least under ideal conditions, and hope that they can make said conditions a reality.

But communication is paramount. Civil discourse has moved a mountain or two before

[vmo]

You're still assuming that the very last step (serialization and writing to disk) is the hard part.

[xixas]

You're still assuming that the very last step (serialization and writing to disk) is the hard part.

Hmm? No.

I'm assuming that an efficient and workable data model is attainable.
I can name or describe at least a few industry-standard ones off the top of my head.
With a bit of time, as a group, we should certainly be able to illustrate at least one or two.

The focus on serialization is a byproduct of a couple conversational convergences.
Namely, my desire for snapshots and iznogood's desire for higher CPU utilization during the savegame process (be that across consolidation or last-leg serialization).

My stance is still that snapshots are possible.
I tried to show that at the most basic level, but there's no reason I can't illustrate it one level up at the full model layer too.
Any further and we're getting into the realm of, as Imperial Good put it, "a worse implementation of Linux process forking"
Depending on performance and the storage required for RPC-style binary serialization, I'd even be willing to compromise on a few seconds up front on the main thread to perform a copy-style snapshot via binary serialization and then a slow-bleed full XML serialization on a background thread if that's an easier path.

But iznogood had a solid point that, effectively, "perfect is the enemy of good", and asking for a vast improvement that requires a much smaller investment is, in all likelihood, a more attainable goal, and therefore laudable.
So I'm trying to give full credence to that notion as well, and approach any code I write with an intent to make it possible to illustrate both paths..

[xixas]

It just occurred to me, I should have thought to say that, code wise, I'm still at the point of DEserialization and proposing potential data models.
We're still a long way off from focusing on serialization.

With all due respect, stop being right and start proving me wrong... with code

[vmo]

And I'll say that with large systems, it's premature to jump to code before you have a solid system design.

[dmk]

You're still assuming that the very last step (serialization and writing to disk) is the hard part.

if binary format would be used (in worst case binary XML) that would give some speed(up to 70%) turning memory data to save,
if parallelization (for example 8 threads (would work well even on my old pc) and all sectors pushed through working threads (first thread which finished sector takes next one from pool)
then it would be another 5x speed increase, and final compression can be done by separate thread or process.

personally i would be happy with game on pause, each player action pushed to queue, which do executes after save,
i.e. while save happens player can issue orders, edit stations, do whatever is possible while game is on pause, just result would be delayed too(until save finishes).

[xixas]

And I'll say that with large systems, it's premature to jump to code before you have a solid system design.

If the uninitiated can talk themselves in circles, we're to speak in Mobius Strips tied into Gordian Knots, is that it?
We've had that conversation. I stand by prototyping and fail fast methodology using well-tread industry practices as trumping 3 years of directionless conversation.

Arguing that it's impossible to make assumptions about underlying systems and simultaneously that solid system design is requisite is simply saying "at no point can we reasonably offer sage advice, so let us leave it to the sages."

At this point it feels pedantic to say so, but I (and I believe soundly) disagree.
Were I to write up a white paper every time a task was requested that I've done 50 times 40 ways, I'd be a technical writer, not a developer.
To borrow an old phrase from around the office, some people just can't code themselves out of a paper bag.

I say stop trying to untie the knot with a proverbial pen, and just cut the thing.
Either you know the subject well enough to jump straight into code, or you don't know it well enough to argue against doing so.

if binary format would be used (in worst case binary XML) that would give some speed(up to 70%) turning memory data to save,
if parallelization (for example 8 threads (would work well even on my old pc) and all sectors pushed through working threads (first thread which finished sector takes next one from pool)
then it would be another 5x speed increase, and final compression can be done by separate thread or process.

While these numbers feel completely made up, I completely dig the sentiment

One of the things I'm hoping we can illustrate is that serialized binary objects with proper referencing can be written and read back in any order by any number of threads.
It deviates from the current XML-driven design, but without any specific input to the contrary, who are we to say that's necessarily undesirable?
I don't want to presume XML was chosen arbitrarily, but a well-formed, architecture-independent binary format can be translated to a rigid XML structure on a whim.

personally i would be happy with game on pause, each player action pushed to queue, which do executes after save,
i.e. while save happens player can issue orders, edit stations, do whatever is possible while game is on pause, just result would be delayed too(until save finishes).

I think a lot of people (myself included) would be satisfied with this outcome.
For as much as I can bluster on about tapping F5 "mid-flight, guns blazing"... 99% of the time I'm saving from the map screen.

[vmo]

I don't expect to change any minds here so I won't press further. Some things you learn from working on (and designing) big enough systems. I know that sounds like I'm trying to be a "sage", and there's no reason to believe some random dude on the internet, which I am at this point.

I'll just offer one piece of advice which I actually do have some professional authority on (whether you believe that claim or not ). If you all apply to any of the leading software companies at a mid level position - say, your "senior software engineer" or higher - I'd recommend making sure you have the right design before coding. This is a common way to miss the solutions to interview problems, or even miss the whole point of the problem altogether. I rejected more than one candidate's job offer for this last week.

[Ehli]

I don't expect to change any minds here so I won't press further. Some things you learn from working on (and designing) big enough systems. I know that sounds like I'm trying to be a "sage", and there's no reason to believe some random dude on the internet, which I am at this point.

I'll just offer one piece of advice which I actually do have some professional authority on (whether you believe that claim or not ). If you all apply to any of the leading software companies at a mid level position - say, your "senior software engineer" or higher - I'd recommend making sure you have the right design before coding. This is a common way to miss the solutions to interview problems, or even miss the whole point of the problem altogether. I rejected more than one candidate's job offer for this last week.

Exactly this.

Also not here to start a fight. The amount of assumptions thrown here just dazzle me!

Dear xixas,

Without trying to sound too American (obviously I'm not) I'm rather experienced with this problem space: I'm a principal architect working in the R&D of one the biggest European tech companies. I'm the lead of a high performance STM I designed and is used in production by a large amount of enterprises. What is STM? See https://en.wikipedia.org/wiki/Software_ ... nal_memory . Yes, I like wiki links; they introduce people to new architectural concepts and are more neutral than any of us can come across. Consistency is not an easy topic.

I programmed in C/C++ for the last 25 years, of which the last 15 as a full time job: I see you put effort in your code, the enthusiasm, time spent to make it better. I have no problem talking about code.. you complain I did not, but did give you feedback about yours. If this thread was about creating an eventual or loosely consistent engine (where your code can be an implementation for) I would've suggested corrections, e.g. the CRTP ( https://en.wikipedia.org/wiki/Curiously ... te_pattern ) is ruined by the "new T": it's unlikely all their entities can be constructed without arguments. Use unique_ptr, not raw pointers. Use modern C++ (17 or 20), not C++98. Using 98 like that would give you a hard time applying for a senior software engineer position in my country: you would not get past the test if you don't automatically use it. Design wise I would've preferred to not copy it in a base, but outside. Using the copy assignment like that is tricky: developers on your team will create buggy code if A has a B field and they both derive from your class. Code design should take this into account.

Anyhow, as vmo said, big topics like these need architecture first. The coding samples you seem to be so fond of comes after you defined the problem space. This is why all of my answers were about architecture. How about thinking about that first? I already tried to explain the architectural issue in my link / presented MVCC as a theoretical (but not practical!) solution. The first code block under "Proposed language support" of the STM wiki also contains a nice simple example of the consistency problem. We also have Egosoft confirming it's the intertwined consistency why they need a full lock (not fixed with the here proposed lock on a small owned closure - for all we know their object structure is one huge directed graph with a large amount of edges).

In the "MVCC" topic I linked before, Egosoft also explained serializing is small part of the performance pie-chart: the rest is in reading the world in a consistent manner. As a result, zooming in on XML, JSON, Protobuff.. it's all irrelevant when they literary stated it's the reading a consistent state which is the biggest bottleneck.

I tried to "steer" the debate into first talking about architecture and you did want to "be technical", so let's make the "consistent save" topic smaller by zooming in on the main issue. Let's take a directed acyclic graph (which probably is undirected and has more edges in reality, but ok). Let's take the image of my favourite website: https://en.wikipedia.org/wiki/Directed_acyclic_graph
Let's use that image, with the following constraints/statements:
- All objects in the game world are part of the same acyclic graph, which IMO is fair to assume given Ego's replies in the original thread
- There's only 1 thread (which is logical) operating on this graph (Ego stated somewhere the main computation is single-threaded)
- This thread receives the game events which will modify sub parts of this graph, e.g. "C" will be deleted, "C" owns "D" and thus will delete along with it, both replaced by "F" and "G"

Now apply all suggestions in this thread to this behaviour. You will quickly see that taking a lock on a part will not work, because there are always other closures/subsets being edited. There's only ONE solution to getting the entire graph in a consistent manner and that's by putting the mutex around the entire closure. Well, that's what Ego is doing: it freezes the thread and traverses the entire graph. This prevents nodes and edges you already copied from being modified and having new edges to nodes you STILL have to copy.

This was my last-ditch effort to explain why this is a large issue and certainly not one with a solution without having crippling penalties elsewhere (there's no free lunch!). No memory dump, plugin or base type with a memory barrier is going to solve this. The "public available" solutions to these are MVCC (too expensive for their operating use-case) or an eventual consistent memory engine (probably a hefty engine rewrite). If you do have an easy solution to this "theorem" I would sincerely advise you to start your own company: you will be very rich. Or drop me a PM if you want a job in this field.. got a dozen sw engineer vacancies we need to fill for our R&D: we'll even help relocate you!

I realize my posts in this thread can come across as cocky/derogatory (not a native English speaker, so insert proper synonyms here..), sorry about that. This stuff is much better done face to face in a room with a whiteboard. Writing about subjects like these to people you don't know on the internet always make it sound much worse. I'm not at all bothered by the pot shots from you and Matthew94.. Even though I'm disappointed they were not about the technical subject at all, but generalistic statements about companies and their developers.

[Cdaragorn]

That's why I think I'm going to start experimenting with Protobuf first.
Protobuf messages compile directly to C++ classes to get us closer to Imperial Good's way-back suggestion of an entity-component-based demo.
But more importantly, Protobuf binary serialization averages a literal order of magnitude (10x) faster serialization time over XML.
It also provides message class reflection, making it a bit easier to quickly additionally serialize to XML without writing a whole custom serializer -- so hopefully we can perform some side-by-side comparisons.

I'd really rather not write over 300 custom to-sql serializers
Still, if you've got an idea how to quickly automate that process from the base field data, it wouldn't hurt to experiment with sqlite too.
Though I'm doubtful it'd outperform any of the RPC-oriented protocol buffer formats (Protobuf, Thrift, Bond, etc), which are all specifically designed as high-performance XML serialization alternatives.

I've looked at Protobuf before but didn't use it because it doesn't allow random access to the data. Basically it suffers the same weakness that any text format does because it's order dependent. If you can't truly split the data up you can't use good Data Oriented Design patterns to speed the data handling up.

Also there are ways to avoid having to write that much custom stuff but when you're talking about performance you're also going to have to accept handling more custom code. Still you can make it a lot simpler than it sounds.

Storing it all in memory first (snapshot style) would just alleviate the speed issue.
At that point, even if serialization took twice as long, if it ran at a lower priority than the rest of the game it'd be a much smaller impact on the average user.
Interesting side effect of that too... if the user quick-loads while serialization is still occurring, the snapshot's still available in memory and could just be hot-reloaded

Storing everything in memory first is a nice theory but in practice when working with this much data that actually creates problems that prevent it from getting the speed you're expecting from it. The best system I've worked with involved streaming the data out of that memory at the same time as you were putting it in so it never takes up too much memory. That gets back to the need for true random access.

I'd love to help if you have some data examples to work with. I guess I could pull apart a save file to come up with an example. I admit I'm having a hard time having much oomf to try just because I don't feel like it's likely to go anywhere.

[Falcrack]

Without trying to sound too American (obviously I'm not)

What is that supposed to mean?

[Alan Phipps]

Re the American comment: let's not disappear down that rabbit hole. I agree that it was inappropriate and shouldn't have been said, but let's leave it at that. Back on topic please.

[BlackRain]

Without trying to sound too American (obviously I'm not)

What is that supposed to mean?

Yeah, what is that supposed to mean? lol. Okay, I won't.

[Ehli]

Re the American comment: let's not disappear down that rabbit hole. I agree that it was inappropriate and shouldn't have been said, but let's leave it at that. Back on topic please.

Ah, heh. That was referring to their cultural trait (self promotion being socially acceptable), e.g. https://theprofessorisin.com/2015/02/06 ... dont-brag/ My American colleagues think it's strange we don't emphasize our strengths during business talks, while e.g. my French colleagues think the opposite is not done. It was just there to indicate it's outside my comfort zone to write like this - nothing more.

[BlackRain]

Re the American comment: let's not disappear down that rabbit hole. I agree that it was inappropriate and shouldn't have been said, but let's leave it at that. Back on topic please.

Ah, heh. That was referring to their cultural trait (self promotion being socially acceptable), e.g. https://theprofessorisin.com/2015/02/06 ... dont-brag/ My American colleagues think it's strange we don't emphasize our strengths during business talks, while e.g. my French colleagues think the opposite is not done. It was just there to indicate it's outside my comfort zone to write like this - nothing more.

I see, I understand, those people are just the most visible (which would be very much in the nature of "self promotion"). However, I wouldn't say they are the majority. Most Americans aren't like that in my opinion, especially depending on what part of the country. There are many wholesome, humble, polite and respectful Americans too, it is just that they don't get any visibility. In the world we live in, those kind of people aren't interesting or/and don't make good stories.

[xixas]

Edit: This is a code-oriented post, I promise... just gotta skim down past the fluff
Here's the Code link from below for quick reference (Real-time DAG snapshotting, No Locks)

--

::yawn::
.
.
.
::stretch::
.
.
.

Oh man... as an American, I must admit, I quite enjoyed my 3 day weekend. Hope it was as nice for "y'all" over there "out yonder" on the other side of the pond

Though I have to say, I didn't expect to come back to a thread derailed by the moderators.
"While there's mice away, the cats will play," is that it?

I don't expect to change any minds here so I won't press further. Some things you learn from working on (and designing) big enough systems. I know that sounds like I'm trying to be a "sage", and there's no reason to believe some random dude on the internet, which I am at this point.

S'ok. Random dudes on the internet are what makes it the internet.
I was referring to EgoSoft as the "sages". But you do you. Pick your title. I don't mind in the least

Anyway, your work advice is totally sensible for anyone below SSE.
I only work on referral, and have for years. I'll give you the first week for free to test my mettle... but I don't think anyone's asked me a real coding question in an interview in 20 years.
Personal opinion -- any SSE+ who's not working purely on a referral model is doing something wrong.
Just my 2 cents.

--

Dear xixas,

If you'd started that way to begin with I think we'd have gotten off on a much better footing

Without trying to sound too American (obviously I'm not) I'm rather experienced with this problem space: I'm a principal architect working in the R&D of one the biggest European tech companies. I'm the lead of a high performance STM I designed and is used in production by a large amount of enterprises.
...
I programmed in C/C++ for the last 25 years, of which the last 15 as a full time job

That's great. So take a few minutes out of your evening to throw a couple code snippets together and illustrate what you believe to be the issues.

• I see you put effort in your code, the enthusiasm, time spent to make it better.
• Use modern C++ (17 or 20), not C++98.
• Using 98 like that would give you a hard time applying for a senior software engineer position in my country.
• This is why all of my answers were about architecture. How about thinking about that first?
• drop me a PM if you want a job in this field.

That's a pretty condescending list... and I'm going to ignore it, because this is your most polite post to date.
Truly -- if you're trying to be more polite, I'll try and respond in kind -- be more personable and what not... we'll see how it goes.

This stuff is much better done face to face in a room with a whiteboard.

I'll bring the beer

Look, I've been writing C++ for 22 years.
Before that it was mostly BASIC, COBOL, and Fortran.
I've got a background in aerospace -- mostly air traffic control and satellite/near-earth-object tracking -- so I'm pretty used to dealing with large quantities of real-time data.

I got into game dev because I didn't like working under clearance, I got tired of contracting, and I've been getting hounded for the last decade or so... and it sounded like fun! (and it is... most of the time)
And the world fell apart... so, working from home instead of a lab ain't so bad.

Anyway, I work in a lot of languages, so we can switch out if your hung up on my semantics -- I'm down for pretty much anything but Erlang, Go, or Rust
Seems like EgoSoft's a C++ shop, so I'm trying to keep the core there.
Tooling's a bit quicker in Python/JavaScript/Perl/etc. so I may switch it up a bit as we're scaffolding (if you decide to stick around, that is).

Unfortunately I feel your perception so far is entirely based on examples designed for discussion by a general audience -- and maybe a healthy dose of just thinking you know better.
If you try and focus on the concepts instead of the execution I think we'd get a lot further.

For future reference, I write most of my examples in C++11 (not 98), both because I regularly find myself working on 10 year old projects -- maintenance is 70% of the product lifecycle after all -- and because it'll compile on just about any system onlookers might have available (I'm looking at you, macOS).
The purpose is example, not production. But, I'm happy to step things up a bit if you wanna join the game

--

On to the code.

I would've suggested corrections, e.g. the CRTP ( https://en.wikipedia.org/wiki/Curiously ... te_pattern ) is ruined...

Please stop saying this.
A CRTP requires a derived class inherit in recurring template form -- thus the name -- e.g.

Code: Select all

class Derived : Base<Derived>

...which it does not. It's simple virtual inheritance.
There are no subsequently derived classes, just wrapped types.
Ergo, no CRTP.

Use unique_ptr, not raw pointers.

Hah. Ok, so you want to see why I didn't use std::unique_ptr?
I'll gladly take a few minutes and type out a direct translation.

Code: Select all

template <typename T> class Saveable : public virtual Updateable {
protected:
    mutable std::unique_ptr<T> a, b;

public:
    Saveable() { a = std::make_unique<T>(); b = std::unique_ptr<T>(a.get()); }
    Saveable(const T& t) : Saveable() { *a = t; }
    Saveable(const Saveable<T>& o) : Saveable() {
        a = std::make_unique<T>(*(o.a));
        if (o.a.get() != o.b.get()) {
            b.release();
            b = std::make_unique<T>(*(o.a));
        }
    }
    ~Saveable() { if (a.get() == b.get()) b.release(); }

    Saveable<T>& operator=(const T& t) { setValue(t); return *this; } // Direct assignment
    operator T() { return getValue(); }                               // Direct use as type
    T operator +() { return getSaveValue(); }                         // Retrieve save value with a '+' prefix

    void setValue(const T& t) { if (isSaving && a.get() == b.get()) { b.release(); b = std::make_unique<T>(); } *b = t; }
    T getValue() const { return *b; }
    T getSaveValue() const { return *a; }

    void update() const override { if (a.get() != b.get() && !isSaving) { a = std::unique_ptr<T>(b.get()); }}
};

That's not better. That's much worse. It's a mess filled with gets and spurious release calls.
And it performs exactly the same, since std::unique_ptr has no overhead once optimized.
Though it performs significantly worse unoptimized (so smart pointers aren't awesome for debugging speed).

Anyway, I didn't use them here because I'm specifically doing with pointers what smart pointers were designed to stop developers from doing.
Not because it's wrong, but because junior devs always spring leaks.

And for the record, I used virtual inheritance because, compiled, the vtable overhead is one assembly call.
For someone who doesn't even want to be writing code yet, you sure do seem like you want to jump to the end.
Optimization is usually the last step in code design... but we'll get to that in a minute.

Anyhow, as vmo said, big topics like these need architecture first. The coding samples you seem to be so fond of comes after you defined the problem space. This is why all of my answers were about architecture. How about thinking about that first?

Back in the day, before the dot-com bubble, I was a hacker for hire.

I think in code, topology, and on-the-ground logistics. Not design patterns, not "problem spaces", not DAGs.
I'm accustomed to zero-day timings and people who -do- instead of -ask- or -plan-.
I don't like diagrams, flow charts, or spread sheets. I don't like red tape. Though, you're right, I do like white boards
What I like most is to do the job -- to do it right -- but more importantly to do it right now.

Let's take the image of my favourite website: https://en.wikipedia.org/wiki/Directed_acyclic_graph

Well... I already said I don't like thinking in DAGs...
But sure, I'm a sport. Let's do so.

Let's use that image, with the following constraints/statements:
- All objects in the game world are part of the same acyclic graph, which IMO is fair to assume given Ego's replies in the original thread
- There's only 1 thread (which is logical) operating on this graph (Ego stated somewhere the main computation is single-threaded)
- This thread receives the game events which will modify sub parts of this graph, e.g. "C" will be deleted, "C" owns "D" and thus will delete along with it, both replaced by "F" and "G"

Oh good, we're getting into specific constraints now.
I can work with this.

Now apply all suggestions in this thread to this behaviour.

Sure, I can do that.

You will quickly see that taking a lock on a part will not work, because there are always other closures/subsets being edited. There's only ONE solution to getting the entire graph in a consistent manner and that's by putting the mutex around the entire closure.

There's only one way?

How about this completely different way? (compilable gist with comments, C++11, no dependencies)

You know, following such narrow constraints... and applying all the suggestions from this thread... I had to think about that one for a few minutes!
Thanks. It was a neat little puzzle to end a long weekend

Quick summary... and tell me if I missed any of the suggestions from this thread:

• It's a straight copy of the DAG from your link -- the first swap is even a C-owned-D to F and G pair... at your specific request
• Single producer/consumer model -- only one producer thread working on the graph -- though it works for any number of consumers you like
• Applying snapshots both incrementally at a low level (sub-vertex values) and last-minute at point-of-save on the graph itself
• Serializes the graph to XML (intentionally slowed down to two edges per second) -- prints the XML to stderr, so run it with 1>/dev/null if for some reason you just wanna look at XML
• Performs 3 serialization passes on snapshots at different points of play and then performs a single-thread stress test
• Not a single new or delete call -- It's not realistic to do it all on the stack in production, but works fine for a simple DAG like this
• No locks (didn't even include mutex)
• Just went with simple request-control sync via overlapping atomics
• No event queues to bottleneck
• Was originally going to, but passed on memory barriers -- just not worth the maintenance complexity for something this simple
• Didn't even use a condition variable -- though it may speed the transition-to-save ever so slightly if I had used one, it's not even worth the extra include
• When you say performance, you seem to mean speed (without any regard for memory consumption). I tend to work on a lot of low memory systems, so that's usually factored into performance for me, but I disregarded it here -- nearly double the memory usage for 800% increase in straight write performance but only during a save by dropping the pointer shifts and heap allocations -- though the new/delete model outperforms it in standard operation on a predetermined graph in both speed and memory efficiency... because they're never called except when a save occurs.
• I left in the vtable overhead because it was a lot simpler than multiple registrars, and it only runs once per snapshottable item per save -- so nearly nonexistent.
• Granted it's only rotating through 6 vertices and 8 edges, but the performance looks alright. One "iteration" translates to swapping a random vertex, zeroing all the edge counts in the graph, recounting and storing the edge count for each vertex, and snapshotting the entire graph for save. I'm getting about 600k iterations/second on a single thread (on my 10 year old test rig)

Took a bit to type it out, but I probably haven't written graph code from scratch in 10 years, so that was kinda fun
Anyway, give it a spin when ya have a few.
Maybe it'll open up another idea or two.

Well, that's what Ego is doing: it freezes the thread and traverses the entire graph.

As you said, there's no free lunch. This strategy offloads the majority of the snapshot management from the point of save across all the value writes.
Again, I think the new/delete method's more efficient, because most of the time you're not in a save state -- but it would bog the save state time significantly more than this method.
This way costs more RAM permanently, but write performance actually improves during save state, so if the RAM's there to spare it may be the better option.

This was my last-ditch effort to explain why this is a large issue...

I'd hate to see you give up just as you're starting to join in
Look, rocky starts aside, if you know the material as well as you say you do, what's it gonna hurt to write a couple tidbits and try out a few things?

Anyhoo, that's about all the time I have this morning.
If I forgot to comment on anyone else's specific notes, I'll get back to ya.

Oh, and quit raggin' on Americans. You're breaking the mods

Edited: A few times, to be more polite and on-topic. I don't want things like throwing around titles and experience to detract from the main point here (actual code)