Split from "Why have you guys traded metal for plastic?" thread

[radcapricorn]

... he was speaking about making a copy of the data in RAM and then save this copy. He didn't directly say (that I can find anywhere) if the actual write-it-to-disk part is during, or after the pause, but what is apparently happening during this "just make a copy in RAM" is that it's making the copy and incidentally inserting some xml markup, while converting some binary stuff to UTF-8 on the side probably. He said the additon of this formatting during the "copy in RAM" part isn't significant from a performance standpoint.

It's not "some" XML markup. It's actually quite a lot. Just to give you an idea, an average moderately-paced 20-hour save would contain:

- about 4 million tags, amounting to more than 34Mb (from a quick grep of opening tags only, i.e. it's underestimated).
- about 5 million quoted strings (i.e. "strings"), ~60Mb. Of those strings, only 35 thousand (0.07%) are unique, adding up to puny 650 kilobytes (1%). That's outrageous overhead, but such is the nature of XML, it's stupidly redundant.

Most of the numeric values, represented as text, are also grossly over-inflated (that is, take up way more space than their binary representation). I haven't the energy to write a script to estimate that particular overhead, at least not right now. But just from the two points above, about a third of the uncompressed save is basically, well, junk, which is completely discarded by the game and is needed purely for upholding XML structure. Analysis of just a couple randomly-chosen chunks in saves shows that at least those chunks could be "compressed" by at least a factor of 4 "simply" by getting rid of XML (that's before employing actual compression such as zlib).

But size is only one side of the issue. The more your on-disk representation deviates from in-memory representation, the more processing power (and therefore, time) you have to spend on conversions. And not just raw conversions (i.e. numbers to text and back), but replicating (when saving) or recreating (when loading) the overall structure. Just a simple example: an array. In memory, that's typically 16 or 24 bytes of upkeep plus the actual contiguous storage, of known bounds. In XML though, no such thing: it's just a stream of tags, which you would have to count one way or another to get at the size of storage that you need for data. Or the aforementioned "strings", most of which are keys for some tables, which implies lookups that are, at worst, O(n) (as opposed to linear indexing which is always O(1)).

Reading even an uncompressed 300Mb save from disk into memory is blazing fast (varies, but would be under half a second on most common hardware, assuming file is "cold"). Traversing it and building the universe from it is not, unless it's a (almost) carbon copy of game's layout.

I just glanced at my paused copy of X4 and it's using almost exactly 4gb of RAM right now. If we "just make a copy" that's doubling our RAM consumption instantly.

No, it's not. The number you're seeing is the amount of memory that allocator(s) which are used by the game (directly or indirectly) have reserved. It's not indicative of the size of actual game state (except to guess that it's no larger). That includes the overhead which allocators naturally create to avoid additional system calls and to create their own acceleration structures for efficient allocations. Plus there are no doubt various ancillary buffers that pertain to submitting and retrieving data to/from the GPU, and all the state associated with rendering, playing sound, etc. etc. In short, there's a lot of stuff that never gets into the save.
"Just make a copy" meant duplicating the structure and contents of game's state that needs to be saved. Depending on how that state is structured, it can be surprisingly easy, or frustratingly tedious and error-prone (and even slower than just directly translating the original into a different representation, i.e. XML). Judging by CBJ's response, it's the latter. Sadly, changing that at this point in time is just not going to fly.

Thanks, CBJ, for clarifying. @Tamina and strask, we understood each other just fine

[strask412]

It's not "some" XML markup. It's actually quite a lot. Just to give you an idea, an average moderately-paced 20-hour save would contain:

Thank you very much for the statistics and other enlightenment; I was previously aware of the rough magnitude of the size cost of the xml stuff (and included that in my rather vague word "some"), but actual (or, as I take it, bounded approximations based on actual measurements) numbers are always helpful.

The number you're seeing is the amount of memory that allocator(s) which are used by the game (directly or indirectly) have reserved. It's not indicative of the size of actual game state (except to guess that it's no larger).

That's of course absolutely correct and I am sorry for posting otherwise, I failed to think prior to writing what I wrote (despite taking some intentional actions, like checking process memory usage, in the course of it!). The ability of the human brain (especially mine!) to NOT think is sometimes amazing. Sorry.

[radcapricorn]

There's no need. The nature of today's Internet conversations sometimes compels one to pedantry

[CBJ]

And be serious 3 seconds ... It do not cost months to have a save file system even one with all you mentioned. I don't even think it took one week to the man who wrote you parser from scratches ... And you can even go with XML if you wished and store a bit more data without repeating again and again the same tags.

Again you are making false assumptions. The parser has nothing to do with this; in fact it's one of the things we use a third party tool (libxml2 to be specific) for, because why re-invent the XML-parsing wheel? I'm talking about going through each of the dozens, maybe hundreds, of objects in code that needs to save something and changing the way they do so, to use some system other than XML. Even taking into account a reasonable level of encapsulation of that process, which we do of course have, that is still a considerable task. Add to that the extensive testing that would be required, and the need to maintain backwards compatibility, then yes, you could well be talking about months of developer time. Even if it were "only" weeks, that time still has to be weighed against the other things that developer could be doing, especially when the likely benefit is extremely limited.

Properties in XML do not required their own tags but are instead inserted in the parent tag between < and >. In your save format it's not, every damn property have it's own tag with the same words copied over again and again. Yes sur it has a cost, not only in memory but as a CPU too, you need to insert all those useless characters again and again, even on a XML perspective.

If you just dump information into the data field using your own internal format then you are back to having to worry about corruption and versioning again as things change, and you lose all the benefits that XML provides. The data field has its uses, for example for long text strings, or for large, static, homogeneous data-sets such as animation offsets, but beyond that it just defeats the purpose of using XML in the first place.

With all due respect, I can't help thinking that the game development experience you say you have must be on much smaller-scale projects than this, because underestimating the amount of work involved in doing this kind of thing is a common mistake for people who have dipped into development without working on a real-world, large-scale project.

[Midnightknight]

Yes you are right, every AAA game use XML for their save game ... I must be the one with issues working on large scale projects ...

Oh by the way, just checked this C lib and

The internal document representation is as close as possible to the DOM interfaces.

lol who spoke about DOM? You know you can have all you asked above without following blindly DOM? This tool is used for structural purpose!! Not data storing.

If you just dump information into the data field using your own internal format then you are back to having to worry about corruption and versioning again as things change, and you lose all the benefits that XML provides. The data field has its uses, for example for long text strings, or for large, static, homogeneous data-sets such as animation offsets, but beyond that it just defeats the purpose of using XML in the first place.

Oh look

<owner factions="khaak xenon" relation="enemy"/>

Here somewhere in my save game you find a way to put 3 properties in one single tag!!! Hurry up solving it, it should be

Code: Select all

<owner factions="khaak">
<standing relation="enemy" />
</owner>
<owner factions="xenon">
<standing relation="enemy" />
</owner>

To avoid any risk of corruption and incompatibility!

[CBJ]

Given your responses, I'm not entirely sure you've understood the explanations I've given, but either way you clearly have no interest in anything other that asserting your position that you are right and the developers are wrong. This is the point at which I stop wasting my free time discussing the subject. Have fun.

[Tamina]

The only one who spoke about DOM was me, (indirectly asking CBJ) if X4 is creating an XML DOM structure, vaguely based on CBJ saying that the savegame represents the internal structure of the game. Again, because I work with XML a lot and it does not cause this big overhead (ofc there is an overhead, but writing a single file is stupidly fast. AFAIK the OS is caching the output stream and writing concurrently? Writing multiple tiny files is a waaaaaaay bigger problem).

[Boringnick]

What happened to my thread? No one cares about file saving formats.

I want my shiny ships.

[Alexraptor]

What happened to my thread? No one cares about file saving formats.

I want my shiny ships.

Easy there, this is a "Split" for this very reason. Your original thread is still here: viewtopic.php?f=146&t=417673

<Insecure http: link changed for secure https: one. Alan Phipps>

[photomankc]

The term is 'Argument from authority'. And if they were soo good like they say they are? Why aren't they working for Egosoft? Or in Devnet helping to fix the game. That's what bothers me personally about these particular arguments.

Another area that drives me nuts is when someone, outside and largely ignorant of the details, proceeds to tell those on the inside they should "just do X and Y to fix Z". Certainly in this world there are tons of incompetent people and poor practices in all areas of business and engineering. Thing is those conditions are not always without cause. There are, very likely, reasons that things developed as they did that involved deliberate choices from conditions only those on the inside know about. Something that I see as sub-optimal may have been the only possible choice among a series of less than optimal choices available. To presume that I can, from the outside, tell anyone how they should 'just do X and Y" seems unbearably presumptuous.

We all know that when people from outside our work environment critique it they are almost always off-base and propose 'solutions' that are unworkable for us and we bristle at that. Yet, somehow people seem born to do it to others even still.

[spankahontis]

The term is 'Argument from authority'. And if they were soo good like they say they are? Why aren't they working for Egosoft? Or in Devnet helping to fix the game. That's what bothers me personally about these particular arguments.

Another area that drives me nuts is when someone, outside and largely ignorant of the details, proceeds to tell those on the inside they should "just do X and Y to fix Z". Certainly in this world there are tons of incompetent people and poor practices in all areas of business and engineering. Thing is those conditions are not always without cause. There are, very likely, reasons that things developed as they did that involved deliberate choices from conditions only those on the inside know about. Something that I see as sub-optimal may have been the only possible choice among a series of less than optimal choices available. To presume that I can, from the outside, tell anyone how they should 'just do X and Y" seems unbearably presumptuous.

We all know that when people from outside our work environment critique it they are almost always off-base and propose 'solutions' that are unworkable for us and we bristle at that. Yet, somehow people seem born to do it to others even still.

People who've bought the game have a right to complain about the product they've bought not working; that's understandable.

Unsure if your critique is towards me or Midnight here?
But like I said, I've seen allot of companies with what you said that promise features and either don't deliver or give a half-arsed product that such questions are asked.
I wasn't questioning your ability as a developer with what i've said, but people do have a right as consumers to ask questions, their money gives them that right, doesn't make them ignorant.
Don't need to be an expert to know that giving their services to Devnet or being so good that Egosoft would be stupid not to acquire their services doesn't take a Brain surgeon to see that it would benefit everyone playing this game if it had more excellent Minds working on improving this game?

But despite that, I fail to see what Mid's argument seeks to accomplish here?
XML seems to be the industry standard for allot of game developers these days? If that's me practicing "ignorance" then i'm sure you can shed some light on that? All the best to you.

[photomankc]

The term is 'Argument from authority'. And if they were soo good like they say they are? Why aren't they working for Egosoft? Or in Devnet helping to fix the game. That's what bothers me personally about these particular arguments.

Another area that drives me nuts is when someone, outside and largely ignorant of the details, proceeds to tell those on the inside they should "just do X and Y to fix Z". Certainly in this world there are tons of incompetent people and poor practices in all areas of business and engineering. Thing is those conditions are not always without cause. There are, very likely, reasons that things developed as they did that involved deliberate choices from conditions only those on the inside know about. Something that I see as sub-optimal may have been the only possible choice among a series of less than optimal choices available. To presume that I can, from the outside, tell anyone how they should 'just do X and Y" seems unbearably presumptuous.

We all know that when people from outside our work environment critique it they are almost always off-base and propose 'solutions' that are unworkable for us and we bristle at that. Yet, somehow people seem born to do it to others even still.

People who've bought the game have a right to complain about the product they've bought not working; that's understandable.

Unsure if your critique is towards me or Midnight here?
But like I said, I've seen allot of companies with what you said that promise features and either don't deliver or give a half-arsed product that such questions are asked.
I wasn't questioning your ability as a developer with what i've said, but people do have a right as consumers to ask questions, their money gives them that right, doesn't make them ignorant.
Don't need to be an expert to know that giving their services to Devnet or being so good that Egosoft would be stupid not to acquire their services doesn't take a Brain surgeon to see that it would benefit everyone playing this game if it had more excellent Minds working on improving this game?

But despite that, I fail to see what Mid's argument seeks to accomplish here?
XML seems to be the industry standard for allot of game developers these days? If that's me practicing "ignorance" then i'm sure you can shed some light on that? All the best to you.

Not a critique of you certainly or really even midnight directly. Just a statement that people should step back and understand that choices made that you disagree with don't ALWAYS represent stupidity, laziness, or lack of care. There are often drivers that you aren't aware of that resulted in the outcomes and choices.

I've got vocal opinions on certain shortcomings in this game, but I **try** not to get into assuming the worst for why they exist.

[Alan Phipps]

I cannot see where this thread is going now in terms of X4 discussion. CBJ has said that the devs will not take further part in it and a general discussion about industry standards and tools does not belong here. Certainly there should be no picking apart or commenting upon other posters or their posting styles.

[spankahontis]

Another area that drives me nuts is when someone, outside and largely ignorant of the details, proceeds to tell those on the inside they should "just do X and Y to fix Z". Certainly in this world there are tons of incompetent people and poor practices in all areas of business and engineering. Thing is those conditions are not always without cause. There are, very likely, reasons that things developed as they did that involved deliberate choices from conditions only those on the inside know about. Something that I see as sub-optimal may have been the only possible choice among a series of less than optimal choices available. To presume that I can, from the outside, tell anyone how they should 'just do X and Y" seems unbearably presumptuous.

We all know that when people from outside our work environment critique it they are almost always off-base and propose 'solutions' that are unworkable for us and we bristle at that. Yet, somehow people seem born to do it to others even still.

People who've bought the game have a right to complain about the product they've bought not working; that's understandable.

Unsure if your critique is towards me or Midnight here?
But like I said, I've seen allot of companies with what you said that promise features and either don't deliver or give a half-arsed product that such questions are asked.
I wasn't questioning your ability as a developer with what i've said, but people do have a right as consumers to ask questions, their money gives them that right, doesn't make them ignorant.
Don't need to be an expert to know that giving their services to Devnet or being so good that Egosoft would be stupid not to acquire their services doesn't take a Brain surgeon to see that it would benefit everyone playing this game if it had more excellent Minds working on improving this game?

But despite that, I fail to see what Mid's argument seeks to accomplish here?
XML seems to be the industry standard for allot of game developers these days? If that's me practicing "ignorance" then i'm sure you can shed some light on that? All the best to you.

Not a critique of you certainly or really even midnight directly. Just a statement that people should step back and understand that choices made that you disagree with don't ALWAYS represent stupidity, laziness, or lack of care. There are often drivers that you aren't aware of that resulted in the outcomes and choices.

I've got vocal opinions on certain shortcomings in this game, but I **try** not to get into assuming the worst for why they exist.

That's cool mate, I agree.

I also share critique of aspects of this game, but the staff are good people, spoken to them enough over the years in this forum that they genuinely want to make the best space sim game out there.
Totally agree that slinging mud at developers doesn't help in the matter.

So much new technology comes out that I can only imagine how hard it must be to keep up with it, that certain flaws or bugs in drivers and engines that aren't foreseen can go about unnoticed until it's too late and can either be fixed or remedied to a point.