Split from "Why have you guys traded metal for plastic?" thread

[Buzz2005]

My game save for about 5-9 seconds. Which is faster than some of my citys in City:Skylines
And Ego using XML might be based on the fact that they used it since X2(?) for multiple things and so they get proficiend and used to it.

I cant see a damn reason why using XML is a "lazy" or "bad" thing....

OMG yes my latest city aka the biggest and longest of saves does take some time, and the fps is like 15

[Falcrack]

Save times are only slightly annoying to me. Not something that detracts from my enjoyment of X4, and definitely not something that I would want to takes weeks or months of development time away from more pressing gameplay issues.

[radcapricorn]

This seems to be a big problem on the forum in general and it isn't just with any particular person but from many that I have noticed. There tends to be many so called self introduced "experts" who think they know better than anyone else how X, Y, and Z should be...

...I have been around way longer than you so I saw all of this and have first hand experience. You clearly came way later and your view/perspective is at a much different time...

...If you can't see the illogical aspects of your argument in regards to this, then there is no point in me discussing with you anymore...

...I don't think you get that...

...Since you are not a modder and don't know anything about the code at all, I guess I can understand why you can't see the work put into it...

Hypocrite much? Seriously, please try to measure up to your own standards before judging others.

EDIT: Misunderstood. Yikes

[radcapricorn]

Yes, the game is "frozen" while you save. If it weren't then the game state would be inconsistent, containing data representing states at different times across the duration of the save process...

Can't you reduce the pause duration by taking just the snapshot of current state (which is what requires pause for consistency), and offload its processing and subsequent I/O to background? Assuming there are provisions in the engine for doing that, of course. E.g. if your state is structured in such a way that copying it wouldn't be any faster that straight-up conversion to XML, then it'd be moot.

Regardless, if anything else fails, please do try to find it in your heart to fix this. I.e. pause simulation before taking control away, and unpause it only after the game is again interactive. Please!

[BlackRain]

This seems to be a big problem on the forum in general and it isn't just with any particular person but from many that I have noticed. There tends to be many so called self introduced "experts" who think they know better than anyone else how X, Y, and Z should be...

...I have been around way longer than you so I saw all of this and have first hand experience. You clearly came way later and your view/perspective is at a much different time...

...If you can't see the illogical aspects of your argument in regards to this, then there is no point in me discussing with you anymore...

...I don't think you get that...

...Since you are not a modder and don't know anything about the code at all, I guess I can understand why you can't see the work put into it...

Hypocrite much? Seriously, please try to measure up to your own standards before judging others.

None of those posts are hypocritical. First, the initial comment I made had to do with perspective. I don't think you got at all what I was saying. What I was trying to say was that because I had been around longer and saw every release of every X game and went through the experience of having immensely buggy games at launch, missing features, the evolution of each of the X games, etc. this gave me a better perception of the X series overall. Which it does, and yes, experience/time does matter in this case. You came around at a time when X3 had already evolved and turned into a much different game than it was at release and then wanted to point out the flaws in X4 which hasn't been out for even a year yet. Not to mention, you couldn't even answer the question I asked and just ranted about bugs. Your comments, I have already seen thousands of times for pretty much every single X game although some of the concepts/features may have changed. You completely didn't understand this I guess.

The second thing, yeah your argument was illogical in my opinion and I got out of the discussion. This is called a personal choice and opinion, did I claim to be some kind of expert? Nope, but you certainly did, should I quote that? Not to mention, you are still being illogical now. I mean, you can't even argue a point without misconstruing my words and falsely accusing me of something that isn't even true. Nothing hypocritical in any of my comments, but it is a bit of irony that you are doing exactly what I said in my post here (cherry picking pieces of evidence which don't even fit the context).

Lastly, you have not a single modding credit to your name. If you do, please point it out for me. You claimed to be some expert on coding, but what have you contributed to the community to prove that? Nothing. Well guess what, I have contributed to the community quite a lot in fact. I have modded the X series since X3. Have you? Maybe you have and I just don't know it, please point me to the mods you have worked on. You just "Claimed" you were a programmer and looked at their code. Sorry if I don't take your word for it. I can prove I have done it though and yeah i know there are issues, but it is still impressive work in my opinion. So again, no hypocrisy there. I find this quite humorous though.

Well, whatever. This is another conversation I don't really want to post more on. I really have to stop wasting my time. So, go ahead, I'm done.

[CBJ]

Can't you reduce the pause duration by taking just the snapshot of current state (which is what requires pause for consistency), and offload its processing and subsequent I/O to background? Assuming there are provisions in the engine for doing that, of course. E.g. if your state is structured in such a way that copying it wouldn't be any faster that straight-up conversion to XML, then it'd be moot.

State information is not stored in one place. The savegame is effectively the snapshot, with the structure of the XML corresponding fairly closely to the encapsulated code and datasets that generate it. Serialising it in a different format isn't going to change much.

On a moderation note: please stop the personal comments and the discussion of moderation.

[Tamina]

Can't you reduce the pause duration by taking just the snapshot of current state (which is what requires pause for consistency), and offload its processing and subsequent I/O to background? Assuming there are provisions in the engine for doing that, of course. E.g. if your state is structured in such a way that copying it wouldn't be any faster that straight-up conversion to XML, then it'd be moot.

State information is not stored in one place. The savegame is effectively the snapshot, with the structure of the XML corresponding fairly closely to the encapsulated code and datasets that generate it. Serialising it in a different format isn't going to change much.

If I understood him correctly, he was speaking about making a copy of the data in RAM and then save this copy, while the game keeps running.

I think it is weird that it takes so long to save. I deal with way bigger XML files (gigabytes) on a daily basis and it takes seconds to load them, parse them, process them and save them again. Then again, I have no clue what is happening in the background in the game but just reading the current state, doesn't sound that complicated at first.

[strask412]

Can't you reduce the pause duration by taking just the snapshot of current state (which is what requires pause for consistency), and offload its processing and subsequent I/O to background? Assuming there are provisions in the engine for doing that, of course. E.g. if your state is structured in such a way that copying it wouldn't be any faster that straight-up conversion to XML, then it'd be moot.

State information is not stored in one place. The savegame is effectively the snapshot, with the structure of the XML corresponding fairly closely to the encapsulated code and datasets that generate it. Serialising it in a different format isn't going to change much.

If I understood him correctly, he was speaking about making a copy of the data in RAM and then save this copy, while the game keeps running.

Heh.
And if I understood CBJ correctly when he said this:

State information is not stored in one place. The savegame is effectively the snapshot, with the structure of the XML corresponding fairly closely to the encapsulated code and datasets that generate it. Serialising it in a different format isn't going to change much.

... he was speaking about making a copy of the data in RAM and then save this copy. He didn't directly say (that I can find anywhere) if the actual write-it-to-disk part is during, or after the pause, but what is apparently happening during this "just make a copy in RAM" is that it's making the copy and incidentally inserting some xml markup, while converting some binary stuff to UTF-8 on the side probably. He said the additon of this formatting during the "copy in RAM" part isn't significant from a performance standpoint.

As to the overall level of surprise expressed by people about duration of the save process, think of it this way: I just glanced at my paused copy of X4 and it's using almost exactly 4gb of RAM right now. If we "just make a copy" that's doubling our RAM consumption instantly. While of course the CPU is busy traversing the game's data in RAM and making the copies, I'm sure that there are also potential memory bandwidth concerns and possibly this sudden allocation of 4gb could trigger the OS to swap some other application out to disk or whatever, resulting in additional delays. I'm not trying to express "this is what happens", because I don't know all the specifics, but I'm trying to express "it is actually reasonable to expect that a lot of work is happening, and the duration of the save process doesn't particularly surprise me".

Edit: The comparison you make to your daily work:

I deal with way bigger XML files (gigabytes) on a daily basis and it takes seconds to load them, parse them, process them and save them again. Then again, I have no clue what is happening in the background in the game but just reading the current state, doesn't sound that complicated at first.

may not be equivalent for a number of reasons; the one I can think of off the top of my head would be the memory requirement above. I don't know your code or anything, but from the words you used (load/parse/process/save) it seems unlikely to me that the entire file is stored in memory at once, it's probably got the input and output files both open for i/o and only the current chunk(s) of the input need to be in memory, once written to the output file they can be forgotten.

[Tamina]

Edit: The comparison you make to your daily work:

I deal with way bigger XML files (gigabytes) on a daily basis and it takes seconds to load them, parse them, process them and save them again. Then again, I have no clue what is happening in the background in the game but just reading the current state, doesn't sound that complicated at first.

may not be equivalent for a number of reasons; the one I can think of off the top of my head would be the memory requirement above. I don't know your code or anything, but from the words you used (load/parse/process/save) it seems unlikely to me that the entire file is stored in memory at once, it's probably got the input and output files both open for i/o and only the current chunk(s) of the input need to be in memory, once written to the output file they can be forgotten.

It's actually loaded as a whole and held in memory all the times but I don't think that makes a difference anyway because at the end it does the same. And for the next question: Yes, the RAM requirements during that time are tremendous, especially because it reads a lot of files in a few dozen threads simultaniously at any given time. And don't forget it holds the computed data in RAM as well

Anyway, every single file is bigger then a single savegame of X4 and the whole process takes roughly the same time. Which kind of counters the "XML is big and slow" (not wrong, but not to the extend many believe it to be). And this is exactly why I don't understand what X4 is doing during that time.

Is it building a DOM first? Please tell me it is not building a DOM!

[Midnightknight]

Of course we could have thrown months of developer time into creating a format that could have achieved most of the above,

And be serious 3 seconds ... It do not cost months to have a save file system even one with all you mentioned. I don't even think it took one week to the man who wrote you parser from scratches ... And you can even go with XML if you wished and store a bit more data without repeating again and again the same tags.

Properties in XML do not required their own tags but are instead inserted in the parent tag between < and >. In your save format it's not, every damn property have it's own tag with the same words copied over again and again. Yes sur it has a cost, not only in memory but as a CPU too, you need to insert all those useless characters again and again, even on a XML perspective.

[radcapricorn]

... he was speaking about making a copy of the data in RAM and then save this copy. He didn't directly say (that I can find anywhere) if the actual write-it-to-disk part is during, or after the pause, but what is apparently happening during this "just make a copy in RAM" is that it's making the copy and incidentally inserting some xml markup, while converting some binary stuff to UTF-8 on the side probably. He said the additon of this formatting during the "copy in RAM" part isn't significant from a performance standpoint.

It's not "some" XML markup. It's actually quite a lot. Just to give you an idea, an average moderately-paced 20-hour save would contain:

- about 4 million tags, amounting to more than 34Mb (from a quick grep of opening tags only, i.e. it's underestimated).
- about 5 million quoted strings (i.e. "strings"), ~60Mb. Of those strings, only 35 thousand (0.07%) are unique, adding up to puny 650 kilobytes (1%). That's outrageous overhead, but such is the nature of XML, it's stupidly redundant.

Most of the numeric values, represented as text, are also grossly over-inflated (that is, take up way more space than their binary representation). I haven't the energy to write a script to estimate that particular overhead, at least not right now. But just from the two points above, about a third of the uncompressed save is basically, well, junk, which is completely discarded by the game and is needed purely for upholding XML structure. Analysis of just a couple randomly-chosen chunks in saves shows that at least those chunks could be "compressed" by at least a factor of 4 "simply" by getting rid of XML (that's before employing actual compression such as zlib).

But size is only one side of the issue. The more your on-disk representation deviates from in-memory representation, the more processing power (and therefore, time) you have to spend on conversions. And not just raw conversions (i.e. numbers to text and back), but replicating (when saving) or recreating (when loading) the overall structure. Just a simple example: an array. In memory, that's typically 16 or 24 bytes of upkeep plus the actual contiguous storage, of known bounds. In XML though, no such thing: it's just a stream of tags, which you would have to count one way or another to get at the size of storage that you need for data. Or the aforementioned "strings", most of which are keys for some tables, which implies lookups that are, at worst, O(n) (as opposed to linear indexing which is always O(1)).

Reading even an uncompressed 300Mb save from disk into memory is blazing fast (varies, but would be under half a second on most common hardware, assuming file is "cold"). Traversing it and building the universe from it is not, unless it's a (almost) carbon copy of game's layout.

I just glanced at my paused copy of X4 and it's using almost exactly 4gb of RAM right now. If we "just make a copy" that's doubling our RAM consumption instantly.

No, it's not. The number you're seeing is the amount of memory that allocator(s) which are used by the game (directly or indirectly) have reserved. It's not indicative of the size of actual game state (except to guess that it's no larger). That includes the overhead which allocators naturally create to avoid additional system calls and to create their own acceleration structures for efficient allocations. Plus there are no doubt various ancillary buffers that pertain to submitting and retrieving data to/from the GPU, and all the state associated with rendering, playing sound, etc. etc. In short, there's a lot of stuff that never gets into the save.
"Just make a copy" meant duplicating the structure and contents of game's state that needs to be saved. Depending on how that state is structured, it can be surprisingly easy, or frustratingly tedious and error-prone (and even slower than just directly translating the original into a different representation, i.e. XML). Judging by CBJ's response, it's the latter. Sadly, changing that at this point in time is just not going to fly.

Thanks, CBJ, for clarifying. @Tamina and strask, we understood each other just fine

[strask412]

It's not "some" XML markup. It's actually quite a lot. Just to give you an idea, an average moderately-paced 20-hour save would contain:

Thank you very much for the statistics and other enlightenment; I was previously aware of the rough magnitude of the size cost of the xml stuff (and included that in my rather vague word "some"), but actual (or, as I take it, bounded approximations based on actual measurements) numbers are always helpful.

The number you're seeing is the amount of memory that allocator(s) which are used by the game (directly or indirectly) have reserved. It's not indicative of the size of actual game state (except to guess that it's no larger).

That's of course absolutely correct and I am sorry for posting otherwise, I failed to think prior to writing what I wrote (despite taking some intentional actions, like checking process memory usage, in the course of it!). The ability of the human brain (especially mine!) to NOT think is sometimes amazing. Sorry.

[radcapricorn]

There's no need. The nature of today's Internet conversations sometimes compels one to pedantry

[CBJ]

And be serious 3 seconds ... It do not cost months to have a save file system even one with all you mentioned. I don't even think it took one week to the man who wrote you parser from scratches ... And you can even go with XML if you wished and store a bit more data without repeating again and again the same tags.

Again you are making false assumptions. The parser has nothing to do with this; in fact it's one of the things we use a third party tool (libxml2 to be specific) for, because why re-invent the XML-parsing wheel? I'm talking about going through each of the dozens, maybe hundreds, of objects in code that needs to save something and changing the way they do so, to use some system other than XML. Even taking into account a reasonable level of encapsulation of that process, which we do of course have, that is still a considerable task. Add to that the extensive testing that would be required, and the need to maintain backwards compatibility, then yes, you could well be talking about months of developer time. Even if it were "only" weeks, that time still has to be weighed against the other things that developer could be doing, especially when the likely benefit is extremely limited.

Properties in XML do not required their own tags but are instead inserted in the parent tag between < and >. In your save format it's not, every damn property have it's own tag with the same words copied over again and again. Yes sur it has a cost, not only in memory but as a CPU too, you need to insert all those useless characters again and again, even on a XML perspective.

If you just dump information into the data field using your own internal format then you are back to having to worry about corruption and versioning again as things change, and you lose all the benefits that XML provides. The data field has its uses, for example for long text strings, or for large, static, homogeneous data-sets such as animation offsets, but beyond that it just defeats the purpose of using XML in the first place.

With all due respect, I can't help thinking that the game development experience you say you have must be on much smaller-scale projects than this, because underestimating the amount of work involved in doing this kind of thing is a common mistake for people who have dipped into development without working on a real-world, large-scale project.

[Midnightknight]

Yes you are right, every AAA game use XML for their save game ... I must be the one with issues working on large scale projects ...

Oh by the way, just checked this C lib and

The internal document representation is as close as possible to the DOM interfaces.

lol who spoke about DOM? You know you can have all you asked above without following blindly DOM? This tool is used for structural purpose!! Not data storing.

If you just dump information into the data field using your own internal format then you are back to having to worry about corruption and versioning again as things change, and you lose all the benefits that XML provides. The data field has its uses, for example for long text strings, or for large, static, homogeneous data-sets such as animation offsets, but beyond that it just defeats the purpose of using XML in the first place.

Oh look

<owner factions="khaak xenon" relation="enemy"/>

Here somewhere in my save game you find a way to put 3 properties in one single tag!!! Hurry up solving it, it should be

Code: Select all

<owner factions="khaak">
<standing relation="enemy" />
</owner>
<owner factions="xenon">
<standing relation="enemy" />
</owner>

To avoid any risk of corruption and incompatibility!

[CBJ]

Given your responses, I'm not entirely sure you've understood the explanations I've given, but either way you clearly have no interest in anything other that asserting your position that you are right and the developers are wrong. This is the point at which I stop wasting my free time discussing the subject. Have fun.

[Tamina]

The only one who spoke about DOM was me, (indirectly asking CBJ) if X4 is creating an XML DOM structure, vaguely based on CBJ saying that the savegame represents the internal structure of the game. Again, because I work with XML a lot and it does not cause this big overhead (ofc there is an overhead, but writing a single file is stupidly fast. AFAIK the OS is caching the output stream and writing concurrently? Writing multiple tiny files is a waaaaaaay bigger problem).

[Boringnick]

What happened to my thread? No one cares about file saving formats.

I want my shiny ships.

[Alexraptor]

What happened to my thread? No one cares about file saving formats.

I want my shiny ships.

Easy there, this is a "Split" for this very reason. Your original thread is still here: viewtopic.php?f=146&t=417673

<Insecure http: link changed for secure https: one. Alan Phipps>

[photomankc]

The term is 'Argument from authority'. And if they were soo good like they say they are? Why aren't they working for Egosoft? Or in Devnet helping to fix the game. That's what bothers me personally about these particular arguments.

Another area that drives me nuts is when someone, outside and largely ignorant of the details, proceeds to tell those on the inside they should "just do X and Y to fix Z". Certainly in this world there are tons of incompetent people and poor practices in all areas of business and engineering. Thing is those conditions are not always without cause. There are, very likely, reasons that things developed as they did that involved deliberate choices from conditions only those on the inside know about. Something that I see as sub-optimal may have been the only possible choice among a series of less than optimal choices available. To presume that I can, from the outside, tell anyone how they should 'just do X and Y" seems unbearably presumptuous.

We all know that when people from outside our work environment critique it they are almost always off-base and propose 'solutions' that are unworkable for us and we bristle at that. Yet, somehow people seem born to do it to others even still.