egosoft.com

You were moderated probably because you made a personal attack on someone using offensive language, I don't know though since I didn't see it. My comment wasn't even harsh and please don't read meaning into it, just stating my observations as I see it. I didn't call anyone an idiot. I mean if you think the behavior I described is idiotic, that is your own personal interpretation. I consider it more to be along the lines of foolish, not necessarily idiotic. People do foolish things all the time, especially with the anonymity of the internet. I mean, on the internet, I can be anything I want to be when anonymous. I can claim myself to be an expert on anything and know everything. My issue is that you were basically insulting egosoft's credibility and the way they work, but you know nothing about why they made the decisions they did. You are not privy to their work environment and business decisions but felt the need to berate them on their choices. You then claimed to know better than the very people who made the game, especially CBJ. It is easy to be the one looking in from the outside and pointing fingers, especially when you aren't the one who had to put in the hard work and make the tough decisions.

And by the way, by no means am I saying there are no issues with the game. There are plenty of things to be worked on, so feel free to report the bugs and request the features you like. As for complaining about how they chose to do saving in the game, what is the point? It isn't going to change and I doubt it is something that could change at this point. This just isn't constructive.

Thats the point, I could have wrote a wall. of text back then about having to wait loading for every sector you get into, it would get me nowhere

and to the saving xml thing, I would say it not a big problem for most of the players, hitting F5 here and there and waiting for half a minute is not end off all things problem, and for a guy that dont know anything about programming xml style save is very friendly to edit and see whats what

the game is very stable (at least for me) and you can configure the autosave interval, if it happens in some menu and its not clear still not a big deal, surely not something that would get me to stop playing

if you want something that's a potential big problem right now would be the NPCs building production modules on shipyards, thats screws up enough things to be mad about

Midnightknight wrote: ↑
Fri, 5. Jul 19, 17:32
First it's not multi threaded, your whole game goes unresponsive, and is completely frozen, so you were closing a menu when auto save kicked in, you are thinking the game crashed and are already ready to ctrl+alt+del ... And it takes forever. It's not a priority, fine, you put priority were you want, but at least you can't deny there is a real issue with this and it should be solved. (And in many other aspects in the game, and that are not "bugs")
My issue is more that sometime we feel like you say "Oh it's not important" like everything was fine and do not take it seriously.

Yes, the game is "frozen" while you save. If it weren't then the game state would be inconsistent, containing data representing states at different times across the duration of the save process. Using threading to make the game "responsive" during saving would be a terrible idea, and indeed most games avoid this. And yes, the save process takes a little time. Most games only have to save your progress through their "level" system, and the states of a handful objects on your current level; the X series games have to save the state of everything in the game universe (apart from a few things that can be re-created from other data, such as the physics environment). Again, the format of the resulting file is a factor in the time taken to save the game, but it's by no means the biggest factor. Would we like to improve save times? Of course. Would creating a whole new file format and re-engineering everything to use it be an effective use of developer time in trying to achieve this? No, it wouldn't.

Buzz2005 wrote: ↑
Fri, 5. Jul 19, 18:01
and to the saving xml thing, I would say it not a big problem for most of the players

Indeed. To put your energy and complain about the saving system in X4 is just hilarious.

Buzz2005 wrote: ↑
Fri, 5. Jul 19, 18:01
Thats the point, I could have wrote a wall. of text back then about having to wait loading for every sector you get into, it would get me nowhere

and to the saving xml thing, I would say it not a big problem for most of the players, hitting F5 here and there and waiting for half a minute is not end off all things problem, and for a guy that dont know anything about programming xml style save is very friendly to edit and see whats what

the game is very stable (at least for me) and you can configure the autosave interval, if it happens in some menu and its not clear still not a big deal, surely not something that would get me to stop playing

if you want something that's a potential big problem right now would be the NPCs building production modules on shipyards, thats screws up enough things to be mad about

My game save for about 5-9 seconds. Which is faster than some of my citys in City:Skylines

And Ego using XML might be based on the fact that they used it since X2(?) for multiple things and so they get proficiend and used to it.

I cant see a damn reason why using XML is a "lazy" or "bad" thing....

IRONOX wrote: ↑
Fri, 5. Jul 19, 18:14
My game save for about 5-9 seconds. Which is faster than some of my citys in City:Skylines
And Ego using XML might be based on the fact that they used it since X2(?) for multiple things and so they get proficiend and used to it.

I cant see a damn reason why using XML is a "lazy" or "bad" thing....

OMG yes my latest city aka the biggest and longest of saves does take some time, and the fps is like 15

Save times are only slightly annoying to me. Not something that detracts from my enjoyment of X4, and definitely not something that I would want to takes weeks or months of development time away from more pressing gameplay issues.

BlackRain wrote: ↑
Fri, 5. Jul 19, 17:13
This seems to be a big problem on the forum in general and it isn't just with any particular person but from many that I have noticed. There tends to be many so called self introduced "experts" who think they know better than anyone else how X, Y, and Z should be...

BlackRain wrote: ...I have been around way longer than you so I saw all of this and have first hand experience. You clearly came way later and your view/perspective is at a much different time...

...If you can't see the illogical aspects of your argument in regards to this, then there is no point in me discussing with you anymore...

...I don't think you get that...

...Since you are not a modder and don't know anything about the code at all, I guess I can understand why you can't see the work put into it...

Hypocrite much? Seriously, please try to measure up to your own standards before judging others.

EDIT: Misunderstood. Yikes

CBJ wrote: ↑
Fri, 5. Jul 19, 18:04
Yes, the game is "frozen" while you save. If it weren't then the game state would be inconsistent, containing data representing states at different times across the duration of the save process...

Can't you reduce the pause duration by taking just the snapshot of current state (which is what requires pause for consistency), and offload its processing and subsequent I/O to background? Assuming there are provisions in the engine for doing that, of course. E.g. if your state is structured in such a way that copying it wouldn't be any faster that straight-up conversion to XML, then it'd be moot.

Regardless, if anything else fails, please do try to find it in your heart to fix this. I.e. pause simulation before taking control away, and unpause it only after the game is again interactive. Please!

radcapricorn wrote: ↑
Fri, 5. Jul 19, 18:52

BlackRain wrote: ↑
Fri, 5. Jul 19, 17:13
This seems to be a big problem on the forum in general and it isn't just with any particular person but from many that I have noticed. There tends to be many so called self introduced "experts" who think they know better than anyone else how X, Y, and Z should be...

BlackRain wrote: ...I have been around way longer than you so I saw all of this and have first hand experience. You clearly came way later and your view/perspective is at a much different time...

...If you can't see the illogical aspects of your argument in regards to this, then there is no point in me discussing with you anymore...

...I don't think you get that...

...Since you are not a modder and don't know anything about the code at all, I guess I can understand why you can't see the work put into it...
Hypocrite much? Seriously, please try to measure up to your own standards before judging others.

None of those posts are hypocritical. First, the initial comment I made had to do with perspective. I don't think you got at all what I was saying. What I was trying to say was that because I had been around longer and saw every release of every X game and went through the experience of having immensely buggy games at launch, missing features, the evolution of each of the X games, etc. this gave me a better perception of the X series overall. Which it does, and yes, experience/time does matter in this case. You came around at a time when X3 had already evolved and turned into a much different game than it was at release and then wanted to point out the flaws in X4 which hasn't been out for even a year yet. Not to mention, you couldn't even answer the question I asked and just ranted about bugs. Your comments, I have already seen thousands of times for pretty much every single X game although some of the concepts/features may have changed. You completely didn't understand this I guess.

The second thing, yeah your argument was illogical in my opinion and I got out of the discussion. This is called a personal choice and opinion, did I claim to be some kind of expert? Nope, but you certainly did, should I quote that? Not to mention, you are still being illogical now. I mean, you can't even argue a point without misconstruing my words and falsely accusing me of something that isn't even true. Nothing hypocritical in any of my comments, but it is a bit of irony that you are doing exactly what I said in my post here (cherry picking pieces of evidence which don't even fit the context).

Lastly, you have not a single modding credit to your name. If you do, please point it out for me. You claimed to be some expert on coding, but what have you contributed to the community to prove that? Nothing. Well guess what, I have contributed to the community quite a lot in fact. I have modded the X series since X3. Have you? Maybe you have and I just don't know it, please point me to the mods you have worked on. You just "Claimed" you were a programmer and looked at their code. Sorry if I don't take your word for it. I can prove I have done it though and yeah i know there are issues, but it is still impressive work in my opinion. So again, no hypocrisy there. I find this quite humorous though.

Well, whatever. This is another conversation I don't really want to post more on. I really have to stop wasting my time. So, go ahead, I'm done.

radcapricorn wrote: ↑
Fri, 5. Jul 19, 19:02
Can't you reduce the pause duration by taking just the snapshot of current state (which is what requires pause for consistency), and offload its processing and subsequent I/O to background? Assuming there are provisions in the engine for doing that, of course. E.g. if your state is structured in such a way that copying it wouldn't be any faster that straight-up conversion to XML, then it'd be moot.

State information is not stored in one place. The savegame is effectively the snapshot, with the structure of the XML corresponding fairly closely to the encapsulated code and datasets that generate it. Serialising it in a different format isn't going to change much.

On a moderation note: please stop the personal comments and the discussion of moderation.

CBJ wrote: ↑
Fri, 5. Jul 19, 19:12

radcapricorn wrote: ↑
Fri, 5. Jul 19, 19:02
Can't you reduce the pause duration by taking just the snapshot of current state (which is what requires pause for consistency), and offload its processing and subsequent I/O to background? Assuming there are provisions in the engine for doing that, of course. E.g. if your state is structured in such a way that copying it wouldn't be any faster that straight-up conversion to XML, then it'd be moot.
State information is not stored in one place. The savegame is effectively the snapshot, with the structure of the XML corresponding fairly closely to the encapsulated code and datasets that generate it. Serialising it in a different format isn't going to change much.

If I understood him correctly, he was speaking about making a copy of the data in RAM and then save this copy, while the game keeps running.

I think it is weird that it takes so long to save. I deal with way bigger XML files (gigabytes) on a daily basis and it takes seconds to load them, parse them, process them and save them again. Then again, I have no clue what is happening in the background in the game but just reading the current state, doesn't sound that complicated at first.

Tamina wrote: ↑
Fri, 5. Jul 19, 19:23

CBJ wrote: ↑
Fri, 5. Jul 19, 19:12

radcapricorn wrote: ↑
Fri, 5. Jul 19, 19:02
Can't you reduce the pause duration by taking just the snapshot of current state (which is what requires pause for consistency), and offload its processing and subsequent I/O to background? Assuming there are provisions in the engine for doing that, of course. E.g. if your state is structured in such a way that copying it wouldn't be any faster that straight-up conversion to XML, then it'd be moot.
State information is not stored in one place. The savegame is effectively the snapshot, with the structure of the XML corresponding fairly closely to the encapsulated code and datasets that generate it. Serialising it in a different format isn't going to change much.
If I understood him correctly, he was speaking about making a copy of the data in RAM and then save this copy, while the game keeps running.

Heh.
And if I understood CBJ correctly when he said this:

CBJ wrote: ↑
Fri, 5. Jul 19, 19:12
State information is not stored in one place. The savegame is effectively the snapshot, with the structure of the XML corresponding fairly closely to the encapsulated code and datasets that generate it. Serialising it in a different format isn't going to change much.

... he was speaking about making a copy of the data in RAM and then save this copy. He didn't directly say (that I can find anywhere) if the actual write-it-to-disk part is during, or after the pause, but what is apparently happening during this "just make a copy in RAM" is that it's making the copy and incidentally inserting some xml markup, while converting some binary stuff to UTF-8 on the side probably. He said the additon of this formatting during the "copy in RAM" part isn't significant from a performance standpoint.

As to the overall level of surprise expressed by people about duration of the save process, think of it this way: I just glanced at my paused copy of X4 and it's using almost exactly 4gb of RAM right now. If we "just make a copy" that's doubling our RAM consumption instantly. While of course the CPU is busy traversing the game's data in RAM and making the copies, I'm sure that there are also potential memory bandwidth concerns and possibly this sudden allocation of 4gb could trigger the OS to swap some other application out to disk or whatever, resulting in additional delays. I'm not trying to express "this is what happens", because I don't know all the specifics, but I'm trying to express "it is actually reasonable to expect that a lot of work is happening, and the duration of the save process doesn't particularly surprise me".

Edit: The comparison you make to your daily work:

Tamina wrote: ↑
Fri, 5. Jul 19, 19:23
I deal with way bigger XML files (gigabytes) on a daily basis and it takes seconds to load them, parse them, process them and save them again. Then again, I have no clue what is happening in the background in the game but just reading the current state, doesn't sound that complicated at first.

may not be equivalent for a number of reasons; the one I can think of off the top of my head would be the memory requirement above. I don't know your code or anything, but from the words you used (load/parse/process/save) it seems unlikely to me that the entire file is stored in memory at once, it's probably got the input and output files both open for i/o and only the current chunk(s) of the input need to be in memory, once written to the output file they can be forgotten.

strask412 wrote: ↑
Fri, 5. Jul 19, 20:47
Edit: The comparison you make to your daily work:
Tamina wrote: ↑
Fri, 5. Jul 19, 19:23
I deal with way bigger XML files (gigabytes) on a daily basis and it takes seconds to load them, parse them, process them and save them again. Then again, I have no clue what is happening in the background in the game but just reading the current state, doesn't sound that complicated at first.
may not be equivalent for a number of reasons; the one I can think of off the top of my head would be the memory requirement above. I don't know your code or anything, but from the words you used (load/parse/process/save) it seems unlikely to me that the entire file is stored in memory at once, it's probably got the input and output files both open for i/o and only the current chunk(s) of the input need to be in memory, once written to the output file they can be forgotten.

It's actually loaded as a whole and held in memory all the times but I don't think that makes a difference anyway because at the end it does the same. And for the next question: Yes, the RAM requirements during that time are tremendous, especially because it reads a lot of files in a few dozen threads simultaniously at any given time. And don't forget it holds the computed data in RAM as well

Anyway, every single file is bigger then a single savegame of X4 and the whole process takes roughly the same time. Which kind of counters the "XML is big and slow" (not wrong, but not to the extend many believe it to be). And this is exactly why I don't understand what X4 is doing during that time.

Is it building a DOM first? Please tell me it is not building a DOM!

CBJ wrote: ↑
Fri, 5. Jul 19, 17:05
Of course we could have thrown months of developer time into creating a format that could have achieved most of the above,

And be serious 3 seconds ... It do not cost months to have a save file system even one with all you mentioned. I don't even think it took one week to the man who wrote you parser from scratches ... And you can even go with XML if you wished and store a bit more data without repeating again and again the same tags.

Properties in XML do not required their own tags but are instead inserted in the parent tag between < and >. In your save format it's not, every damn property have it's own tag with the same words copied over again and again. Yes sur it has a cost, not only in memory but as a CPU too, you need to insert all those useless characters again and again, even on a XML perspective.

strask412 wrote: ↑
Fri, 5. Jul 19, 20:47
... he was speaking about making a copy of the data in RAM and then save this copy. He didn't directly say (that I can find anywhere) if the actual write-it-to-disk part is during, or after the pause, but what is apparently happening during this "just make a copy in RAM" is that it's making the copy and incidentally inserting some xml markup, while converting some binary stuff to UTF-8 on the side probably. He said the additon of this formatting during the "copy in RAM" part isn't significant from a performance standpoint.

It's not "some" XML markup. It's actually quite a lot. Just to give you an idea, an average moderately-paced 20-hour save would contain:

- about 4 million tags, amounting to more than 34Mb (from a quick grep of opening tags only, i.e. it's underestimated).
- about 5 million quoted strings (i.e. "strings"), ~60Mb. Of those strings, only 35 thousand (0.07%) are unique, adding up to puny 650 kilobytes (1%). That's outrageous overhead, but such is the nature of XML, it's stupidly redundant.

Most of the numeric values, represented as text, are also grossly over-inflated (that is, take up way more space than their binary representation). I haven't the energy to write a script to estimate that particular overhead, at least not right now. But just from the two points above, about a third of the uncompressed save is basically, well, junk, which is completely discarded by the game and is needed purely for upholding XML structure. Analysis of just a couple randomly-chosen chunks in saves shows that at least those chunks could be "compressed" by at least a factor of 4 "simply" by getting rid of XML (that's before employing actual compression such as zlib).

But size is only one side of the issue. The more your on-disk representation deviates from in-memory representation, the more processing power (and therefore, time) you have to spend on conversions. And not just raw conversions (i.e. numbers to text and back), but replicating (when saving) or recreating (when loading) the overall structure. Just a simple example: an array. In memory, that's typically 16 or 24 bytes of upkeep plus the actual contiguous storage, of known bounds. In XML though, no such thing: it's just a stream of tags, which you would have to count one way or another to get at the size of storage that you need for data. Or the aforementioned "strings", most of which are keys for some tables, which implies lookups that are, at worst, O(n) (as opposed to linear indexing which is always O(1)).

Reading even an uncompressed 300Mb save from disk into memory is blazing fast (varies, but would be under half a second on most common hardware, assuming file is "cold"). Traversing it and building the universe from it is not, unless it's a (almost) carbon copy of game's layout.

I just glanced at my paused copy of X4 and it's using almost exactly 4gb of RAM right now. If we "just make a copy" that's doubling our RAM consumption instantly.

No, it's not. The number you're seeing is the amount of memory that allocator(s) which are used by the game (directly or indirectly) have reserved. It's not indicative of the size of actual game state (except to guess that it's no larger). That includes the overhead which allocators naturally create to avoid additional system calls and to create their own acceleration structures for efficient allocations. Plus there are no doubt various ancillary buffers that pertain to submitting and retrieving data to/from the GPU, and all the state associated with rendering, playing sound, etc. etc. In short, there's a lot of stuff that never gets into the save.
"Just make a copy" meant duplicating the structure and contents of game's state that needs to be saved. Depending on how that state is structured, it can be surprisingly easy, or frustratingly tedious and error-prone (and even slower than just directly translating the original into a different representation, i.e. XML). Judging by CBJ's response, it's the latter. Sadly, changing that at this point in time is just not going to fly.

Thanks, CBJ, for clarifying. @Tamina and strask, we understood each other just fine

radcapricorn wrote: ↑
Fri, 5. Jul 19, 23:01
It's not "some" XML markup. It's actually quite a lot. Just to give you an idea, an average moderately-paced 20-hour save would contain:

Thank you very much for the statistics and other enlightenment; I was previously aware of the rough magnitude of the size cost of the xml stuff (and included that in my rather vague word "some"), but actual (or, as I take it, bounded approximations based on actual measurements) numbers are always helpful.

radcapricorn wrote: ↑
Fri, 5. Jul 19, 23:01
The number you're seeing is the amount of memory that allocator(s) which are used by the game (directly or indirectly) have reserved. It's not indicative of the size of actual game state (except to guess that it's no larger).

That's of course absolutely correct and I am sorry for posting otherwise, I failed to think prior to writing what I wrote (despite taking some intentional actions, like checking process memory usage, in the course of it!). The ability of the human brain (especially mine!) to NOT think is sometimes amazing. Sorry.

There's no need. The nature of today's Internet conversations sometimes compels one to pedantry

Midnightknight wrote: ↑
Fri, 5. Jul 19, 23:00
And be serious 3 seconds ... It do not cost months to have a save file system even one with all you mentioned. I don't even think it took one week to the man who wrote you parser from scratches ... And you can even go with XML if you wished and store a bit more data without repeating again and again the same tags.

Again you are making false assumptions. The parser has nothing to do with this; in fact it's one of the things we use a third party tool (libxml2 to be specific) for, because why re-invent the XML-parsing wheel? I'm talking about going through each of the dozens, maybe hundreds, of objects in code that needs to save something and changing the way they do so, to use some system other than XML. Even taking into account a reasonable level of encapsulation of that process, which we do of course have, that is still a considerable task. Add to that the extensive testing that would be required, and the need to maintain backwards compatibility, then yes, you could well be talking about months of developer time. Even if it were "only" weeks, that time still has to be weighed against the other things that developer could be doing, especially when the likely benefit is extremely limited.

Midnightknight wrote: ↑
Fri, 5. Jul 19, 23:00
Properties in XML do not required their own tags but are instead inserted in the parent tag between < and >. In your save format it's not, every damn property have it's own tag with the same words copied over again and again. Yes sur it has a cost, not only in memory but as a CPU too, you need to insert all those useless characters again and again, even on a XML perspective.

If you just dump information into the data field using your own internal format then you are back to having to worry about corruption and versioning again as things change, and you lose all the benefits that XML provides. The data field has its uses, for example for long text strings, or for large, static, homogeneous data-sets such as animation offsets, but beyond that it just defeats the purpose of using XML in the first place.

With all due respect, I can't help thinking that the game development experience you say you have must be on much smaller-scale projects than this, because underestimating the amount of work involved in doing this kind of thing is a common mistake for people who have dipped into development without working on a real-world, large-scale project.

Yes you are right, every AAA game use XML for their save game ... I must be the one with issues working on large scale projects ...

Oh by the way, just checked this C lib and

The internal document representation is as close as possible to the DOM interfaces.

lol who spoke about DOM? You know you can have all you asked above without following blindly DOM? This tool is used for structural purpose!! Not data storing.

CBJ wrote: ↑
Sat, 6. Jul 19, 16:19
If you just dump information into the data field using your own internal format then you are back to having to worry about corruption and versioning again as things change, and you lose all the benefits that XML provides. The data field has its uses, for example for long text strings, or for large, static, homogeneous data-sets such as animation offsets, but beyond that it just defeats the purpose of using XML in the first place.

Oh look

<owner factions="khaak xenon" relation="enemy"/>

Here somewhere in my save game you find a way to put 3 properties in one single tag!!! Hurry up solving it, it should be

Code: Select all

<owner factions="khaak">
<standing relation="enemy" />
</owner>
<owner factions="xenon">
<standing relation="enemy" />
</owner>

To avoid any risk of corruption and incompatibility!

egosoft.com

Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread

Re: Split from "Why have you guys traded metal for plastic?" thread