Happy 2022! Now, about those save times...
Moderator: Moderators for English X Forum
-
- Moderator (English)
- Posts: 30422
- Joined: Fri, 16. Apr 04, 19:21
Re: Happy 2022! Now, about those save times...
Re the American comment: let's not disappear down that rabbit hole. I agree that it was inappropriate and shouldn't have been said, but let's leave it at that. Back on topic please.
A dog has a master; a cat has domestic staff.
Re: Happy 2022! Now, about those save times...
Ah, heh. That was referring to their cultural trait (self promotion being socially acceptable), e.g. https://theprofessorisin.com/2015/02/06 ... dont-brag/ My American colleagues think it's strange we don't emphasize our strengths during business talks, while e.g. my French colleagues think the opposite is not done. It was just there to indicate it's outside my comfort zone to write like this - nothing more.Alan Phipps wrote: ↑Sat, 15. Jan 22, 15:50Re the American comment: let's not disappear down that rabbit hole. I agree that it was inappropriate and shouldn't have been said, but let's leave it at that. Back on topic please.
Re: Happy 2022! Now, about those save times...
I see, I understand, those people are just the most visible (which would be very much in the nature of "self promotion"). However, I wouldn't say they are the majority. Most Americans aren't like that in my opinion, especially depending on what part of the country. There are many wholesome, humble, polite and respectful Americans too, it is just that they don't get any visibility. In the world we live in, those kind of people aren't interesting or/and don't make good stories.Ehli wrote: ↑Sat, 15. Jan 22, 16:38Ah, heh. That was referring to their cultural trait (self promotion being socially acceptable), e.g. https://theprofessorisin.com/2015/02/06 ... dont-brag/ My American colleagues think it's strange we don't emphasize our strengths during business talks, while e.g. my French colleagues think the opposite is not done. It was just there to indicate it's outside my comfort zone to write like this - nothing more.Alan Phipps wrote: ↑Sat, 15. Jan 22, 15:50Re the American comment: let's not disappear down that rabbit hole. I agree that it was inappropriate and shouldn't have been said, but let's leave it at that. Back on topic please.
Re: Happy 2022! Now, about those save times...
Edit: This is a code-oriented post, I promise... just gotta skim down past the fluff
Here's the Code link from below for quick reference (Real-time DAG snapshotting, No Locks)
--
::yawn::
.
.
.
::stretch::
.
.
.
Oh man... as an American, I must admit, I quite enjoyed my 3 day weekend. Hope it was as nice for "y'all" over there "out yonder" on the other side of the pond
Though I have to say, I didn't expect to come back to a thread derailed by the moderators.
"While there's mice away, the cats will play," is that it?
I was referring to EgoSoft as the "sages". But you do you. Pick your title. I don't mind in the least
Anyway, your work advice is totally sensible for anyone below SSE.
I only work on referral, and have for years. I'll give you the first week for free to test my mettle... but I don't think anyone's asked me a real coding question in an interview in 20 years.
Personal opinion -- any SSE+ who's not working purely on a referral model is doing something wrong.
Just my 2 cents.
--
Truly -- if you're trying to be more polite, I'll try and respond in kind -- be more personable and what not... we'll see how it goes.
Look, I've been writing C++ for 22 years.
Before that it was mostly BASIC, COBOL, and Fortran.
I've got a background in aerospace -- mostly air traffic control and satellite/near-earth-object tracking -- so I'm pretty used to dealing with large quantities of real-time data.
I got into game dev because I didn't like working under clearance, I got tired of contracting, and I've been getting hounded for the last decade or so... and it sounded like fun! (and it is... most of the time)
And the world fell apart... so, working from home instead of a lab ain't so bad.
Anyway, I work in a lot of languages, so we can switch out if your hung up on my semantics -- I'm down for pretty much anything but Erlang, Go, or Rust
Seems like EgoSoft's a C++ shop, so I'm trying to keep the core there.
Tooling's a bit quicker in Python/JavaScript/Perl/etc. so I may switch it up a bit as we're scaffolding (if you decide to stick around, that is).
Unfortunately I feel your perception so far is entirely based on examples designed for discussion by a general audience -- and maybe a healthy dose of just thinking you know better.
If you try and focus on the concepts instead of the execution I think we'd get a lot further.
For future reference, I write most of my examples in C++11 (not 98), both because I regularly find myself working on 10 year old projects -- maintenance is 70% of the product lifecycle after all -- and because it'll compile on just about any system onlookers might have available (I'm looking at you, macOS).
The purpose is example, not production. But, I'm happy to step things up a bit if you wanna join the game
--
On to the code.
A CRTP requires a derived class inherit in recurring template form -- thus the name -- e.g.
...which it does not. It's simple virtual inheritance.
There are no subsequently derived classes, just wrapped types.
Ergo, no CRTP.
I'll gladly take a few minutes and type out a direct translation.
That's not better. That's much worse. It's a mess filled with gets and spurious release calls.
And it performs exactly the same, since std::unique_ptr has no overhead once optimized.
Though it performs significantly worse unoptimized (so smart pointers aren't awesome for debugging speed).
Anyway, I didn't use them here because I'm specifically doing with pointers what smart pointers were designed to stop developers from doing.
Not because it's wrong, but because junior devs always spring leaks.
And for the record, I used virtual inheritance because, compiled, the vtable overhead is one assembly call.
For someone who doesn't even want to be writing code yet, you sure do seem like you want to jump to the end.
Optimization is usually the last step in code design... but we'll get to that in a minute.
I think in code, topology, and on-the-ground logistics. Not design patterns, not "problem spaces", not DAGs.
I'm accustomed to zero-day timings and people who -do- instead of -ask- or -plan-.
I don't like diagrams, flow charts, or spread sheets. I don't like red tape. Though, you're right, I do like white boards
What I like most is to do the job -- to do it right -- but more importantly to do it right now.
But sure, I'm a sport. Let's do so.
I can work with this.
How about this completely different way? (compilable gist with comments, C++11, no dependencies)
You know, following such narrow constraints... and applying all the suggestions from this thread... I had to think about that one for a few minutes!
Thanks. It was a neat little puzzle to end a long weekend
Quick summary... and tell me if I missed any of the suggestions from this thread:
Anyway, give it a spin when ya have a few.
Maybe it'll open up another idea or two.
Again, I think the new/delete method's more efficient, because most of the time you're not in a save state -- but it would bog the save state time significantly more than this method.
This way costs more RAM permanently, but write performance actually improves during save state, so if the RAM's there to spare it may be the better option.
Look, rocky starts aside, if you know the material as well as you say you do, what's it gonna hurt to write a couple tidbits and try out a few things?
Anyhoo, that's about all the time I have this morning.
If I forgot to comment on anyone else's specific notes, I'll get back to ya.
Oh, and quit raggin' on Americans. You're breaking the mods
Edited: A few times, to be more polite and on-topic. I don't want things like throwing around titles and experience to detract from the main point here (actual code)
Here's the Code link from below for quick reference (Real-time DAG snapshotting, No Locks)
--
::yawn::
.
.
.
::stretch::
.
.
.
Oh man... as an American, I must admit, I quite enjoyed my 3 day weekend. Hope it was as nice for "y'all" over there "out yonder" on the other side of the pond
Though I have to say, I didn't expect to come back to a thread derailed by the moderators.
"While there's mice away, the cats will play," is that it?
S'ok. Random dudes on the internet are what makes it the internet.vmo wrote: ↑Fri, 14. Jan 22, 11:23I don't expect to change any minds here so I won't press further. Some things you learn from working on (and designing) big enough systems. I know that sounds like I'm trying to be a "sage", and there's no reason to believe some random dude on the internet, which I am at this point.
I was referring to EgoSoft as the "sages". But you do you. Pick your title. I don't mind in the least
Anyway, your work advice is totally sensible for anyone below SSE.
I only work on referral, and have for years. I'll give you the first week for free to test my mettle... but I don't think anyone's asked me a real coding question in an interview in 20 years.
Personal opinion -- any SSE+ who's not working purely on a referral model is doing something wrong.
Just my 2 cents.
--
If you'd started that way to begin with I think we'd have gotten off on a much better footing
That's great. So take a few minutes out of your evening to throw a couple code snippets together and illustrate what you believe to be the issues.Ehli wrote: ↑Fri, 14. Jan 22, 23:57Without trying to sound too American (obviously I'm not) I'm rather experienced with this problem space: I'm a principal architect working in the R&D of one the biggest European tech companies. I'm the lead of a high performance STM I designed and is used in production by a large amount of enterprises.
...
I programmed in C/C++ for the last 25 years, of which the last 15 as a full time job
That's a pretty condescending list... and I'm going to ignore it, because this is your most polite post to date.Ehli wrote: ↑Fri, 14. Jan 22, 23:57
- I see you put effort in your code, the enthusiasm, time spent to make it better.
- Use modern C++ (17 or 20), not C++98.
- Using 98 like that would give you a hard time applying for a senior software engineer position in my country.
- This is why all of my answers were about architecture. How about thinking about that first?
- drop me a PM if you want a job in this field.
Truly -- if you're trying to be more polite, I'll try and respond in kind -- be more personable and what not... we'll see how it goes.
I'll bring the beer
Look, I've been writing C++ for 22 years.
Before that it was mostly BASIC, COBOL, and Fortran.
I've got a background in aerospace -- mostly air traffic control and satellite/near-earth-object tracking -- so I'm pretty used to dealing with large quantities of real-time data.
I got into game dev because I didn't like working under clearance, I got tired of contracting, and I've been getting hounded for the last decade or so... and it sounded like fun! (and it is... most of the time)
And the world fell apart... so, working from home instead of a lab ain't so bad.
Anyway, I work in a lot of languages, so we can switch out if your hung up on my semantics -- I'm down for pretty much anything but Erlang, Go, or Rust
Seems like EgoSoft's a C++ shop, so I'm trying to keep the core there.
Tooling's a bit quicker in Python/JavaScript/Perl/etc. so I may switch it up a bit as we're scaffolding (if you decide to stick around, that is).
Unfortunately I feel your perception so far is entirely based on examples designed for discussion by a general audience -- and maybe a healthy dose of just thinking you know better.
If you try and focus on the concepts instead of the execution I think we'd get a lot further.
For future reference, I write most of my examples in C++11 (not 98), both because I regularly find myself working on 10 year old projects -- maintenance is 70% of the product lifecycle after all -- and because it'll compile on just about any system onlookers might have available (I'm looking at you, macOS).
The purpose is example, not production. But, I'm happy to step things up a bit if you wanna join the game
--
On to the code.
Please stop saying this.Ehli wrote: ↑Fri, 14. Jan 22, 23:57I would've suggested corrections, e.g. the CRTP ( https://en.wikipedia.org/wiki/Curiously ... te_pattern ) is ruined...
A CRTP requires a derived class inherit in recurring template form -- thus the name -- e.g.
Code: Select all
class Derived : Base<Derived>
There are no subsequently derived classes, just wrapped types.
Ergo, no CRTP.
Hah. Ok, so you want to see why I didn't use std::unique_ptr?
I'll gladly take a few minutes and type out a direct translation.
Code: Select all
template <typename T> class Saveable : public virtual Updateable {
protected:
mutable std::unique_ptr<T> a, b;
public:
Saveable() { a = std::make_unique<T>(); b = std::unique_ptr<T>(a.get()); }
Saveable(const T& t) : Saveable() { *a = t; }
Saveable(const Saveable<T>& o) : Saveable() {
a = std::make_unique<T>(*(o.a));
if (o.a.get() != o.b.get()) {
b.release();
b = std::make_unique<T>(*(o.a));
}
}
~Saveable() { if (a.get() == b.get()) b.release(); }
Saveable<T>& operator=(const T& t) { setValue(t); return *this; } // Direct assignment
operator T() { return getValue(); } // Direct use as type
T operator +() { return getSaveValue(); } // Retrieve save value with a '+' prefix
void setValue(const T& t) { if (isSaving && a.get() == b.get()) { b.release(); b = std::make_unique<T>(); } *b = t; }
T getValue() const { return *b; }
T getSaveValue() const { return *a; }
void update() const override { if (a.get() != b.get() && !isSaving) { a = std::unique_ptr<T>(b.get()); }}
};
And it performs exactly the same, since std::unique_ptr has no overhead once optimized.
Though it performs significantly worse unoptimized (so smart pointers aren't awesome for debugging speed).
Anyway, I didn't use them here because I'm specifically doing with pointers what smart pointers were designed to stop developers from doing.
Not because it's wrong, but because junior devs always spring leaks.
And for the record, I used virtual inheritance because, compiled, the vtable overhead is one assembly call.
For someone who doesn't even want to be writing code yet, you sure do seem like you want to jump to the end.
Optimization is usually the last step in code design... but we'll get to that in a minute.
Back in the day, before the dot-com bubble, I was a hacker for hire.
I think in code, topology, and on-the-ground logistics. Not design patterns, not "problem spaces", not DAGs.
I'm accustomed to zero-day timings and people who -do- instead of -ask- or -plan-.
I don't like diagrams, flow charts, or spread sheets. I don't like red tape. Though, you're right, I do like white boards
What I like most is to do the job -- to do it right -- but more importantly to do it right now.
Well... I already said I don't like thinking in DAGs...Ehli wrote: ↑Fri, 14. Jan 22, 23:57Let's take the image of my favourite website: https://en.wikipedia.org/wiki/Directed_acyclic_graph
But sure, I'm a sport. Let's do so.
Oh good, we're getting into specific constraints now.Ehli wrote: ↑Fri, 14. Jan 22, 23:57Let's use that image, with the following constraints/statements:
- All objects in the game world are part of the same acyclic graph, which IMO is fair to assume given Ego's replies in the original thread
- There's only 1 thread (which is logical) operating on this graph (Ego stated somewhere the main computation is single-threaded)
- This thread receives the game events which will modify sub parts of this graph, e.g. "C" will be deleted, "C" owns "D" and thus will delete along with it, both replaced by "F" and "G"
I can work with this.
Sure, I can do that.
There's only one way?
How about this completely different way? (compilable gist with comments, C++11, no dependencies)
You know, following such narrow constraints... and applying all the suggestions from this thread... I had to think about that one for a few minutes!
Thanks. It was a neat little puzzle to end a long weekend
Quick summary... and tell me if I missed any of the suggestions from this thread:
- It's a straight copy of the DAG from your link -- the first swap is even a C-owned-D to F and G pair... at your specific request
- Single producer/consumer model -- only one producer thread working on the graph -- though it works for any number of consumers you like
- Applying snapshots both incrementally at a low level (sub-vertex values) and last-minute at point-of-save on the graph itself
- Serializes the graph to XML (intentionally slowed down to two edges per second) -- prints the XML to stderr, so run it with 1>/dev/null if for some reason you just wanna look at XML
- Performs 3 serialization passes on snapshots at different points of play and then performs a single-thread stress test
- Not a single new or delete call -- It's not realistic to do it all on the stack in production, but works fine for a simple DAG like this
- No locks (didn't even include mutex)
- Just went with simple request-control sync via overlapping atomics
- No event queues to bottleneck
- Was originally going to, but passed on memory barriers -- just not worth the maintenance complexity for something this simple
- Didn't even use a condition variable -- though it may speed the transition-to-save ever so slightly if I had used one, it's not even worth the extra include
- When you say performance, you seem to mean speed (without any regard for memory consumption). I tend to work on a lot of low memory systems, so that's usually factored into performance for me, but I disregarded it here -- nearly double the memory usage for 800% increase in straight write performance but only during a save by dropping the pointer shifts and heap allocations -- though the new/delete model outperforms it in standard operation on a predetermined graph in both speed and memory efficiency... because they're never called except when a save occurs.
- I left in the vtable overhead because it was a lot simpler than multiple registrars, and it only runs once per snapshottable item per save -- so nearly nonexistent.
- Granted it's only rotating through 6 vertices and 8 edges, but the performance looks alright. One "iteration" translates to swapping a random vertex, zeroing all the edge counts in the graph, recounting and storing the edge count for each vertex, and snapshotting the entire graph for save. I'm getting about 600k iterations/second on a single thread (on my 10 year old test rig)
Anyway, give it a spin when ya have a few.
Maybe it'll open up another idea or two.
As you said, there's no free lunch. This strategy offloads the majority of the snapshot management from the point of save across all the value writes.
Again, I think the new/delete method's more efficient, because most of the time you're not in a save state -- but it would bog the save state time significantly more than this method.
This way costs more RAM permanently, but write performance actually improves during save state, so if the RAM's there to spare it may be the better option.
I'd hate to see you give up just as you're starting to join in
Look, rocky starts aside, if you know the material as well as you say you do, what's it gonna hurt to write a couple tidbits and try out a few things?
Anyhoo, that's about all the time I have this morning.
If I forgot to comment on anyone else's specific notes, I'll get back to ya.
Oh, and quit raggin' on Americans. You're breaking the mods
Edited: A few times, to be more polite and on-topic. I don't want things like throwing around titles and experience to detract from the main point here (actual code)
Last edited by xixas on Thu, 27. Jan 22, 19:50, edited 4 times in total.
Re: Happy 2022! Now, about those save times...
Hey Cdaragorn, I missed your post this morning. My apologies.
But each message is serialized independently, and as I alluded to in a prior message, there's no reason we can't leave an empty block at the beginning of the file, drop a size header on every message (tallying them as we write), and then just seek back to file head and write the total data size written block -- maybe leave a few blocks and store a small index -- for example where the closest message starts are located with the file size divided in 8 pieces. Then we could fire up multiple reader threads to decrease load time
But that's why I'm trying to automate scaffolding the "message" (serializer) classes.
You know, DRY principle and all that.
So I don't think that's an issue... unless I'm misunderstanding what you're saying, in which case, please elucidate.
I've had a couple crossover releases this past week and haven't had nearly as much time as I usually do, so some extra input would be fantastic -- and that goes triple for actual code!
Take a look at the X4 Savegame XML Analyzer I wrote last week.
It'll break down an unarchived savegame file into a flat tag/ownership/property/type list that you should be able to use as a base to write any automated scaffolding tools ya want.
If you need any more features, just let me know and I'll see what I can add in as time permits.
For files, they do have a standard sequential read, that's true, because they're designed for RPC -- basically just reading a message stream in off the wire.Cdaragorn wrote: ↑Sat, 15. Jan 22, 05:33I've looked at Protobuf before but didn't use it because it doesn't allow random access to the data. Basically it suffers the same weakness that any text format does because it's order dependent. If you can't truly split the data up you can't use good Data Oriented Design patterns to speed the data handling up.
But each message is serialized independently, and as I alluded to in a prior message, there's no reason we can't leave an empty block at the beginning of the file, drop a size header on every message (tallying them as we write), and then just seek back to file head and write the total data size written block -- maybe leave a few blocks and store a small index -- for example where the closest message starts are located with the file size divided in 8 pieces. Then we could fire up multiple reader threads to decrease load time
Yeah... boilerplate never goes away.
But that's why I'm trying to automate scaffolding the "message" (serializer) classes.
You know, DRY principle and all that.
We know the entire universe, as written to the save file, is already fully stored in memory (in a much more space-efficient manner).Cdaragorn wrote: ↑Sat, 15. Jan 22, 05:33Storing everything in memory first is a nice theory but in practice when working with this much data that actually creates problems that prevent it from getting the speed you're expecting from it.
The best system I've worked with involved streaming the data out of that memory at the same time as you were putting it in so it never takes up too much memory. That gets back to the need for true random access.
So I don't think that's an issue... unless I'm misunderstanding what you're saying, in which case, please elucidate.
That'd be awesome.
I've had a couple crossover releases this past week and haven't had nearly as much time as I usually do, so some extra input would be fantastic -- and that goes triple for actual code!
Take a look at the X4 Savegame XML Analyzer I wrote last week.
It'll break down an unarchived savegame file into a flat tag/ownership/property/type list that you should be able to use as a base to write any automated scaffolding tools ya want.
If you need any more features, just let me know and I'll see what I can add in as time permits.
Re: Happy 2022! Now, about those save times...
Complimentary Code Link -- DAG Snapshots - Double-Write vs New/Delete - Large Set Comparison
Well it's been a couple days... and for a change... the code sample I posted didn't get flamed with the fires of a thousand dying suns...
So I'll take that as a sign we're moving in a positive direction
In that light, I found a bit of time this evening to run some larger scale performance tests on the previous code set.
I must admit, I need a bit of a retraction on my opinion that the new/delete method supremely outperforms the double-write method in general play (non-save) operation.
After a good bit of testing, it does win out by a reasonable margin in non-save mode -- but not as much as I expected.
And it loses out in save mode by a larger margin -- as expected, but to a greater degree than I thought it would.
--
Tested on randomized graphs
Double-Write: Best (Save Mode) = 2,318,061 writes per second -- maxed at ~630 MB RAM
New/Delete: Best (Non Save Mode) = 2,049,558 writes per second -- maxed at ~522 MB RAM
New/Delete did out-perform Double-Write in non-save mode as expected (2,049,558 > 1,922,466) -- but not by as much as expected.
Double-Write did out-perform New/Delete in save mode as expected (2,318,061 > 1,727,155) -- but by a lot more than expected.
So... using this method of distributed writes to minimize pre-serialization save time, if there's extra RAM to spare, I gotta admit the Double-Write method wins out handily.
Test code and full results writeup available in the GitHub gist.
Raw Results:
Note: Test Time == Average Iteration because I reduced the loopCount to 1 to grab a quick capture -- increase for finer tuned results.
Well it's been a couple days... and for a change... the code sample I posted didn't get flamed with the fires of a thousand dying suns...
So I'll take that as a sign we're moving in a positive direction
In that light, I found a bit of time this evening to run some larger scale performance tests on the previous code set.
I must admit, I need a bit of a retraction on my opinion that the new/delete method supremely outperforms the double-write method in general play (non-save) operation.
After a good bit of testing, it does win out by a reasonable margin in non-save mode -- but not as much as I expected.
And it loses out in save mode by a larger margin -- as expected, but to a greater degree than I thought it would.
--
Tested on randomized graphs
- 1 million vertices
- ~2.5 million edges (random 1-4 forward edges per vertex)
- Full graph copy performed per "iteration"
Double-Write: Best (Save Mode) = 2,318,061 writes per second -- maxed at ~630 MB RAM
New/Delete: Best (Non Save Mode) = 2,049,558 writes per second -- maxed at ~522 MB RAM
New/Delete did out-perform Double-Write in non-save mode as expected (2,049,558 > 1,922,466) -- but not by as much as expected.
Double-Write did out-perform New/Delete in save mode as expected (2,318,061 > 1,727,155) -- but by a lot more than expected.
So... using this method of distributed writes to minimize pre-serialization save time, if there's extra RAM to spare, I gotta admit the Double-Write method wins out handily.
Test code and full results writeup available in the GitHub gist.
Raw Results:
Code: Select all
========== GENERATE VERTICES ==========
Vertex Count: 1000000
Time: 842.312 ms
========== GENERATE GRAPH ==========
Edge Count: 2500110
Time: 1695.74 ms
========== PERFORMANCE TEST ==========
-[ isSaving: false ]-----
Test Time: 3.12099 s
Average Iteration: 3120.99 ms
Iterations per Second: 0.320411
Saveable Writes: 6000000
Saveable Writes per Second: 1922466
-[ isSaving: true ]-----
Test Time: 2.58837 s
Average Iteration: 2588.37 ms
Iterations per Second: 0.386344
Saveable Writes: 6000000
Saveable Writes per Second: 2318061
========== GENERATE VERTICES ==========
Vertex Count: 1000000
Time: 1043.79 ms
========== GENERATE GRAPH ==========
Edge Count: 2499328
Time: 2293.16 ms
========== PERFORMANCE TEST ==========
-[ isSaving: false ]-----
Test Time: 2.92746 s
Average Iteration: 2927.46 ms
Iterations per Second: 0.341593
Saveable Writes: 6000000
Saveable Writes per Second: 2049558
-[ isSaving: true ]-----
Test Time: 3.47392 s
Average Iteration: 3473.92 ms
Iterations per Second: 0.287859
Saveable Writes: 6000000
Saveable Writes per Second: 1727155
Last edited by xixas on Fri, 21. Jan 22, 13:15, edited 1 time in total.
Re: Happy 2022! Now, about those save times...
Speaking as an American who knows absolutely nothing about the technical aspects of this discussion, I seem to recall a previous discussion on save/load times where the save game needed to take longer because of some issue related to maintaining save game compatability with new patches. Is that part of the issue here?
-
- EGOSOFT
- Posts: 13093
- Joined: Sat, 9. Nov 02, 11:45
Re: Happy 2022! Now, about those save times...
A wild dev appears
Just wanted to say that there may be some improvements to savegame times through the 5.0 beta. Given that you're discussing it I thought I'd mention it. Given that it may behave differently on different systems we'd appreciate any observations you have. If you do have any feedback on 5.0, best to post that in the Public Beta forum.
Also, it's worth checking if your savegame folder is excluded from Windows file indexing. Here's a guide on how to check. Ditto for folder compression.
Carry on
Just wanted to say that there may be some improvements to savegame times through the 5.0 beta. Given that you're discussing it I thought I'd mention it. Given that it may behave differently on different systems we'd appreciate any observations you have. If you do have any feedback on 5.0, best to post that in the Public Beta forum.
Also, it's worth checking if your savegame folder is excluded from Windows file indexing. Here's a guide on how to check. Ditto for folder compression.
Carry on
Come watch me on Twitch where I occasionally play several of the X games
Re: Happy 2022! Now, about those save times...
For anyone following along that doesn't understand this "DAG" bit, or its relevance to save games:
One can visualize a Directed Acyclic Graph like a tree structure with interweaving branches.
You can only move forward through the graph by following its edges, though there may be more than one path to get to each vertex after the second level.
A vertex is an informational junction point in the graph -- for our purposes, it's where we store game datapoints (ship/weapon stats, station coordinates, etc).
An edge is a directional relationship between two vertices -- here it represents the this-owns-that or this-is-targeting-that nature between game elements.
A savegame is an XML representation of this sort of relational data model.
Though the in-game data may not be directed and reference identifiers are stored to jump the graph, it must still be structured as such for a nested format like XML.
These tests are attempting to determine if the save time on such a graph can be offloaded from point-of-save (i.e. reading the whole tree and data at once) to a supplemental memory write pattern that requires minimal-to-no delay before translating and writing to disk.
In an ideal version of such a system, a live rolling snapshot of the graph can be pulled at any time and a save can be performed in the background without pausing play.
One can visualize a Directed Acyclic Graph like a tree structure with interweaving branches.
You can only move forward through the graph by following its edges, though there may be more than one path to get to each vertex after the second level.
A vertex is an informational junction point in the graph -- for our purposes, it's where we store game datapoints (ship/weapon stats, station coordinates, etc).
An edge is a directional relationship between two vertices -- here it represents the this-owns-that or this-is-targeting-that nature between game elements.
A savegame is an XML representation of this sort of relational data model.
Though the in-game data may not be directed and reference identifiers are stored to jump the graph, it must still be structured as such for a nested format like XML.
These tests are attempting to determine if the save time on such a graph can be offloaded from point-of-save (i.e. reading the whole tree and data at once) to a supplemental memory write pattern that requires minimal-to-no delay before translating and writing to disk.
In an ideal version of such a system, a live rolling snapshot of the graph can be pulled at any time and a save can be performed in the background without pausing play.
Last edited by xixas on Fri, 21. Jan 22, 20:43, edited 1 time in total.
Re: Happy 2022! Now, about those save times...
Thanks for the updateXenon_Slayer wrote: ↑Fri, 21. Jan 22, 10:45A wild dev appears
Just wanted to say that there may be some improvements to savegame times through the 5.0 beta. Given that you're discussing it I thought I'd mention it. Given that it may behave differently on different systems we'd appreciate any observations you have. If you do have any feedback on 5.0, best to post that in the Public Beta forum.
Also, it's worth checking if your savegame folder is excluded from Windows file indexing. Here's a guide on how to check. Ditto for folder compression.
Carry on
I'm on Linux myself, but the Windows note is good for any Windows users that might be following along here.
I don't think I've found the time to play anything in the past month, so it's unlikely I'll hop into the beta.
But if anyone here does, I'd love to hear about any marked improvements.
I don't recall coming across that comment, but I'm not sure why patch compatibility would extend the save time... unless there's heavy branching logic to support multiple savegame formats (?)
Re: Happy 2022! Now, about those save times...
Complimentary Code Link: Tiny Space (github gist, C++11, no dependencies)
Hey all. I had some free time this evening, so I wrote up a mini space sim -- affectionately called Tiny Space -- to act as a base for some more realistic testing.
It's all rolled up into a single cpp for easy copy. I wrote it up pretty quick, so it still needs some cleanup, but the basics are there and I'd rather share sooner than later in case someone has a use.
Ships fly around, hopping sectors. Speed depends on ship class.
I'll probably add some simple combat, jump gates, stations, tasks, and basic path finding as time permits -- whatever will add some complexity for snapshot testing
Displays in-sector ship information, a sector map, and a global map showing how many ships are in each sector.
There are no controls. It's just a self-perpetuating sim. Quit with <Ctrl+C>.
Adjust refresh rate, ship count, etc. in main().
Notably, I find setting sectorSize = {20, 20} more comfortable to look at... but it takes up more screen real estate while I'm coding
Sample Display (v1, 10x10):
Hey all. I had some free time this evening, so I wrote up a mini space sim -- affectionately called Tiny Space -- to act as a base for some more realistic testing.
It's all rolled up into a single cpp for easy copy. I wrote it up pretty quick, so it still needs some cleanup, but the basics are there and I'd rather share sooner than later in case someone has a use.
Ships fly around, hopping sectors. Speed depends on ship class.
I'll probably add some simple combat, jump gates, stations, tasks, and basic path finding as time permits -- whatever will add some complexity for snapshot testing
Displays in-sector ship information, a sector map, and a global map showing how many ships are in each sector.
There are no controls. It's just a self-perpetuating sim. Quit with <Ctrl+C>.
Adjust refresh rate, ship count, etc. in main().
Notably, I find setting sectorSize = {20, 20} more comfortable to look at... but it takes up more screen real estate while I'm coding
Sample Display (v1, 10x10):
Code: Select all
> ( name:F001 code:AEI-563 loc:{-1.4, 1.7} dir:{-0.4,-0.9} class:Frigate )
( name:C031 code:YHV-953 loc:{ 0.3, 1.3} dir:{ 0.3, 1.0} class:Corvette )
( name:S059 code:RKC-685 loc:{-3.6, 3.3} dir:{-1.0, 0.1} class:Scout )
( name:S062 code:CTW-818 loc:{ 4.4, 4.2} dir:{-1.0,-0.0} class:Scout )
( name:F078 code:RTF-828 loc:{-1.7,-4.2} dir:{ 0.1, 1.0} class:Frigate )
( name:F119 code:YZC-396 loc:{ 2.6,-1.9} dir:{ 0.3,-1.0} class:Frigate )
+-[ 08D ]-------------------------+
| |
| . |
| |
| . |
| |
| |
| . |
| ^ |
| . |
| . |
| |
+---------------------------------+
A B C D E F G H I J
.----------------------------------------------------------------------
01 | 3 5 7 7 7 4 3 6 5 5
02 | 6 5 7 7 7 1 4 5 1 4
03 | 7 3 2 11 5 6 2 5 4
04 | 2 5 10 3 3 8 2 3 3 8
05 | 2 3 5 7 7 8 7 8 7 7
06 | 4 4 4 6 3 5 5 3 7 5
07 | 6 2 3 5 3 1 8 5 3 2
08 | 4 5 7 [ 6 ] 3 4 5 8 3 6
09 | 5 4 7 3 6 6 5 3 6 6
10 | 9 7 2 5 6 10 6 7 6 4
work: 0.084438ms display: 0.598263ms
Re: Happy 2022! Now, about those save times...
Misc update -- I found a bit of time last night to add a color UI and gate travel to Tiny Space v2.
If I can find a couple hours here and there in the evenings this week to get combat and stations in, it should have enough complexity to perform a basic mimic for snapshotting
New Options:
If I can find a couple hours here and there in the evenings this week to get combat and stations in, it should have enough complexity to perform a basic mimic for snapshotting
New Options:
- --color - Enable color display (for terminals that support ANSI color codes)
- --no-jumpgates - Disable jumpgate travel and revert to the original fly-between-sectors style.
Last edited by Terre on Tue, 25. Jan 22, 07:08, edited 1 time in total.
Reason: Images posted directly to the forums should not be greater than 640x480 or 100kb, oversize image now linked
Reason: Images posted directly to the forums should not be greater than 640x480 or 100kb, oversize image now linked
Re: Happy 2022! Now, about those save times...
no significant improvement detected for 5.0b2, sorry,Xenon_Slayer wrote: ↑Fri, 21. Jan 22, 10:45If you do have any feedback on 5.0, best to post that in the Public Beta forum.
27 sec for early game (20 min after start),
and 1 minute for late game.
save time.
p.s. may be 10% improvement if strictly use same unmodified games (i don't have them, so i used others unmodified saves for measurement),
but it feels same.
Re: Happy 2022! Now, about those save times...
Note: I know it's been all week without an update, but the thread's not dead, I've just been busy
Added randomized jumpgates, stations, ship factions, and player-owned ships to Tiny Space across a few late-night sessions.
I just haven't quite had time to finish basic combat.
It's close -- just need to finish the respawn timers -- and maybe add a kill screen for the wait.
They're just dots on a screen, but I didn't want to write sub-par combat.
Military ships' main guns fire in a 90 degree forward arc, save for frigates, who have side-fire.
Beam weapons are damage over time, while others are per-shot damage.
Targeting is performed per-ship and independently per-weapon, with turrets having full freedom of rotation... so while a courier or transport may not select a primary target, it will rear-fire turrets while escaping.
My brain says that I'm missing a part-second weapon cooldown reset somewhere...
...and I haven't had time to write prioritized follow, strafe, or angle-of-engagement code yet, so the frigates are currently a bit gimped since they're not rotating for full engagement... but I'll get there in the next version.
Anyway, I expect to have an update tomorrow.
Current specs:
Of course, scripting engine variability can't be taken into consideration in any external model... so, even though it hasn't been mentioned, if that's a developer concern, someone let me know and I'll find a way to fake it in the model.
For that matter, if anything -- save for render time, obviously -- requires additional complexity for argument's sake, let me know!
As an aside, mining is also in the works, but I haven't mainlined it... I haven't had time to set up a supply-demand chain yet.
Once v3 drops, I'll hop back on the real-time save mechanics and get this thread back on track
Added randomized jumpgates, stations, ship factions, and player-owned ships to Tiny Space across a few late-night sessions.
I just haven't quite had time to finish basic combat.
It's close -- just need to finish the respawn timers -- and maybe add a kill screen for the wait.
They're just dots on a screen, but I didn't want to write sub-par combat.
Military ships' main guns fire in a 90 degree forward arc, save for frigates, who have side-fire.
Beam weapons are damage over time, while others are per-shot damage.
Targeting is performed per-ship and independently per-weapon, with turrets having full freedom of rotation... so while a courier or transport may not select a primary target, it will rear-fire turrets while escaping.
My brain says that I'm missing a part-second weapon cooldown reset somewhere...
...and I haven't had time to write prioritized follow, strafe, or angle-of-engagement code yet, so the frigates are currently a bit gimped since they're not rotating for full engagement... but I'll get there in the next version.
Anyway, I expect to have an update tomorrow.
Current specs:
- 100 sectors
- 500 ships -- comfortably handles over 10k, but who wants to look at that many overlapping dots? Increase the count if ya want to. Won't hurt my feelings
- Constant position, trajectory, waypoint, and target storage (per ship and per-weapon) -- with system-unique IDs
- No OOS combat -- all combat happens in real time in all sectors -- though I'll gladly update this with remote delay timers if anyone thinks slower combat will increase real-time complexity
Of course, scripting engine variability can't be taken into consideration in any external model... so, even though it hasn't been mentioned, if that's a developer concern, someone let me know and I'll find a way to fake it in the model.
For that matter, if anything -- save for render time, obviously -- requires additional complexity for argument's sake, let me know!
As an aside, mining is also in the works, but I haven't mainlined it... I haven't had time to set up a supply-demand chain yet.
Once v3 drops, I'll hop back on the real-time save mechanics and get this thread back on track
Re: Happy 2022! Now, about those save times...
Tiny Space v3 is finally up
screenshot - C++11, no dependencies
New in this version (v3):
screenshot - C++11, no dependencies
New in this version (v3):
- Randomized jumpgate positions
- Added factions and player-owned ships
- Added stations, docking, and repair
- Added simple combat, killscreen, and respawn
Re: Happy 2022! Now, about those save times...
With 5.00 Public Beta 3 now available to those participating, now would seem like a good time to briefly respond to this thread. The TL;DR version for those for whose eyes glaze over when programmers start talking: saving is now a bit quicker!
For those interested in a more detailed explanation, the changes we've made are, in increasing order of significance:
1. We've improved the raw performance of the serialisation of numeric values with the help of fmt.
2. We've also improved the process by which savegame signatures are calculated, so that it's done during the serialisation process rather than as a separate step at the end. The performance effect of this is relatively small; the primary benefit is that it eliminates a very large spike in memory usage, which could previously occasionally cause the game to run out of memory and crash for some people.
3. This is probably the most interesting one for the OP of this thread, who proposed something similar: we've changed the interaction between two key processes: the process of extracting data that needs to be saved, and the process of serializing that data. These were already separate steps, but they were previously interleaved on the same thread; now the two can run mostly in parallel. Note that this doesn't change the fundamental fact that the game needs to be paused while the first of these two processes completes.
We do have a few other ideas up our sleeves for further improvements, but none that are likely to be possible before 5.00 hits full release, and none that are magically going to make it possible to save the game without a noticeable pause.
For those interested in a more detailed explanation, the changes we've made are, in increasing order of significance:
1. We've improved the raw performance of the serialisation of numeric values with the help of fmt.
2. We've also improved the process by which savegame signatures are calculated, so that it's done during the serialisation process rather than as a separate step at the end. The performance effect of this is relatively small; the primary benefit is that it eliminates a very large spike in memory usage, which could previously occasionally cause the game to run out of memory and crash for some people.
3. This is probably the most interesting one for the OP of this thread, who proposed something similar: we've changed the interaction between two key processes: the process of extracting data that needs to be saved, and the process of serializing that data. These were already separate steps, but they were previously interleaved on the same thread; now the two can run mostly in parallel. Note that this doesn't change the fundamental fact that the game needs to be paused while the first of these two processes completes.
We do have a few other ideas up our sleeves for further improvements, but none that are likely to be possible before 5.00 hits full release, and none that are magically going to make it possible to save the game without a noticeable pause.
Re: Happy 2022! Now, about those save times...
Awesome - many thanks for coming back to us CBJ
--- chew-ie, lurker
--- chew-ie, lurker
Spoiler
Show
BurnIt: Boron and leaks don't go well together...
Königinnenreich von Boron: Sprich mit deinem Flossenführer
Nila Ti: Folgt mir, ihr Kavalkade von neugierigen Kreaturen!
Pick yourpoison seed [for custom gamestarts]
Feature request: paint jobs on custom starts
Königinnenreich von Boron: Sprich mit deinem Flossenführer
Nila Ti: Folgt mir, ihr Kavalkade von neugierigen Kreaturen!
Pick your
Feature request: paint jobs on custom starts
Re: Happy 2022! Now, about those save times...
Ah, I see. Did you ever consider using spectral morphology quantum encapsulated binary... holistic transitions instead? Just sayin'... but I'm sure your solution is just as good.CBJ wrote: ↑Thu, 3. Feb 22, 19:091. We've improved the raw performance of the serialisation of numeric values with the help of fmt.
2. We've also improved the process by which savegame signatures are calculated, so that it's done during the serialisation process rather than as a separate step at the end. The performance effect of this is relatively small; the primary benefit is that it eliminates a very large spike in memory usage, which could previously occasionally cause the game to run out of memory and crash for some people.
3. This is probably the most interesting one for the OP of this thread, who proposed something similar: we've changed the interaction between two key processes: the process of extracting data that needs to be saved, and the process of serializing that data. These were already separate steps, but they were previously interleaved on the same thread; now the two can run mostly in parallel. Note that this doesn't change the fundamental fact that the game needs to be paused while the first of these two processes completes.
"In the beginning the Universe was created. This has made a lot of people very angry and has been widely regarded as a bad move." - D.N.A
Re: Happy 2022! Now, about those save times...
Surprised me how well this works compaired to beta 2.CBJ wrote: ↑Thu, 3. Feb 22, 19:09With 5.00 Public Beta 3 now available to those participating, now would seem like a good time to briefly respond to this thread. The TL;DR version for those for whose eyes glaze over when programmers start talking: saving is now a bit quicker!
For those interested in a more detailed explanation, the changes we've made are, in increasing order of significance:
1. We've improved the raw performance of the serialisation of numeric values with the help of fmt.
2. We've also improved the process by which savegame signatures are calculated, so that it's done during the serialisation process rather than as a separate step at the end. The performance effect of this is relatively small; the primary benefit is that it eliminates a very large spike in memory usage, which could previously occasionally cause the game to run out of memory and crash for some people.
3. This is probably the most interesting one for the OP of this thread, who proposed something similar: we've changed the interaction between two key processes: the process of extracting data that needs to be saved, and the process of serializing that data. These were already separate steps, but they were previously interleaved on the same thread; now the two can run mostly in parallel. Note that this doesn't change the fundamental fact that the game needs to be paused while the first of these two processes completes.
We do have a few other ideas up our sleeves for further improvements, but none that are likely to be possible before 5.00 hits full release, and none that are magically going to make it possible to save the game without a noticeable pause.
Circa 17 seconds for a well established game, not even time to make a cup of tea
Laptop Dell G15 5510 : Win 11 x64
CPU - 10th Gen' Core I7 10870H 2.2-5.0ghz, GPU - NVidia Geforce RTX 3060, VRAM 6gb GDDR5,
RAM - 32gb (2x16gb, Dual Channel mode set in BIOS) DDR4 2933mhz Kingston Fury Impact,
SSD - Kioxia M.2 NVME 512gb (System), + Samsung M.2 NVME 970 Evo Plus 1tb (Games)
Long live Queen Polypheides and may her tentacles always be supple.
Seeker of Sohnen.
CPU - 10th Gen' Core I7 10870H 2.2-5.0ghz, GPU - NVidia Geforce RTX 3060, VRAM 6gb GDDR5,
RAM - 32gb (2x16gb, Dual Channel mode set in BIOS) DDR4 2933mhz Kingston Fury Impact,
SSD - Kioxia M.2 NVME 512gb (System), + Samsung M.2 NVME 970 Evo Plus 1tb (Games)
Long live Queen Polypheides and may her tentacles always be supple.
Seeker of Sohnen.