Happy 2022! Now, about those save times...

This forum is the ideal place for all discussion relating to X4. You will also find additional information from developers here.

Moderator: Moderators for English X Forum

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Wed, 19. Jan 22, 01:25

Hey Cdaragorn, I missed your post this morning. My apologies.
Cdaragorn wrote:
Sat, 15. Jan 22, 05:33
I've looked at Protobuf before but didn't use it because it doesn't allow random access to the data. Basically it suffers the same weakness that any text format does because it's order dependent. If you can't truly split the data up you can't use good Data Oriented Design patterns to speed the data handling up.
For files, they do have a standard sequential read, that's true, because they're designed for RPC -- basically just reading a message stream in off the wire.

But each message is serialized independently, and as I alluded to in a prior message, there's no reason we can't leave an empty block at the beginning of the file, drop a size header on every message (tallying them as we write), and then just seek back to file head and write the total data size written block -- maybe leave a few blocks and store a small index -- for example where the closest message starts are located with the file size divided in 8 pieces. Then we could fire up multiple reader threads to decrease load time :)
Cdaragorn wrote:
Sat, 15. Jan 22, 05:33
Also there are ways to avoid having to write that much custom stuff but when you're talking about performance you're also going to have to accept handling more custom code. Still you can make it a lot simpler than it sounds.
Yeah... boilerplate never goes away.
But that's why I'm trying to automate scaffolding the "message" (serializer) classes.
You know, DRY principle and all that.
Cdaragorn wrote:
Sat, 15. Jan 22, 05:33
Storing everything in memory first is a nice theory but in practice when working with this much data that actually creates problems that prevent it from getting the speed you're expecting from it.
The best system I've worked with involved streaming the data out of that memory at the same time as you were putting it in so it never takes up too much memory. That gets back to the need for true random access.
We know the entire universe, as written to the save file, is already fully stored in memory (in a much more space-efficient manner).
So I don't think that's an issue... unless I'm misunderstanding what you're saying, in which case, please elucidate.
Cdaragorn wrote:
Sat, 15. Jan 22, 05:33
I'd love to help if you have some data examples to work with. I guess I could pull apart a save file to come up with an example. I admit I'm having a hard time having much oomf to try just because I don't feel like it's likely to go anywhere.
That'd be awesome.
I've had a couple crossover releases this past week and haven't had nearly as much time as I usually do, so some extra input would be fantastic -- and that goes triple for actual code! :)

Take a look at the X4 Savegame XML Analyzer I wrote last week.
It'll break down an unarchived savegame file into a flat tag/ownership/property/type list that you should be able to use as a base to write any automated scaffolding tools ya want.
If you need any more features, just let me know and I'll see what I can add in as time permits.

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Fri, 21. Jan 22, 04:29

Complimentary Code Link -- DAG Snapshots - Double-Write vs New/Delete - Large Set Comparison

Well it's been a couple days... and for a change... the code sample I posted didn't get flamed with the fires of a thousand dying suns... :lol:
So I'll take that as a sign we're moving in a positive direction :)

In that light, I found a bit of time this evening to run some larger scale performance tests on the previous code set.
I must admit, I need a bit of a retraction on my opinion that the new/delete method supremely outperforms the double-write method in general play (non-save) operation.

After a good bit of testing, it does win out by a reasonable margin in non-save mode -- but not as much as I expected.
And it loses out in save mode by a larger margin -- as expected, but to a greater degree than I thought it would.

--

Tested on randomized graphs
  • 1 million vertices
  • ~2.5 million edges (random 1-4 forward edges per vertex)
  • Full graph copy performed per "iteration"
(Note that these tests are save-ready, immediately serializable value writes)

Double-Write: Best (Save Mode) = 2,318,061 writes per second -- maxed at ~630 MB RAM
New/Delete: Best (Non Save Mode) = 2,049,558 writes per second -- maxed at ~522 MB RAM

New/Delete did out-perform Double-Write in non-save mode as expected (2,049,558 > 1,922,466) -- but not by as much as expected.
Double-Write did out-perform New/Delete in save mode as expected (2,318,061 > 1,727,155) -- but by a lot more than expected.

So... using this method of distributed writes to minimize pre-serialization save time, if there's extra RAM to spare, I gotta admit the Double-Write method wins out handily.

Test code and full results writeup available in the GitHub gist.

Raw Results:

Code: Select all

========== GENERATE VERTICES ==========
Vertex Count: 1000000
Time: 842.312 ms
========== GENERATE GRAPH ==========
Edge Count: 2500110
Time: 1695.74 ms
========== PERFORMANCE TEST ==========

-[ isSaving: false ]-----
Test Time:                  3.12099 s
Average Iteration:          3120.99 ms
Iterations per Second:      0.320411
Saveable Writes:            6000000
Saveable Writes per Second: 1922466

-[ isSaving: true ]-----
Test Time:                  2.58837 s
Average Iteration:          2588.37 ms
Iterations per Second:      0.386344
Saveable Writes:            6000000
Saveable Writes per Second: 2318061

========== GENERATE VERTICES ==========
Vertex Count: 1000000
Time: 1043.79 ms
========== GENERATE GRAPH ==========
Edge Count: 2499328
Time: 2293.16 ms
========== PERFORMANCE TEST ==========

-[ isSaving: false ]-----
Test Time:                  2.92746 s
Average Iteration:          2927.46 ms
Iterations per Second:      0.341593
Saveable Writes:            6000000
Saveable Writes per Second: 2049558

-[ isSaving: true ]-----
Test Time:                  3.47392 s
Average Iteration:          3473.92 ms
Iterations per Second:      0.287859
Saveable Writes:            6000000
Saveable Writes per Second: 1727155
Note: Test Time == Average Iteration because I reduced the loopCount to 1 to grab a quick capture -- increase for finer tuned results.
Last edited by xixas on Fri, 21. Jan 22, 13:15, edited 1 time in total.

Falcrack
Posts: 4927
Joined: Wed, 29. Jul 09, 00:46
x4

Re: Happy 2022! Now, about those save times...

Post by Falcrack » Fri, 21. Jan 22, 06:12

Speaking as an American who knows absolutely nothing about the technical aspects of this discussion, I seem to recall a previous discussion on save/load times where the save game needed to take longer because of some issue related to maintaining save game compatability with new patches. Is that part of the issue here?

Xenon_Slayer
EGOSOFT
EGOSOFT
Posts: 13088
Joined: Sat, 9. Nov 02, 11:45
x4

Re: Happy 2022! Now, about those save times...

Post by Xenon_Slayer » Fri, 21. Jan 22, 10:45

A wild dev appears

Just wanted to say that there may be some improvements to savegame times through the 5.0 beta. Given that you're discussing it I thought I'd mention it. Given that it may behave differently on different systems we'd appreciate any observations you have. If you do have any feedback on 5.0, best to post that in the Public Beta forum.

Also, it's worth checking if your savegame folder is excluded from Windows file indexing. Here's a guide on how to check. Ditto for folder compression.

Carry on :)
Come watch me on Twitch where I occasionally play several of the X games

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Fri, 21. Jan 22, 13:59

For anyone following along that doesn't understand this "DAG" bit, or its relevance to save games:

One can visualize a Directed Acyclic Graph like a tree structure with interweaving branches.
You can only move forward through the graph by following its edges, though there may be more than one path to get to each vertex after the second level.

A vertex is an informational junction point in the graph -- for our purposes, it's where we store game datapoints (ship/weapon stats, station coordinates, etc).
An edge is a directional relationship between two vertices -- here it represents the this-owns-that or this-is-targeting-that nature between game elements.

A savegame is an XML representation of this sort of relational data model.
Though the in-game data may not be directed and reference identifiers are stored to jump the graph, it must still be structured as such for a nested format like XML.

These tests are attempting to determine if the save time on such a graph can be offloaded from point-of-save (i.e. reading the whole tree and data at once) to a supplemental memory write pattern that requires minimal-to-no delay before translating and writing to disk.
In an ideal version of such a system, a live rolling snapshot of the graph can be pulled at any time and a save can be performed in the background without pausing play.
Last edited by xixas on Fri, 21. Jan 22, 20:43, edited 1 time in total.

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Fri, 21. Jan 22, 20:19

Xenon_Slayer wrote:
Fri, 21. Jan 22, 10:45
A wild dev appears

Just wanted to say that there may be some improvements to savegame times through the 5.0 beta. Given that you're discussing it I thought I'd mention it. Given that it may behave differently on different systems we'd appreciate any observations you have. If you do have any feedback on 5.0, best to post that in the Public Beta forum.

Also, it's worth checking if your savegame folder is excluded from Windows file indexing. Here's a guide on how to check. Ditto for folder compression.

Carry on :)
Thanks for the update :)

I'm on Linux myself, but the Windows note is good for any Windows users that might be following along here.

I don't think I've found the time to play anything in the past month, so it's unlikely I'll hop into the beta.
But if anyone here does, I'd love to hear about any marked improvements.
Falcrack wrote:
Fri, 21. Jan 22, 06:12
I seem to recall a previous discussion on save/load times where the save game needed to take longer because of some issue related to maintaining save game compatability with new patches. Is that part of the issue here?
I don't recall coming across that comment, but I'm not sure why patch compatibility would extend the save time... unless there's heavy branching logic to support multiple savegame formats (?)

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Sun, 23. Jan 22, 11:02

Complimentary Code Link: Tiny Space (github gist, C++11, no dependencies)

Hey all. I had some free time this evening, so I wrote up a mini space sim -- affectionately called Tiny Space -- to act as a base for some more realistic testing.
It's all rolled up into a single cpp for easy copy. I wrote it up pretty quick, so it still needs some cleanup, but the basics are there and I'd rather share sooner than later in case someone has a use.

Ships fly around, hopping sectors. Speed depends on ship class.
I'll probably add some simple combat, jump gates, stations, tasks, and basic path finding as time permits -- whatever will add some complexity for snapshot testing :)

Displays in-sector ship information, a sector map, and a global map showing how many ships are in each sector.

There are no controls. It's just a self-perpetuating sim. Quit with <Ctrl+C>.

Adjust refresh rate, ship count, etc. in main().
Notably, I find setting sectorSize = {20, 20} more comfortable to look at... but it takes up more screen real estate while I'm coding :P

Sample Display (v1, 10x10):

Code: Select all

> ( name:F001 code:AEI-563 loc:{-1.4, 1.7} dir:{-0.4,-0.9} class:Frigate   )
  ( name:C031 code:YHV-953 loc:{ 0.3, 1.3} dir:{ 0.3, 1.0} class:Corvette  )
  ( name:S059 code:RKC-685 loc:{-3.6, 3.3} dir:{-1.0, 0.1} class:Scout     )
  ( name:S062 code:CTW-818 loc:{ 4.4, 4.2} dir:{-1.0,-0.0} class:Scout     )
  ( name:F078 code:RTF-828 loc:{-1.7,-4.2} dir:{ 0.1, 1.0} class:Frigate   )
  ( name:F119 code:YZC-396 loc:{ 2.6,-1.9} dir:{ 0.3,-1.0} class:Frigate   )

                   +-[ 08D ]-------------------------+
                   |                                 |
                   |          .                      |
                   |                                 |
                   |                         .       |
                   |                                 |
                   |                                 |
                   |                .                |
                   |             ^                   |
                   |    .                            |
                   |                            .    |
                   |                                 |
                   +---------------------------------+

        A      B      C      D      E      F      G      H      I      J  
   .----------------------------------------------------------------------
01 |    3      5      7      7      7      4      3      6      5      5  
02 |    6      5      7      7      7      1      4      5      1      4  
03 |    7      3      2     11      5             6      2      5      4  
04 |    2      5     10      3      3      8      2      3      3      8  
05 |    2      3      5      7      7      8      7      8      7      7  
06 |    4      4      4      6      3      5      5      3      7      5  
07 |    6      2      3      5      3      1      8      5      3      2  
08 |    4      5      7  [   6 ]    3      4      5      8      3      6  
09 |    5      4      7      3      6      6      5      3      6      6  
10 |    9      7      2      5      6     10      6      7      6      4  

work: 0.084438ms  display: 0.598263ms

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Tue, 25. Jan 22, 00:01

Misc update -- I found a bit of time last night to add a color UI and gate travel to Tiny Space v2.

If I can find a couple hours here and there in the evenings this week to get combat and stations in, it should have enough complexity to perform a basic mimic for snapshotting :)

New Options:
  • --color - Enable color display (for terminals that support ANSI color codes)
  • --no-jumpgates - Disable jumpgate travel and revert to the original fly-between-sectors style.
https://user-images.githubusercontent.c ... 17741b.png
Last edited by Terre on Tue, 25. Jan 22, 07:08, edited 1 time in total.
Reason: Images posted directly to the forums should not be greater than 640x480 or 100kb, oversize image now linked

dmk
Posts: 682
Joined: Fri, 25. Nov 05, 00:59
x4

Re: Happy 2022! Now, about those save times...

Post by dmk » Thu, 27. Jan 22, 20:03

Xenon_Slayer wrote:
Fri, 21. Jan 22, 10:45
If you do have any feedback on 5.0, best to post that in the Public Beta forum.
no significant improvement detected for 5.0b2, sorry,
27 sec for early game (20 min after start),
and 1 minute for late game.
save time.
p.s. may be 10% improvement if strictly use same unmodified games (i don't have them, so i used others unmodified saves for measurement),
but it feels same.

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Sun, 30. Jan 22, 09:59

Note: I know it's been all week without an update, but the thread's not dead, I've just been busy :)

Added randomized jumpgates, stations, ship factions, and player-owned ships to Tiny Space across a few late-night sessions.
I just haven't quite had time to finish basic combat.
It's close -- just need to finish the respawn timers -- and maybe add a kill screen for the wait.

They're just dots on a screen, but I didn't want to write sub-par combat.
Military ships' main guns fire in a 90 degree forward arc, save for frigates, who have side-fire.
Beam weapons are damage over time, while others are per-shot damage.
Targeting is performed per-ship and independently per-weapon, with turrets having full freedom of rotation... so while a courier or transport may not select a primary target, it will rear-fire turrets while escaping.

My brain says that I'm missing a part-second weapon cooldown reset somewhere...
...and I haven't had time to write prioritized follow, strafe, or angle-of-engagement code yet, so the frigates are currently a bit gimped since they're not rotating for full engagement... but I'll get there in the next version.

Anyway, I expect to have an update tomorrow.

Current specs:
  • 100 sectors
  • 500 ships -- comfortably handles over 10k, but who wants to look at that many overlapping dots? Increase the count if ya want to. Won't hurt my feelings ;)
  • Constant position, trajectory, waypoint, and target storage (per ship and per-weapon) -- with system-unique IDs
  • No OOS combat -- all combat happens in real time in all sectors -- though I'll gladly update this with remote delay timers if anyone thinks slower combat will increase real-time complexity
The complexity's s high enough now that I ought to be able to write a simple serializer and real-time snapshot scheme that approximates X's complexity.
Of course, scripting engine variability can't be taken into consideration in any external model... so, even though it hasn't been mentioned, if that's a developer concern, someone let me know and I'll find a way to fake it in the model.

For that matter, if anything -- save for render time, obviously -- requires additional complexity for argument's sake, let me know!

As an aside, mining is also in the works, but I haven't mainlined it... I haven't had time to set up a supply-demand chain yet.

Once v3 drops, I'll hop back on the real-time save mechanics and get this thread back on track :)

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Mon, 31. Jan 22, 01:27

Tiny Space v3 is finally up :)
screenshot - C++11, no dependencies

New in this version (v3):
  • Randomized jumpgate positions
  • Added factions and player-owned ships
  • Added stations, docking, and repair
  • Added simple combat, killscreen, and respawn
I'll write up a separate gist with snapshots and XML-serialized saves over the next couple days as time permits.

CBJ
EGOSOFT
EGOSOFT
Posts: 51740
Joined: Tue, 29. Apr 03, 00:56
x4

Re: Happy 2022! Now, about those save times...

Post by CBJ » Thu, 3. Feb 22, 19:09

With 5.00 Public Beta 3 now available to those participating, now would seem like a good time to briefly respond to this thread. The TL;DR version for those for whose eyes glaze over when programmers start talking: saving is now a bit quicker!

For those interested in a more detailed explanation, the changes we've made are, in increasing order of significance:

1. We've improved the raw performance of the serialisation of numeric values with the help of fmt.
2. We've also improved the process by which savegame signatures are calculated, so that it's done during the serialisation process rather than as a separate step at the end. The performance effect of this is relatively small; the primary benefit is that it eliminates a very large spike in memory usage, which could previously occasionally cause the game to run out of memory and crash for some people.
3. This is probably the most interesting one for the OP of this thread, who proposed something similar: we've changed the interaction between two key processes: the process of extracting data that needs to be saved, and the process of serializing that data. These were already separate steps, but they were previously interleaved on the same thread; now the two can run mostly in parallel. Note that this doesn't change the fundamental fact that the game needs to be paused while the first of these two processes completes.

We do have a few other ideas up our sleeves for further improvements, but none that are likely to be possible before 5.00 hits full release, and none that are magically going to make it possible to save the game without a noticeable pause.

User avatar
chew-ie
Posts: 5439
Joined: Mon, 5. May 08, 00:05
x4

Re: Happy 2022! Now, about those save times...

Post by chew-ie » Thu, 3. Feb 22, 19:33

Awesome - many thanks for coming back to us CBJ :)

--- chew-ie, lurker

Image

Spoiler
Show
BurnIt: Boron and leaks don't go well together...
Königinnenreich von Boron: Sprich mit deinem Flossenführer
Nila Ti: Folgt mir, ihr Kavalkade von neugierigen Kreaturen!

:idea: Pick your poison seed [for custom gamestarts]

Artean
Posts: 1094
Joined: Tue, 14. Feb 06, 17:41
x4

Re: Happy 2022! Now, about those save times...

Post by Artean » Thu, 3. Feb 22, 20:31

CBJ wrote:
Thu, 3. Feb 22, 19:09
1. We've improved the raw performance of the serialisation of numeric values with the help of fmt.
2. We've also improved the process by which savegame signatures are calculated, so that it's done during the serialisation process rather than as a separate step at the end. The performance effect of this is relatively small; the primary benefit is that it eliminates a very large spike in memory usage, which could previously occasionally cause the game to run out of memory and crash for some people.
3. This is probably the most interesting one for the OP of this thread, who proposed something similar: we've changed the interaction between two key processes: the process of extracting data that needs to be saved, and the process of serializing that data. These were already separate steps, but they were previously interleaved on the same thread; now the two can run mostly in parallel. Note that this doesn't change the fundamental fact that the game needs to be paused while the first of these two processes completes.
Ah, I see. Did you ever consider using spectral morphology quantum encapsulated binary... holistic transitions instead? Just sayin'... but I'm sure your solution is just as good.
"In the beginning the Universe was created. This has made a lot of people very angry and has been widely regarded as a bad move." - D.N.A

User avatar
alt3rn1ty
Posts: 2297
Joined: Thu, 26. Jan 06, 19:45
x4

Re: Happy 2022! Now, about those save times...

Post by alt3rn1ty » Thu, 3. Feb 22, 23:25

CBJ wrote:
Thu, 3. Feb 22, 19:09
With 5.00 Public Beta 3 now available to those participating, now would seem like a good time to briefly respond to this thread. The TL;DR version for those for whose eyes glaze over when programmers start talking: saving is now a bit quicker!

For those interested in a more detailed explanation, the changes we've made are, in increasing order of significance:

1. We've improved the raw performance of the serialisation of numeric values with the help of fmt.
2. We've also improved the process by which savegame signatures are calculated, so that it's done during the serialisation process rather than as a separate step at the end. The performance effect of this is relatively small; the primary benefit is that it eliminates a very large spike in memory usage, which could previously occasionally cause the game to run out of memory and crash for some people.
3. This is probably the most interesting one for the OP of this thread, who proposed something similar: we've changed the interaction between two key processes: the process of extracting data that needs to be saved, and the process of serializing that data. These were already separate steps, but they were previously interleaved on the same thread; now the two can run mostly in parallel. Note that this doesn't change the fundamental fact that the game needs to be paused while the first of these two processes completes.

We do have a few other ideas up our sleeves for further improvements, but none that are likely to be possible before 5.00 hits full release, and none that are magically going to make it possible to save the game without a noticeable pause.
Surprised me how well this works compaired to beta 2.
Circa 17 seconds for a well established game, not even time to make a cup of tea :)
Laptop Dell G15 5510 : Win 11 x64
CPU - 10th Gen' Core I7 10870H 2.2-5.0ghz, GPU - NVidia Geforce RTX 3060, VRAM 6gb GDDR5,
RAM - 32gb (2x16gb, Dual Channel mode set in BIOS) DDR4 2933mhz Kingston Fury Impact,
SSD - Kioxia M.2 NVME 512gb (System), + Samsung M.2 NVME 970 Evo Plus 1tb (Games)

:boron: Long live Queen Polypheides and may her tentacles always be supple.
Seeker of Sohnen.

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Thu, 3. Feb 22, 23:32

CBJ wrote:
Thu, 3. Feb 22, 19:09
With 5.00 Public Beta 3 now available to those participating, now would seem like a good time to briefly respond to this thread. The TL;DR version for those for whose eyes glaze over when programmers start talking: saving is now a bit quicker!
I appreciate the reply, CBJ :)
I really am trying to avoid bothering the devs, at least until we can post some agreed upon samples/strategies, but I figured you guys were probably occasionally getting updates from this thread.
CBJ wrote:
Thu, 3. Feb 22, 19:09
1. We've improved the raw performance of the serialisation of numeric values with the help of fmt.
We're doing something similar to what fmt does in a game I'm on using a slightly more proprietary library -- basically just uses underlying malloc and sprintf calls attached directly to the custom string class.
For as much as the standard library makes C++ the handy, type-safe environment it's known for these days, performance sacrifices have been made all over the place to make that possible.
CBJ wrote:
Thu, 3. Feb 22, 19:09
2. We've also improved the process by which savegame signatures are calculated, so that it's done during the serialisation process rather than as a separate step at the end. The performance effect of this is relatively small; the primary benefit is that it eliminates a very large spike in memory usage, which could previously occasionally cause the game to run out of memory and crash for some people.
Cool -- though I have to wonder if a rolling signature was possible, why it couldn't be applied to an istream of the serailized data at the end before instead of spiking the memory, presumably to e.g. checksum the whole data set (?)
No response necessary, just pondering as I read along :)
CBJ wrote:
Thu, 3. Feb 22, 19:09
3. This is probably the most interesting one for the OP of this thread, who proposed something similar: we've changed the interaction between two key processes: the process of extracting data that needs to be saved, and the process of serializing that data. These were already separate steps, but they were previously interleaved on the same thread; now the two can run mostly in parallel. Note that this doesn't change the fundamental fact that the game needs to be paused while the first of these two processes completes.
That sounds like a great step forward!
Whether you're synchronizing around blocks or swapping queues across a thread pool, anything you can offload to save threads as soon as it's ready is a step in the right direction.
CBJ wrote:
Thu, 3. Feb 22, 19:09
We do have a few other ideas up our sleeves for further improvements, but none that are likely to be possible before 5.00 hits full release
The release cycle's always more important than any one feature (though less important than some bugs!)
No rush :)
CBJ wrote:
Thu, 3. Feb 22, 19:09
and none that are magically going to make it possible to save the game without a noticeable pause.
I'm still of the mindset that it's always possible to snapshot any recorded system, though the work required to apply associated strategies to an existing project can certainly be prohibitive.
It's certainly not magic, and I hope I've said nothing to make anyone believe I think otherwise.
In this line of work it's usually a hard matter of priorities, and P1 items are rarely the ones you want them to be.

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Fri, 4. Feb 22, 00:17

Alright, onto my tiny update.
I've not found as much time to put into this as usual this week, but it's still been simmering on the brain's back burner.

I did add a quick n' dirty XML serializer to Tiny Space that mimics the X4 process of freezing up, taking stock, and writing out an XML stream.
I wasn't planning on sharing 'til I'd had time to write the snapshot version, but I'd rather not hold back anything anyone might have occasion to look at ahead of time, so here ya go.
It runs the sim for 10 seconds, pauses execution, serializes to XML (std::cerr), and then exits because that was the whole point of the sample.
If you're skimming the code for changes instead of running a diff, just search for "serial" or "xml".

Sample Serialization:

Code: Select all

<savegame>
  <sectors count="100"/>
  <jumpgates count="252"/>
  <stations count="178"/>
  <ships count="500"/>
  <sector id="[0x0001]" rowcol="{0,0}" name="A01" size="{20,20}">
    <jumpgates count="2">
      <jumpgate id="[0x006a]" nesw="east" position="{19.5654,7.55381}" target="[0x0069]"/>
      <jumpgate id="[0x0065]" nesw="south" position="{10.8666,16.2743}" target="[0x0066]"/>
    </jumpgates>
    <stations count="5">
      <station id="[0x0161]" position="{15.7822,16.9681}"/>
      <station id="[0x0162]" position="{8.29011,7.06524}"/>
      <station id="[0x0163]" position="{3.39972,5.71385}"/>
      <station id="[0x0164]" position="{3.78286,13.9912}"/>
      <station id="[0x0165]" position="{17.1097,4.24493}"/>
    </stations>
    <ships count="9">
      <ship id="[0x0266]" type="Frigate" faction="Friend" code="UHF-839" name="F005" max-hull="1800" current-hull="1800" position="{6.85746,2.7734}" direction="{-0.76179,0.647824}" speed="0.4" destination-object="[0x0163]" destination-sector="[0x0001]" destination-position="{3.39972,5.71385}">
        <weapons count="4">
          <weapon id="[0x0267]" type="Cannon" weapon-position="Starboard"/>
          <weapon id="[0x0268]" type="Cannon" weapon-position="Starboard"/>
          <weapon id="[0x0269]" type="Cannon" weapon-position="Port"/>
          <weapon id="[0x026a]" type="Cannon" weapon-position="Port"/>
        </weapons>
        <turrets count="4">
          <weapon id="[0x026b]" type="Pulse"/>
          <weapon id="[0x026c]" type="Pulse"/>
          <weapon id="[0x026d]" type="Beam"/>
          <weapon id="[0x026e]" type="Beam"/>
        </turrets>
      </ship>
      <ship id="[0x02b6]" type="Corvette" faction="Neutral" code="BMD-350" name="C008" max-hull="1200" current-hull="1200" position="{15.9493,15.2375}" direction="{-0.0961514,0.995367}" speed="0.8" destination-object="[0x0161]" destination-sector="[0x0001]" destination-position="{15.7822,16.9681}">
        <weapons count="3">
          <weapon id="[0x02b7]" type="Pulse"/>
          <weapon id="[0x02b8]" type="Pulse"/>
          <weapon id="[0x02b9]" type="Cannon"/>
        </weapons>
        <turrets count="2">
          <weapon id="[0x02ba]" type="Pulse"/>
          <weapon id="[0x02bb]" type="Pulse"/>
        </turrets>
      </ship>
      <ship id="[0x0389]" type="Transport" faction="Friend" code="FXB-671" name="T039" max-hull="800" current-hull="800" position="{8.33853,13.9802}" direction="{0.928029,0.372509}" speed="0.6" destination-object="[0x0161]" destination-sector="[0x0001]" destination-position="{15.7822,16.9681}">
        <turrets count="2">
          <weapon id="[0x038a]" type="Pulse"/>
          <weapon id="[0x038b]" type="Pulse"/>
        </turrets>
      </ship>
      
      ...
Serializes to ~5300-6000 lines, written to stderr (so feel free to 1>/dev/null).
That doesn't sound like much compared to X4's ~20mil line save files, but it's about the real-time complexity of the data, not the quantity.
Quantity makes it take longer, but a required freeze or heavy collection sync is due to real-time complexity.

I actually did hack in a full snapshot as well, but it was super dirty due to inadequate type erasure on the original Saveable<T> class, requiring all kinds of ugly operator adjustments around conflicting operator overloads...
...so I'm not sharing that one :)

To try and keep base code adjustments minimal, I'm rewriting the Saveable class with simpler type erasure.
And thinking I may key data access off atomic thread references for this sample, forcibly blocking live and snapshot threads from unintentionally accessing each others' data.
I haven't thought much past that. We'll see how it goes when I find a few hours to dig in.

Importantly, I didn't write Tiny Space for nothing -- and I very intentionally didn't write it to take an easy snapshot.
I just cobbled together a quick game prototype same as I'd do for any other.
Dirty little secret -- a lot of the time prototype code makes it into final products :)

Anyway, if nothing else, I'll find some time this weekend.

--

P.S. The word of the day is dirty.

Skeeter
Posts: 3665
Joined: Thu, 9. Jan 03, 19:47
x3

Re: Happy 2022! Now, about those save times...

Post by Skeeter » Fri, 4. Feb 22, 05:47

I think pre beta3 i was getting 35 ish secs on a save for about a 5 hour ish ingame time play, 22mill credits and few plot missions done and small miner fleet in a mining sector in terran space using terran start. Now its about 17 secs. So it has improved.
[ external image ]
7600x cpu 5.4ghz 32gb DDR5 5600mhz 6700XT 32" 1440p mon

DavidGW
Posts: 345
Joined: Sat, 18. May 13, 06:40
x4

Re: Happy 2022! Now, about those save times...

Post by DavidGW » Fri, 4. Feb 22, 09:27

Ditto. 8700K CPU, 3200MHz DDR4 RAM.

Was around the 30 second per save mark, maybe a bit longer, on my 10 day game.

Now around 16 secs, Yay!!!

xixas
Posts: 40
Joined: Sat, 1. Jan 22, 22:47

Re: Happy 2022! Now, about those save times...

Post by xixas » Fri, 4. Feb 22, 10:40

alt3rn1ty wrote:
Thu, 3. Feb 22, 23:25
Surprised me how well this works compaired to beta 2.
Circa 17 seconds for a well established game, not even time to make a cup of tea :)
Skeeter wrote:
Fri, 4. Feb 22, 05:47
I think pre beta3 i was getting 35 ish secs on a save for about a 5 hour ish ingame time play, 22mill credits and few plot missions done and small miner fleet in a mining sector in terran space using terran start. Now its about 17 secs. So it has improved.
DavidGW wrote:
Fri, 4. Feb 22, 09:27
Ditto. 8700K CPU, 3200MHz DDR4 RAM.
Was around the 30 second per save mark, maybe a bit longer, on my 10 day game.
Now around 16 secs, Yay!!!
This is all very good news. Thanks for posting, guys :)

Locked

Return to “X4: Foundations”