For files, they do have a standard sequential read, that's true, because they're designed for RPC -- basically just reading a message stream in off the wire.Cdaragorn wrote: ↑Sat, 15. Jan 22, 05:33I've looked at Protobuf before but didn't use it because it doesn't allow random access to the data. Basically it suffers the same weakness that any text format does because it's order dependent. If you can't truly split the data up you can't use good Data Oriented Design patterns to speed the data handling up.
But each message is serialized independently, and as I alluded to in a prior message, there's no reason we can't leave an empty block at the beginning of the file, drop a size header on every message (tallying them as we write), and then just seek back to file head and write the total data size written block -- maybe leave a few blocks and store a small index -- for example where the closest message starts are located with the file size divided in 8 pieces. Then we could fire up multiple reader threads to decrease load time
Yeah... boilerplate never goes away.
But that's why I'm trying to automate scaffolding the "message" (serializer) classes.
You know, DRY principle and all that.
We know the entire universe, as written to the save file, is already fully stored in memory (in a much more space-efficient manner).Cdaragorn wrote: ↑Sat, 15. Jan 22, 05:33Storing everything in memory first is a nice theory but in practice when working with this much data that actually creates problems that prevent it from getting the speed you're expecting from it.
The best system I've worked with involved streaming the data out of that memory at the same time as you were putting it in so it never takes up too much memory. That gets back to the need for true random access.
So I don't think that's an issue... unless I'm misunderstanding what you're saying, in which case, please elucidate.
That'd be awesome.
I've had a couple crossover releases this past week and haven't had nearly as much time as I usually do, so some extra input would be fantastic -- and that goes triple for actual code!
Take a look at the X4 Savegame XML Analyzer I wrote last week.
It'll break down an unarchived savegame file into a flat tag/ownership/property/type list that you should be able to use as a base to write any automated scaffolding tools ya want.
If you need any more features, just let me know and I'll see what I can add in as time permits.