An optimization pass on the game would be an extremely welcome sight

[vvvvvvvv]

This implies that for each participant even slightest deviation from starting position, way less than a centimeter or a degree of angle in game units, can lead us to drastic difference in encounter results. In chaotic system that very well might be the case.

It can. X4 is a chaotic system. Depending on encounter composition, moving one ship by one nanometer can alter the encounter from defeat to victory. This is also why people just simulate it. An infinitely small value can result in a difference of ship dying vs not dying. It is not like fluid dynamics. It is like chess with without a grid.

Make a prototype. You've been discussing it for four months, it has been more than enough time to make several.

[Shadow_rainbow]

This implies that for each participant even slightest deviation from starting position, way less than a centimeter or a degree of angle in game units, can lead us to drastic difference in encounter results. In chaotic system that very well might be the case.

It can. X4 is a chaotic system. Depending on encounter composition, moving one ship by one nanometer can alter the encounter from defeat to victory. This is also why people just simulate it. An infinitely small value can result in a difference of ship dying vs not dying. It is not like fluid dynamics. It is like chess with without a grid.

It may seem chaotic when a lot of moving parts are involved. But, according to what I've sen so far, each part is predictable.
I may not have seen the entirety of X4 situations, but for example, a one on one small ship encounter generally consists of a head-on, short two circle fight with some extensions via short afterburner bursts, one participant winning and shooting the other's rear for a few seconds, after which the latter disengages using a long afterburner burst, then re-engages. This is rather simple fight plan -- and these are the most dynamic participants in the most free of situations available. Am I incorrect in this assessment? Because if I'm not, then even 180 degrees of difference in direction can play little to no role since both opponents can hit the speed limit before they merge for the first time. Only capital ships seem to be significantly affected by starting direction, when confronted by small and fast attackers.
But if you can propose such an encounter -- please do so. It would be interesting to analyze.

Make a prototype. You've been discussing it for four months, it has been more than enough time to make several.

I've been discussing it for around a day totals, perhaps. Even thinking of design document for the project (let alone two) would take much longer, at least for me.
Also, how do I even know what to include and test it on? You mention encounters where slight shift in position or angle can give one side an advantage, for the latest example. How would I even test the case of such encounter if I don't know what it entails?

And like I have mentioned before, I don't need a prototype to know it -- I have seen it in action myself. Sorry to ask, but have you played Space Rangers 1?

/EDIT clarification

[vvvvvvvv]

It may seem chaotic when

No. It is chaotic system by definition where a small numerical change can dramatically alter the result.

Make a prototype. You do need a prototype, because so far you've been discussing it in circles, theoretically. Discussing it further is pointless.

[Alan Phipps]

Perhaps more significantly, an abstract theoretical discussion will not persuade the devs to change or look into anything in-game. A demonstrable and credible empirical analysis arising from statistically-significant trials of situations and changed-modelling just *might*. I suspect this is why credible and reliable modders have had so much influence on the dev team in the past. (In my personal opinion.)

[Shadow_rainbow]

It may seem chaotic when

No. It is chaotic system by definition where a small numerical change can dramatically alter the result.

Demonstrate it, please. You have mentioned cases where even a tiny difference in starting conditions of encounter drastically alters its results. Describe them and explain why do you think that's the case to support your point. You've mentioned those to illustrate your point, right? Please do so.

Spoiler

Show

Make a prototype. You do need a prototype, because so far you've been discussing it in circles, theoretically. Discussing it further is pointless.

It's not like I am resetting everything to square one every time it starts to be going at least somewhere. Like, I get the feeling that when I ask for explanation for the thing mentioned -- its immediately becomes something new, with the old thing I asked about buried, forgotten and not talked about ever again. It's not circles, it's a star. That kind of discussion is unproductive, but there's little I can do about that, be it with or without prototype or anything else.

Besides, what prototype do you even mean? Storing paths and combat data and replaying it? It's not like Tacview (which is already here, although doesn't have X4 support built in) is anything new this day. What do you model with that and how do you discern your models are worthy without being able to do that in parallel in X4 engine, or even play your recording in game? Or do you propose I start opening some tracks in separate instances of Tacview and continue to do so until everything slows down to demonstrate that points on rails calculate faster than physics?

And I'd like to repeat my question about outlook. Space Rangers 1. Did you play it?

[Shadow_rainbow]

Perhaps more significantly, an abstract theoretical discussion will not persuade the devs to change or look into anything in-game. A demonstrable and credible empirical analysis arising from statistically-significant trials of situations and changed-modelling just *might*. I suspect this is why credible and reliable modders have had so much influence on the dev team in the past. (In my personal opinion.)

I'm not as arrogant as to believe someone will start changing stuff in the game on my behalf If any of developers peers into this topic one day and it gives him some new perspective on ideas that might just work -- that's already more than enough than I've been hoping for. More likely than not neither will happen, of course, they're both busy with their jobs on Diplomacy release and full of their own ideas. Hopefully it will run better than current version (I'm on Stable), although I'm hoping it at least won't get worse.
Nice to discuss it, nonetheless.

As for the prototyping thing that other member has mentioned, it's pretty pointless. To make it work I'll need my own space sim I have full access to, and it will only be correct in relation to that space sim. Of course, we could probably generate data tracks for Tacview in X4, but it wouldn't really prove much in any regard since we can't run specific X4 simulation scenarios on repeat to see how that turns out and compare the results. Can we?

[vvvvvvvv]

Demonstrate it, please.

You're hit by an Asgard beam in a destroyer. The beam grazes you, you die. Asgard wins.
You're moved to the left by 1 nanometer. The beam now misses. You take out Asgard's engine and kill it because you have paranid plasma that outranges its turrets. You win.

There is a LOT of stuff like this. It is not fluid dynamics, it is chess withot grid, where a near miss can result in a tide-turning event. And cascading events like that are properties of chaotic systems.

---

One more thing.

Sharing information with others and explaining the obvious is a common courtesy. You've devoured inordinate amount of attention and do not appear to engage in the discussion. As in you ignore what people say.

So that'll be the end of this chat, and that was the last explanation I'll give.

Now, go and make that prototype. You had enough time to make a dozen.

[Shadow_rainbow]

Demonstrate it, please.

You're hit by an Asgard beam in a destroyer.

Remember that we're modelling a number of encounters and are discussing how many starting points there can be in those (while trying to minimize their number, of course). An encounter should start beyond the range of either opponent's weaponry. As such, Asgard can target, but cannot fire at its opponent. By the time it can, the opponent can move along its trajectory far beyond the starting point (if it can't move, arrival time to shoot will depend on Asgard's movement alone, making 22.5° derivation irrelevant at that distance).

near miss can result in a tide-turning event.

Events such as this (especially if it was a hit) are recorded in the event history of the encounter. So if someone grazes someone while trajectory not being accurate enough to demonstrate a shot hit, the hit will still register.
On the other hand, actively using foresight in such a model is a good thing, so if we're looking at a hit while target's boundaries are away from such an event, we should preemptively alter the trajectory in such a way as to demonstrate a hit. Or alter the projectile/beam not to continue after reaching target's space, playing hit effect instead.

Also, it raises the question of stray shots in the time between pre-simulated fragments. Like, when the pre-simulation fragment's central ship is destroyed with its projectiles still in flight, and attackers are rearranged to other parts with some of their projectiles also mid flight, what do we do with all of these? The most obvious solution would be just assume all these miss or fuzzed out -- since total most are going to. But we could probably write some transition state to the encounter switching. And even we can safely ignore that for OOS, IS would probably benefit a lot from having such a state (otherwise it'll look like sudden burst of participants to new points at engagement range of their new target).

See? Despite you providing an argument that's incorrect by definition of what we're doing, we still have added to the model we discuss -- not by prototyping, but by simply discussing it. Namely, adding user stories to extend our problem domain to make it more fitting to the thing we're trying to describe with it.

you ignore what people say.

I have literally explained just now how scenario you proposed in your previous post would (not, but still) fit into my proposal. If I haven't answered some of your points, please refer me to them, which would be much more productive than trying to accuse me of ignoring arguments.

Speaking of which, you still haven't answered my questions about using Ship ID. Please do so, even if it may not help us in pre-calculating the battle. Or it may, who knows right now?

Now, go and make that prototype.

I think I have provided enough arguments on why I think doing that would be pointless at the moment.
Of course, if Egosoft added scenario editor and data exporter (at least raw data, although preferably with ability to choose Tacview export directly), that would at least receive that point since we could run test scenarios and see differences and similarities both. Like, some scenarios probably don't even need the pre-calculation record, just a timer to tick the hit points down.

[vvvvvvvv]

No, I'm done here. You already got every argument in existence. Enough is enough.

[chew-ie]

A raw data exporter is there - savegames contain everything.

[Shadow_rainbow]

A raw data exporter is there - savegames contain everything.

How do you imagine receiving flight path and event data through save files? Taking a save each time ship changes trajectory, fires a gun or takes a hit? There's no ingame mechanism to do that in automatic way. That's besides the fact that saving takes a long time, so doing this through save files would take forever.
Besides, if we want to see how a single encounter may evolve through many attempts, we'd need some way to repeat it -- in isolation from other stuff around it.
Also, having custom encounters would help a lot with performance metrics a lot. We would be able to see exactly how many ships/turrets/stations does it take to make performance degrade, and in what way. That's besides the fun part of creating and participating in custom scenarios or even generating ones. For example, like done in DCS, where you can create a large, complex and relatively variable scenario to fly quickly with just a few clicks.

[chew-ie]

You get it all wrong. (deliberately?) Take a save game, there is all the data you need. Write your prototype, show comparisons. Get to work. Stop talking about it [until you have something of substance to show].

Same as vvvvvvvv, I'm out.

[Shadow_rainbow]

Take a save game, there is all the data you need.

Wow, 800+MB XML files to parse! Notepad++ has trouble just trying to scroll that volume of text any notable distance (like, it seems to hang up for a couple of minutes). I wonder why the game saves and loads saves this slow.
As I look at it, I see some very strange things. Like, subcomponent coordinate offset instead of slot number, seemingly just eating up valuable space (we can't offset subcomponents, only pick a slot for them), or skill level list naming skills (like those would be saved in different positions each time, so <skills boarding="3" engineering="11" management="4" morale="8" piloting="2"/> written as <skills>3 11 4 8 2</skills> would load differently). I seem to start to understand where u/CBJ's short talk about multiplication magnitude orders comes from.
Anyhow, what I see are parameters like

Spoiler

Show

Code: Select all

<movement forceposition="0" class="navigation">
<position>
<read space="[0x241c5169]" numpathpoints="1" avoidbigobjects="1" avoidsmallobjects="1">
<offset>
<position x="39649.023" y="-934.793" z="-4356.271"/>
<rotation yaw="-101.94152" pitch="0.416885"/>
</offset>
<velocity>
<linear x="-413.099" y="3.07225" z="-87.3663"/>
</velocity>
<acceleration/>
</read>
<write space="[0x241c5169]" numpathpoints="1" avoidbigobjects="1" avoidsmallobjects="1">
<offset>
<position x="39649.023" y="-934.793" z="-4356.271"/>
<rotation yaw="-101.94152" pitch="0.416885"/>
</offset>
<velocity>
<linear x="-413.099" y="3.07225" z="-87.3663"/>
</velocity>
<acceleration/>
</write>
</position>
<targetpoints>
<targetpoint refobject="[0xe0203b4]" behavior="default" direction="forward" avoidbigobjects="1" avoidsmallobjects="1" resetroll="1">
<offset>
<position x="121.062" y="407.716" z="-9970.672"/>
<rotation yaw="-102.00718"/>
</offset>
</targetpoint>
</targetpoints>
<flightcontrolmodel type="linear"/>

I'm unsure what read and write space clauses do, but I assume that is the ship's position, rotation and speed, but it (of course) does not include either previous or next points on the trajectory. Without which you cannot plot its trajectory. So no, it does not and is not intended to have data "I" need (since "I" need continuous data stream for building trajectory externally).

Write your prototype, show comparisons. Get to work. Stop talking about it [until you have something of substance to show].

If you're going to demand something of someone in such a blatant and arrogant way, do so with someone who at least took your money. Demand of Egosoft to create scenario editor to run scenarios (including ability to run the same test ones with only AI pilots, starting in different relative positions up to collision range) in. Demand of Egosoft to create continuous data export (like Dovetail, Eagle Dynamics or 777 did). Although with that attitude I'd wager on you getting kicked out with your money back, instead.
On my side, I would just humbly inform Egosoft that these features would benefit everyone involved, not just someone who'd like to do some experimenting or analysis. But that would also enable such endeavors, if anyone would be up to it.

[Y-llian]

Respectfully, I think the reason folks are stopping engagement is that you’ve not demonstrated any viability for your idea. You are proposing to remove a core component of the game many of us enjoy. Indeed, an aspect that has seen development and refinement throughout the X-series where Ego has listened (and responded) to their customers.

Since you believe your statistical methodology will work then it’s up to you to prove your concept. Hence, the understandable request for a prototype. Alienating the technical folks who has given their time and energy to respond will not help your cause.

Not meaning to be unkind here.

Take care.

Best
Y.

[jlehtone]

As I look at it, I see some very strange things. Like, subcomponent coordinate offset instead of slot number, seemingly just eating up valuable space (we can't offset subcomponents, only pick a slot for them), or skill level list naming skills (like those would be saved in different positions each time, so <skills boarding="3" engineering="11" management="4" morale="8" piloting="2"/> written as <skills>3 11 4 8 2</skills> would load differently).

The used format has very good rationale, but that is not important for the discussion here.

I assume that is the ship's position, rotation and speed, but it (of course) does not include either previous or next points on the trajectory. Without which you cannot plot its trajectory.

True, no trajectories. The entire history of your playthrough, of all the trajectories in the save would be something big. (I've seen dynamic simulation trajectories of molecules that needed a 64 GB machine to show with.)

Yet, when a save loads, the game can generate the next "point in trajectory" -- for the next "timepoint" -- from the data that you did find. That depends on the logic that is in the game, not in the save.

[Shadow_rainbow]

Respectfully, I think the reason folks are stopping engagement is that you’ve not demonstrated any viability for your idea. You are proposing to remove a core component of the game many of us enjoy. Indeed, an aspect that has seen development and refinement throughout the X-series where Ego has listened (and responded) to their customers.

I'd like to reiterate that the main aspect that some of the most active participants have protested against (which they also called 'removing the game's beating heart') is not even the main proposal here. First and foremost I propose pre-calculating ship-on-ship and station-on-ships engagements to compose both accurate and believable picture outside of player's direct surroundings (player-target-attackers) without overloading computer with calculus that will be repeated in the same or very similar way many times over and is better done (and 'baked') once in all its applicable multitude. At least as I see it today.
Effectively, we'd be going through several event recordings at once as battle fragments, with the battle being reconstructed by moving ships as points on trajectories, shots being fired at targets they've already hit during the pre-simulation and around (if they didn't) and ships damaged or destroyed by events in the event log. You don't really need 'THE prototype" to know reading some tables, placing some points in space and registering some events would take significantly less math than calculate everything involved in direct way would (since the latter is all of the aforementioned plus math). For those who don't believe it, I even mentioned an example of software that already does this sort of reconstruction (Tacview, namely).
Optimizing the universe is just an extension of the method.

And as of now, I don't even see any notable method of constructing such a prototype. There's no viable way to extract the trajectory and event data required from X4, and seemingly no way to even arrange for encounters we would try to create models for. So there's nothing to compare and nothing to compare it to. Unless, of course, they were meaning creating some curves in some engine like Blender with points moving on them, then demonstrating it runs incomparably faster than X4's simulation with more points taking less computing power (which is obvious, so I would take that for straight up trolling). Besides, that would have little to do with model accuracy (the thing we're striving for here?) since it would not use actual X4's systems like physics, AI, etc into consideration.

As I look at it, I see some very strange things. Like, subcomponent coordinate offset instead of slot number, seemingly just eating up valuable space (we can't offset subcomponents, only pick a slot for them), or skill level list naming skills (like those would be saved in different positions each time, so <skills boarding="3" engineering="11" management="4" morale="8" piloting="2"/> written as <skills>3 11 4 8 2</skills> would load differently).

The used format has very good rationale, but that is not important for the discussion here.

Well, it is very decent at being readable at a glance with minimum documentation. The problem is its sheer size. I'd honestly prefer abbreviated and heavily encoded format with fixed symbol placements, paired with specific editor to view it -- but fraction of that size. It also is stored compressed in .gz format which, as I recall, does not support either content indexing or multithreading.
It's relevant to this thread BTW since I've also mentioned save-load times in opening post

I assume that is the ship's position, rotation and speed, but it (of course) does not include either previous or next points on the trajectory. Without which you cannot plot its trajectory.

True, no trajectories. The entire history of your playthrough, of all the trajectories in the save would be something big. (I've seen dynamic simulation trajectories of molecules that needed a 64 GB machine to show with.)

Yet, when a save loads, the game can generate the next "point in trajectory" -- for the next "timepoint" -- from the data that you did find. That depends on the logic that is in the game, not in the save.

Yup, big and unnecessary. We need trajectories and events for very specific, niche purpose that is not required anywhere else in-game (unless Egosoft releases their version of battle history player). Also, we need rather short pieces of ship's lifetime trajectory, encompassing only a single, characteristic encounter we'd try to model. That's why data stream export (along with encounter editor) would help us so much. I don't really see any other way around that.
As for ingame logic predicting the next trajectory point, I doubt that, since no mechanism of that kind was mentioned anywhere I know -- more likely, it just calculates it.

[CBJ]

...gz format which, as I recall, does not support either content indexing or multithreading.
It's relevant to this thread BTW since I've also mentioned save-load times in opening post

Just to point out to anyone still reading this thread, this is yet another factually incorrect premise. Both loading and saving are multi-threaded (including the compression/decompression) and, as has been explained many times in the past, are not performance-limited by the file format.

[jlehtone]

As for ingame logic predicting the next trajectory point, I doubt that, since no mechanism of that kind was mentioned anywhere I know -- more likely, it just calculates it.

Prediction is a calculation. Neither computers nor people are psychics.

[Shadow_rainbow]

...gz format which, as I recall, does not support either content indexing or multithreading.
It's relevant to this thread BTW since I've also mentioned save-load times in opening post

Just to point out to anyone still reading this thread, this is yet another factually incorrect premise. Both loading and saving are multi-threaded (including the compression/decompression) and, as has been explained many times in the past, are not performance-limited by the file format.

You can freely address just me, anyone interested will read this anyway
I'm glad to be incorrect in this matter. Perhaps Egosoft uses something like pigz to do the job instead of gzip (which, apparently, has recently received multithreading on packing, but not on unpacking). But the sad fact is loading my current save takes 1 minute 42 seconds, give or take (like, give a few seconds more, closer to 2 minutes when done in combat). That is appalling when trying to save-scum myself out of some situations (many of which should be avoidable entirely IMO, but that's another question). Saving takes around 20 seconds -- on SSD. So it's not fast, either.

Perhaps since you're in the mood anyway, you can even share wisdom on why is the save file not split between multiple files storing notable chunks of data, like general info, faction data, per-sector storage? If readability was the point, what kind of xml viewer is even designed to handle such file sizes? Like, just now I have attempted to collapse 'factions' element to see Notepad++ visibly hung up with no expected recovery -- and that's just 1714 lines at the start. Out of 11 million 823 thousand and something!

As for ingame logic predicting the next trajectory point, I doubt that, since no mechanism of that kind was mentioned anywhere I know -- more likely, it just calculates it.

Prediction is a calculation. Neither computers nor people are psychics.

Well, the game engine should point stuff (or information about it) somewhere, so of course it's still calculus. What I have meant by 'prediction' is simpler calculus, aimed at reducing the calculations required at present time.

[alt3rn1ty]

Wow, 800+MB XML files to parse! Notepad++ has trouble just trying to scroll that volume of text any notable distance (like, it seems to hang up for a couple of minutes).

https://www.sublimetext.com/
Much better at handling large files than Notepad++