An optimization pass on the game would be an extremely welcome sight

[Shadow_rainbow]

Hello!
I've had a lot of fun with Timelines scenarios. Still, Graph 6 - 1 ('Frontline dealings') strains my system to slide-show levels of performance.
Upon entering Open Universe, I've also noticed that saving and loading takes a lot of time, as well as simulation going slide-show in some battles.
When I looked for it, I found threads like this one all over the internets. So, to my understanding, the problem is the engine attempting to feed the system a load it cannot digest properly -- and I'm far from being the only one to have it.
Yet in the days of yore in MS CFS 3 I've seen a whole of Europe in a humongous detailed frontline. When this line moved, it did cause a slowdown (it was a time of single-thread) and the move was incremental, but otherwise it was simulated in real time with some simple in-depth production, logistics, artillery strikes, attacks, multiple strike/intercept flights in the air, etc.

And if my X-Universe was stored and simulated in statistically interacting chunks that were only unfolded when I personally observed them, I totally wouldn't mind. There's no real reason for me to have a need to know that some unseen transport carrying some specific goods to some factory in a far away sector I can't observe did make it on time, was late or intercepted by a pirate. If, on sector observation, it was generated from faction sector ship population (or even outright out of sector's economic 'health') in a random position in one of pre-computed situations, I wouldn't even know a difference.
Also, if that situation was amidst a battle of several thousands of ships along hundreds of stations, I would totally understand if that battle was also consisting of chunks that evolved through their pre-computed statistical paths, only displaying approximate results unless I am close enough to directly observe and am able to see the substitution. From 10km away I won't be even able to tell if that's 16 stacked -spheres- hex boxes that dish out health damage against each other Panzer General style, with groups of large ships following line paths and smaller ships blobbing in randomly selected precomputed elliptical paths, displaying some cannon and missile effects around and losing ships when statistics says so.
Even if I'm close to a station, knowing where each of its guns is pointed is an extreme excess of information. Having a field of random shrapnel hitting random targets in specific sector when I'm not looking in that direction doesn't make much difference to computing each gun's trajectory intersections and terminal ballistics, simulation wise. Of course, each station has a build to it, but only player-built stations can be truly random -- and even those can be pre/re-computed with defence module visibility on module completion or destruction.

In fact, the main things I care about the most (since I observe those directly) are my ship's status, my weapons' fire, my target and, in many cases, my attackers. That's what really has no alternative to simulating in an exact way. All the rest can be purely statistical and/or be simulated with different delays -- like, refreshing sector economy state isn't really required every minute unless I watch it, specifically. And, this I stress, with no loss of simulation quality -- with proper statistic sets it would even improve upon it by excluding different quirks that can mess the simulation part up.

Also, when I quick-load my save several minutes after saving when something goes horribly wrong, it's not like I get into some other X-Universe that's completely different from the one I was in. On the contrary, usually, the Universe is largely the same, yet it seems to load everything from the ground up. This, again, creates much longer load times than could be if only the differences could be rolled back.

Speaking of pre... in an attempt to save as much FPS as possible in those large battles, I have turned off the lighting. But the fact that there doesn't seem to be pre-computed lighting was an unpleasant surprise for me. It's not like ship or station interiors can be redecorated with new lights or bulkheads, so storing it all in textures would make a lot of sense (and retained a lot of its beauty) to me.

Being like this, I honestly think the game would have become much more scalable, much less reliant on raw computing power and would be so much closer to both what we see in promo videos and what we 'see' inside our mind's eye.
Thanks for reading a wall of text. Hope it helps someone, someday.

[TheDeliveryMan]

Hello!
And if my X-Universe was stored and simulated in statistically interacting chunks that were only unfolded when I personally observed them, I totally wouldn't mind. There's no real reason for me to have a need to know that some unseen transport carrying some specific goods to some factory in a far away sector I can't observe did make it on time, was late or intercepted by a pirate. If, on sector observation, it was generated from faction sector ship population (or even outright out of sector's economic 'health') in a random position in one of pre-computed situations, I wouldn't even know a difference.

In X4 every ship or station exists from the time it is constructed until the time it is destroyed, that's a core feature "a living, breathing universe". Those ships and stations interact with each other all the time. The universe is so heavily interlinked that each sector has to be up to date, no matter if you're looking, or not.

[alt3rn1ty]

In addition to what deliveryman said, you can also help yourself by not having too many ships/fleets/stations/mega stations.

Egosoft are constantly refining performance of the game, right now 7.60 betas are specifically targetting memory management and stability ..
viewforum.php?f=192&sid=13df55a5e9e44ff ... 256b5df7a0

In the Tech support side of the forum, Imperial Good has provided a sticky topic with all the known hints/tips for a better game experience ..
viewtopic.php?t=420952

Worst case scenario is when landed on a large station, with the map open, and a large battle is going on nearby, even members of the community with the best desktop specifications get fps hits in that circumstance. VIG sectors with large swarms of fighters are another bad hit to fps

Its a complex game that is still evolving, we hope, as Egosoft usually do, that the game will be refined even further by the end of its development.
But what you suggest is unlikely as Egosoft have great pride in the game being a living breathing universe/economy.

Do you have a good machine specification? Good single core speed and memory speed are the main factors
Are you using mods?

[vvvvvvvv]

The game is being optimized, which was demonstrated during the beta where they boosted performance. So passes are being performed, although there are certainly still many pain points.

"Everything is being simulated for real" is a core feature of the game.

[jlehtone]

"Everything is being simulated for real" is a core feature of the game.

This. We can be everything everywhere all at once. There is no spot we want approximated.

Why would we tune a ship or station to perfection if it would reduce to a fraction of average the moment we look elsewhere?


@Shadow_rainbow:
You were not driving around with bicycle in Panzer General, at risk of being squashed by enemy tanks, were you?

The game has been optimized. Multiple passes and ongoing. However, and this has been repeated every time a thread like this starts, there are hard facts on what can and cannot be done. Yes, there are many previous threads on the topic and many players, who know more or less about "threads, parallelization, and optimization", but none of us knows whether any of that is feasible in X4. What we do know is that the devs know.

[Shadow_rainbow]

In X4 every ship or station exists from the time it is constructed until the time it is destroyed, that's a core feature "a living, breathing universe".

A living breathing universe can certainly exist without constantly calculating every last detail of it. Like I mentioned in OP, dissolving things into statistics and reborning those from it when necessary should be more precise while still being exponentially less compute-intensive. I've already mentioned living world of MS CFS 3 that didn't feel more artificial than its maker made it to be, but I can also mention living breathing world of Falcon 4, still alive today, that also feels alive with its complex world mechanics. It does it in a more of a blunt way (player bubble), but it also looks and feels good.
Worse, deciding to simulate everything is certain to limit the simulation. There are objective limits to computing, especially when we talk about gaming PCs. Again, a pair of examples -- in IL2BoS, each airplane (including AI) has flight model based on wing element theory, while in DCS only player's aircraft has that while AI physics is using a simplified model. Both can do a lot in terms of optimization, but the former is constrained by computing power of gaming rigs much more than the latter (and even worse, it shows). Again, it devolves into that wing element theory feels better and gives an airframe another layer of 'living' quirks, but 'feel' only applies to human feeling what they can observe. And if done properly, simplified physics model gives results better aligned to the prototypes it's based upon.
What's even worse, trying to scale the 'true' model and hitting these computational limitations, developer has to make a choice between not scaling, raising required specs or simplifying something in the model. Or leaving it to chance. Each of these ways is bad.

So I do have some solid reasons to reiterate that living, breathing worlds are not suffering from not being calculated in every detail possible. If anything, they become more precise in their operation while (when done properly which I must stress here) not providing player with clues to this swapping.
For example, a large battle is underway. You are looking from afar at a group of L and XL ships, exchanging fire while fighters and corvettes do their thing around. What do you care about inclination of anti-fighter turrets on each of those if all you can observe is a small silhouette going pop from time to time among gun special effects? What purpose serves calculating each cannon hit if it can be pre-computed how this battle will evolve and how health of each ship involved decreases? If you probe a ship to determine its current health, what do you care if it snaps to percentage pre-computed prior depending on length of that engagement? How can you feel momentum of each fighter compared to them moving in pre-calculated trajectories? These are things outside of your vision and can be calculated in an exact statistical way without quirks of physics simulation (or with them, should someone decide to include that into statistical model).
And still the battle would be as real as if it was simulated (well, different parts of it were simulated at least once, when statistics was created). Only it doesn't load your calculation with stuff like physics, bullet trajectories/intersection, AI and the like. It can be decided at the very start of the engagement how will it end, when each ship attempts to pull off and if that will be successful -- and you won't know any difference from case where it all would be calculated in real time. It would only change if you flew in real close to be actually hit by some of these bullets and engaged by some of these ships -- and again, with enough complexity it could be swapped in only as required.

Worst case scenario is when landed on a large station, with the map open, and a large battle is going on nearby, even members of the community with the best desktop specifications get fps hits in that circumstance. VIG sectors with large swarms of fighters are another bad hit to fps

Its a complex game that is still evolving, we hope, as Egosoft usually do, that the game will be refined even further by the end of its development.
But what you suggest is unlikely as Egosoft have great pride in the game being a living breathing universe/economy.

Do you have a good machine specification? Good single core speed and memory speed are the main factors
Are you using mods?

Thank you for the links, although I've visited them prior. Of course I'm always hoping for new updates to bring performance improvements.
What you have described is exactly the hit into physical limits. I have tried to thoroughly explain above how that is really not necessary despite world being alive.
As for Egosoft's pride, I'm certain they would enjoy even more pride of living breathing and scalable Universe with economy. Of course, that would require much work and rework, but I'm totally sure in the end it would be worth it.
And yes, I have a space-age PC of 4 cores capable of executing 3.4 billion basic operations per second in parallel, with 32 GB RAM exceeding 640KB (which, as we all know it, are enough for anyone) 50 thousand times. It won't get much faster than that anytime these next two decades.

"Everything is being simulated for real" is a core feature of the game.

This. We can be everything everywhere all at once. There is no spot we want approximated.

Why would we tune a ship or station to perfection if it would reduce to a fraction of average the moment we look elsewhere?


@Shadow_rainbow:
You were not driving around with bicycle in Panzer General, at risk of being squashed by enemy tanks, were you?

The game has been optimized. Multiple passes and ongoing. However, and this has been repeated every time a thread like this starts, there are hard facts on what can and cannot be done. Yes, there are many previous threads on the topic and many players, who know more or less about "threads, parallelization, and optimization", but none of us knows whether any of that is feasible in X4. What we do know is that the devs know.

Driving around with bicycle in PG would require bicycle ride simulation, true. It would also be required to simulate other stuff going around. But simulating pressure difference in each of cylinders in each of engines of bombers on their run to target and back (which is a real thing in some games) would be a horrific excess if you're not driving your bicycle through that exact bomber (and even then, it would mostly affect sound which means pre-computing is better). And if you don't see those bombers or their bombs, simulating them in any physical way would be exactly the same excess. If these bombs hit somewhere in your hearing range, you wouldn't be able to differentiate the case of 'X bombs fell with Y deviation into Z area with Phi sound refraction and reverberation qualities' from 'this variation of this airstrike sounds like this one sound file'.
Also, those bombers taking damage and losses from FLAK would be more correct through pre-calculated statistics than any other physical calculation way with their limitations and quirks. And only if you drove your bicycle to their airfield would require existence of them as objects of different characteristics (such as different damage states). In the same time, infantry position you left behind would not require to be consisting of physical objects.

That's the point. We cannot be everything everywhere all at once. Even if we stick all the sectors with adv. satellites we'd have to watch total most of it through strategy screen, and zooming in to improve resolution of observation would require losing sight of stuff around of viewport (and its immediate surroundings).

Again, you can simulate everything in the world -- but that would require compromises (which reduce simulation accuracy), severely limit object count and require uncomparably high computing powers still. While providing no better simulation of living world.

[jlehtone]

Driving around with bicycle in PG would require bicycle ride simulation, true.

In fact, the main things I care about the most (since I observe those directly) are my ship's status, my weapons' fire, my target and, in many cases, my attackers.

And that is the bicycle in X4.

refreshing sector economy state isn't really required every minute unless I watch it, specifically.

The things we do not watch, the "low attention" ("OOS" in previous game), are modeled with less accuracy than the things we see.

when I quick-load my save several minutes after saving
...
[has] much longer load times than could be if only the differences could be rolled back.

How much memory and effort do you expect that it would take to "keep the differences" along?

[Nanook]

...
For example, a large battle is underway. You are looking from afar at a group of L and XL ships, exchanging fire while fighters and corvettes do their thing around. What do you care about inclination of anti-fighter turrets on each of those if all you can observe is a small silhouette going pop from time to time among gun special effects? What purpose serves calculating each cannon hit if it can be pre-computed how this battle will evolve and how health of each ship involved decreases? If you probe a ship to determine its current health, what do you care if it snaps to percentage pre-computed prior depending on length of that engagement? How can you feel momentum of each fighter compared to them moving in pre-calculated trajectories? These are things outside of your vision and can be calculated in an exact statistical way without quirks of physics simulation (or with them, should someone decide to include that into statistical model).
And still the battle would be as real as if it was simulated (well, different parts of it were simulated at least once, when statistics was created). Only it doesn't load your calculation with stuff like physics, bullet trajectories/intersection, AI and the like. It can be decided at the very start of the engagement how will it end, when each ship attempts to pull off and if that will be successful -- and you won't know any difference from case where it all would be calculated in real time. It would only change if you flew in real close to be actually hit by some of these bullets and engaged by some of these ships -- and again, with enough complexity it could be swapped in only as required. ...

Despite your walls of text, I'm getting the distinct impression you haven't really put a lot of time into the open universe. What you describe above actually takes place in the game. It's called 'OOS' (Out of Sector), 'LA' (low attention) and 'HA' (high attention). OOS is pretty self explanatory, LA is when you're in the same sector but farther away from the action, and HA is when you're right in the mix of a battle (or whatever else is going on). There are different levels of calculations going on for each, with HA having the most detailed since the player can see what's happening and statistics aren't really appropriate.

However, game pieces persist across the whole universe, and their tasks and progress do need to be calculated individually, or the universe wouldn't work the way it does. For example, I send a trader off to a distant station in a distant sector. I want to be able to monitor its progress to make sure it doesn't run into any bad guys, so I need to be able to track it across the whole trip, not have it devolve into some statistical calculation dependent on the unholy RNG.

And before you get your undies in a bunch, I'm not an Egosoft employee. Like all the other moderators on these forums, I'm an unpaid volunteer, and just a gamer the same as everyone else.

[Shadow_rainbow]

Driving around with bicycle in PG would require bicycle ride simulation, true.

In fact, the main things I care about the most (since I observe those directly) are my ship's status, my weapons' fire, my target and, in many cases, my attackers.

And that is the bicycle in X4.

Sadly, it is not even close to that, as evident by explanations, including one linked in the OP.

refreshing sector economy state isn't really required every minute unless I watch it, specifically.

The things we do not watch, the "low attention" ("OOS" in previous game), are modeled with less accuracy than the things we see.

And if these were modelled in a statistical way, there would be no need for reduced accuracy to save resources. That's my point. One of, anyway.

when I quick-load my save several minutes after saving
...
[has] much longer load times than could be if only the differences could be rolled back.

How much memory and effort do you expect that it would take to "keep the differences" along?

We're not keeping the differences along, we're checking world states and computing the difference on load. Thus, no extra memory or computing.

[Shadow_rainbow]

...
For example, a large battle is underway.

Spoiler

Show

You are looking from afar at a group of L and XL ships, exchanging fire while fighters and corvettes do their thing around. What do you care about inclination of anti-fighter turrets on each of those if all you can observe is a small silhouette going pop from time to time among gun special effects? What purpose serves calculating each cannon hit if it can be pre-computed how this battle will evolve and how health of each ship involved decreases? If you probe a ship to determine its current health, what do you care if it snaps to percentage pre-computed prior depending on length of that engagement? How can you feel momentum of each fighter compared to them moving in pre-calculated trajectories? These are things outside of your vision and can be calculated in an exact statistical way without quirks of physics simulation (or with them, should someone decide to include that into statistical model).
And still the battle would be as real as if it was simulated (well, different parts of it were simulated at least once, when statistics was created). Only it doesn't load your calculation with stuff like physics, bullet trajectories/intersection, AI and the like. It can be decided at the very start of the engagement how will it end, when each ship attempts to pull off and if that will be successful -- and you won't know any difference from case where it all would be calculated in real time. It would only change if you flew in real close to be actually hit by some of these bullets and engaged by some of these ships -- and again, with enough complexity it could be swapped in only as required. ...

Despite your walls of text, I'm getting the distinct impression you haven't really put a lot of time into the open universe. What you describe above actually takes place in the game. It's called 'OOS' (Out of Sector), 'LA' (low attention) and 'HA' (high attention). OOS is pretty self explanatory, LA is when you're in the same sector but farther away from the action, and HA is when you're right in the mix of a battle (or whatever else is going on). There are different levels of calculations going on for each, with HA having the most detailed since the player can see what's happening and statistics aren't really appropriate.

What you describe (and what is evidently implemented) is reduced modelling complexity to save resources. Exactly one of approaches I have described above. It is apparently different from what I have described -- relevant stuff is still being calculated, just in a less precise way. This approach isn't new, but you can reduce complexity within the same model only so much before it stops working. And it shows.
I may not have 'put a lot of time into the open universe' as many here, but I do have experience in it, including glaring problems described in OP. The game clearly overcalculates, by a huge lot. What I have described are just some thoughts on how can it be done better. My experience with other projects, including similarly-scaled ones, provides me -- and, by extension of this forum, all of us -- with perspective on how can it be.

However, game pieces persist across the whole universe, and their tasks and progress do need to be calculated individually, or the universe wouldn't work the way it does. For example, I send a trader off to a distant station in a distant sector. I want to be able to monitor its progress to make sure it doesn't run into any bad guys, so I need to be able to track it across the whole trip, not have it devolve into some statistical calculation dependent on the unholy RNG.

If a piece is important, it can be persisting and I have nothing to protest about. It's the matter of assigning enough importance (and computing power) to it that matters. Like in scenario you have described -- if you have personally sent a ship to complete a specific order, it clearly is important enough to track individually. But do you track each of the pirate ships individually, as well? Since some pirates mask themselves, we're looking at strict 'no' for an answer. So their appearance is truly random to you even in current wholly computed model despite it not being statistical one. So the question is -- can you discern pirate ship attack in such scenario from a pirate ship(s) materializing randomly out of, say, 'Sector security level', ''SCA sector presence level', 'SCA ships available', 'SCA-present sector connections controlled' and a few other factors on randomly selected intercept scenario? I think it would be near impossible to tell apart even with finest scrutiny. And you wouldn't pay for it in computing power and load times.
And we're talking clearly high-ranked actor here. If your mining ship is automining or trading ship is autotrading when you're not looking, does it deserve the same resources when you're not monitoring it? I think not quite. And some random miner automining for some random station -- even less so. Just have some dozens of scenarios pre-calculated based on simulation and select one of the most appropriate ones if you look that way (with a satellite or in-sector presence). Otherwise, it's just a stat point with predetermined fate like another cog in the machine.
The X-Universe would still work the same, be it pre-calculated or calculated in real time. But imagine if the first one could be run on 20-year old PC and yet still be larger than what we have now -- opening new possibilities for world building and complexity increase. Like, mega fleet engagements? Planets or mega stations (not in the current definition but, say, a sector-sized station)? Also, I wouldn't mind a lot of things like wingman non-basic orders, ship package coordination and so much more that would be required in large scale battles. There's always a way to sink computing resources and just so much of these resources to squander these on determining some non-relevant things that may never even become relevant.

[vvvvvvvv]

The X-Universe would still work the same, be it pre-calculated or calculated in real time.

Nope, it wouldn't. Statistical average does not account for small changes resulting in cascading effects. This was also done in Elite, and no it doesn't feel the same. In Elite you very quickly become aware that NPC ships you encounter are fake, and at any point no more than 40 ships exists along with you within the entire galaxy. Extra fun when you relize that those 40 ships perform worse than entire galaxy in X4.

You're arguing against major and central core feature of the game, one that has the highest importance and is the game's main selling point. If the path developers has chosen is not up to you, the best idea would be to try something else. No offense intended. And yes, the game tracks individual pirates. For example, there are sectors with a valuable ship the pirates may get it first. The game also allows you to cover entire map with satellites and see activity in the entire world at once.

Regarding 20 years old PC, the best idea would be to put that poor machine to rest, and I do not recall a game that has X4 visuals and scale coming out in 2005. Aside from X3 reunion which had a lot of X4 features missing.

Additionally you're also free to develop your own game according to this vision. Asking to remove the primary feature of the game does not seem to be a very reasonable request.

[Shadow_rainbow]

The X-Universe would still work the same, be it pre-calculated or calculated in real time.

Nope, it wouldn't. Statistical average does not account for small changes resulting in cascading effects. This was also done in Elite, and no it doesn't feel the same. In Elite you very quickly become aware that NPC ships you encounter are fake, and at any point no more than 40 ships exists along with you within the entire galaxy. Extra fun when you relize that those 40 ships perform worse than entire galaxy in X4.

If it would (or wouldn't) work the same strictly depends on how statistics would be implemented. Statistical approach is not about averaging complex things out -- on the contrary, it's about taking everything found possible into account. Otherwise it would be 'averaged approach', no?
I didn't play Elite, but I know of its development and its severe hardware limitations. 22 kilobytes probably isn't large enough to fit large pre-computed statistics scenario data for combat, economy and other aspects of sector life, true -- I'd wonder if that was the reason of the feeling discrepancy you described. It's good we have a bit more memory nowadays, as well as computing power on par.

You're arguing against major and central core feature of the game, one that has the highest importance and is the game's main selling point. If the path developers has chosen is not up to you, the best idea would be to try something else. No offense intended. And yes, the game tracks individual pirates. For example, there are sectors with a valuable ship the pirates may get it first. The game also allows you to cover entire map with satellites and see activity in the entire world at once.

Major and central core feature is the game's living world. And I like that. It's gross overcalculation that hits really hard on experience -- and is bound to happen at some point, no matter what rig you have.
Note that having every detail calculated isn't an option even now, like ships and stations outside the sector you're in don't have physics, or civilian traffic on station doesn't exist outside of player proximity. That doesn't deter many, right? So why not come further? Let them live like they have physics and all that -- just not as an object in game, unless required? Even in the same sector with player?
Also, I was not talking about the game not tracking pirates (it obviously does now). I was referring to Nanook being informed of those. If he can't track them, they are random for him.
And, in case of stat model, if there is satellite coverage around the entry gates, that would clearly force those pirates to manifest as ships. If not intercepted inside a proper area, they would dissolve into pirate presence level to interdict trade vessels or to be interdicted by patrols. Again, manifesting if player does observe the encounter, or being calculated without that if he doesn't. And on the plus side they would have their physics calculated through that encounter -- just not in real time when you play the game, but prior, by developers when preparing statistical scenarios.
Increasing satellite numbers could indeed require more resources, but I think there are ways to combat resource drain even in that case. And there would be a way to reduce the load just by shutting these satellites down. Also, the main performance hit is from wide scale in-sector combat -- at least as far as I have experienced so far.

Regarding 20 years old PC, the best idea would be to put that poor machine to rest, and I do not recall a game that has X4 visuals and scale coming out in 2005. Aside from X3 reunion which had a lot of X4 features missing.

Additionally you're also free to develop your own game according to this vision. Asking to remove the primary feature of the game does not seem to be a very reasonable request.

Well, X-series predate 2005 by far. The first game was released in 1999. Aforementioned F4, 1998. CFS3, 2002. Both have scale at least on par with X4 (arguably much, much more). If utilized correctly, we do wield ungodly amounts of computing power compared to machines available at the time. BTW my machine is clearly much newer than that (which was described in the second post), but that's besides the point. There are ways to utilize its powers in the right way.
Also remember -- not only was I talking about pre-computed engagement variants, but also pre-computed lighting. Seriously, try turning graphics to a minimum (esp. shadows) and see what I'm talking about.

[vvvvvvvv]

I didn't play Elite, but I know of its development and its severe hardware limitations. 22 kilo...

I'm talking about Elite Dangerous. You know, with 52 gigabyte install size, with latest update being released in March this year. The game does what you describe. Ships exist in a small bubble around the player, and few exist in the system. If you follow them around, the pattern is nonsensical. And pirates spawn out of thin air. About 40 ships will badly lag most systems, compared to X4's 1-2 thousands.

To have a statistical equivalent of fully simulated universe you have to simulate the universe. If you use precomputed scenarios, then you'll cut potential gameplay time by a huge factor because player will quickly learn those precomputed patterns then it'll become repetitive. It is a common problem with all "procedural" games, and it does not occur to the same extent in x4, BECAUSE it simulates everything honestly.

Major and central core feature is the game's living world. And I like that. It's gross overcalculation

And if you remove it the world will be no longer living. As simple as that.

Previous titles have significantly reduced complexity.

As for lighting, that a misplaced optimization optimization attempt. The renderer is quite well optimized and handles scenarios which kill Elite Dangerous. Because that game manages to be slow while displaying one station which is nowhere near x4 scale.

---

You're free to prove me wrong by making the game you describe, but as of know a game that does this better than X4 does not exist, and that indicates the problem is not as easy at it appears. X4 uses dwarf fortress approach. Most other developers use tricks and do not bother, that's why there is only one dwarf fortress, only one x4 and no good alternative to either of them. I think the only close possibility was Eve Online, where it was clicky MMO and you couldn't fly your ship directly.

Anyway, I'll be done with this. While a lot of things could be improved, requesting to remove critical core feature is not exactly reasonable. It is a very important feature and compromises you propose would ruin the gameplay. Yes the "precomputed scenario" trick will be noticeable. Think you could do it better, do it. You'll have to match picture quality and duplicate all features of x4. Good luck.

[Shadow_rainbow]

I didn't play Elite, but I know of its development and its severe hardware limitations. 22 kilo...

I'm talking about Elite Dangerous. You know, with 52 gigabyte install size, with latest update being released in March this year. The game does what you describe. Ships exist in a small bubble around the player, and few exist in the system. If you follow them around, the pattern is nonsensical. And pirates spawn out of thin air. About 40 ships will badly lag most systems, compared to X4's 1-2 thousands.

In ED, ships generally appear inside missions you fly between in warp mode. This has little relation to either X4 or what I have described. What harms ED is little variety of said mission variations that get repetitive rather quickly. And no, pirates there generally don't appear out of thin air, but that's besides the point. To my knowledge ED doesn't try and simulate trade route traffic to the same extent as X4, its strong points are different. So blaming it for seemingly random traffic (never tried to follow it anyway so can't really fact-check) is not helping. Neither is blaming it for optimization problems (although I never have observed anything to X4 levels of slowdown). We are not comparing X4 to ED here.
The main point is that it does not operate according to what I have described, being mostly direct simulation akin to X.

To have a statistical equivalent of fully simulated universe you have to simulate the universe. If you use precomputed scenarios, then you'll cut potential gameplay time by a huge factor because player will quickly learn those precomputed patterns then it'll become repetitive. It is a common problem with all "procedural" games, and it does not occur to the same extent in x4, BECAUSE it simulates everything honestly.

You know what helps against repetitiveness? Variety and interoperability. You've seen kaleidoscope, right? With enough pieces it will never feel repetitive. You can't pre-simulate the whole universe since player agency is involved, but if it wasn't -- then no one could prevent Egosoft from pre-simulating its evolution a few million times to provide enough variety to look at and storing this data for quick reference. The size of that model in case of stable universe would be pretty hefty though due to long evolution times.
Again I stress that it's important to implement statistical models in a good way, but if that is done properly, player won't be able to tell. And that's the important part here. Aside from it not slowing down even for truly massive battles.

Major and central core feature is the game's living world. And I like that. It's gross overcalculation

And if you remove it the world will be no longer living. As simple as that.

Previous titles have significantly reduced complexity.

As for lighting, that a misplaced optimization optimization attempt. The renderer is quite well optimized and handles scenarios which kill Elite Dangerous. Because that game manages to be slow while displaying one station which is nowhere near x4 scale.

Through this whole topic I have been explaining that calculating the world in ways other than direct real-time calculation of all its parts does not make it less living. If you can imagine scenario where this is not true in your opinion, please feel free to share it. That would be an interesting exercise.

As for lighting, every bit helps when FPS drops below 30. But that wasn't even the point. The point is textures with baked lighting would not take more space or anything, but looked incomparably better.

a game that does this better than X4 does not exist

I have literally mentioned at least two games that equal and surpass X4 in world scale and complexity...

[vvvvvvvv]

...

Dude. I played a ton of stuff, and your well-meaning idea does not work in practice. It does not work the way you expect it to. "Precomputed scenarios" result in patterns and those become quickly recognizable. X4 the way it works now lacks this typical flaw of smoke and mirror procedural games. Your idea will cut down replayability by something like factor of 10 or more. Right now replayability is stellar. "If it is hard to run the simulation" then getting more powerful hardware is an option. It is one of the kind game, after all, makes sense to throw a stronger CPU on it if you're having fun with it.

Elite, by the way, simulates a ton of macro data including trade routes. However none of this is reflected in the world below at a player level, so we get a dead world with excel sheet attached. And yes, pirates spawn out of thin air. A typical scenario is a RES site where you'll be facing infinite pirates who will arrive in waves, in powerful ships worth millions all ready to throw their life away to steal the damaged escape pod you picked 5 minutes ago.

Like I said, you're arguing against important core feature. Maybe it doesn't look important or core to you, but other people exist and it matters for them. If you disagree with design decisions, the best idea is to play something else, maybe Avorion. The simulation aspect is the reason why I play x4 instead of Elite even though I'm a VR user, and Elite offers limited VR while X4 does not. The simulation aspect improves the game THAT much. Also for "but it is easy to do and have larger scale..." a good idea is to question why is there no other game that did this. Because clearly other people had this idea before, and it is highly likely egosoft did think of this before and tried it. Yet no game exists that does this. Why is that? Think about it.

That'll be the end of argument.

Have fun.

[Blaze1st]

I'm not entirely convinced that absolutely every ship is a tracked entity because when I flew to the megastructure in Eleventh Hour there were regular contacts popping up from every AI faction in a place they have no business being since it is so far off grid.

If they are tracked entities then the AI is sending them to some very strange locations potentialy...

[vvvvvvvv]

I'm not entirely convinced that absolutely every ship is a tracked entity because when I flew to the megastructure in Eleventh Hour there were regular contacts popping up from every AI faction in a place they have no business being since it is so far off grid.

If they are tracked entities then the AI is sending them to some very strange locations potentialy...

You have a fair point here because those contacts you encountered DO spawn out of thin air.

The game has "random encounter" feature, which functions like what OP described. When you go outside of sector borders, every few dozen kilometers you get preset encounter materialize on your radar, usually it is "Faction X vs Faction Y". Which is what OP pretty much described.

Except those are dull, just like I said. You quickly begin to see patterns there and nothing fun ever happens with them. There were mods that disable that feature.

Pirates that roam sectors, however, are tracked. And reducing number of simulated entities is not an acceptable compromise.

[CBJ]

You have a fair point here because those contacts you encountered DO spawn out of thin air.

That is not generally true. In most cases they are simply "borrowed" from elsewhere.

[vvvvvvvv]

You have a fair point here because those contacts you encountered DO spawn out of thin air.

That is not generally true. In most cases they are simply "borrowed" from elsewhere.

SCA pirate boss: "All pilots, sssstand ready. The target approachesss, but with six of usss, the chancessss are good."
SCA pirate underling: "Three of us, boss"
SCA pirate boss: "What? We clearly had ssssix ships when we left, where are those idiotssss?"
SCA pirate underling: "You know how it is boss. White flash of light, and then you're suddenly away from the sector on the fringes fighting xenon. If you win, you return. Happens often."
SCA pirate boss: "What sort of ssstupidity are you talking abo..." *poofs in a flash of light*
SCA pirate underling: "And here we go. They never believe me. Back to the base, guys? .... GUYS? Hello?... Oh, hell, not again."

[Blaze1st]

You have a fair point here because those contacts you encountered DO spawn out of thin air.

That is not generally true. In most cases they are simply "borrowed" from elsewhere.

SCA pirate boss: "All pilots, sssstand ready. The target approachesss, but with six of usss, the chancessss are good."
SCA pirate underling: "Three of us, boss"
SCA pirate boss: "What? We clearly had ssssix ships when we left, where are those idiotssss?"
SCA pirate underling: "You know how it is boss. White flash of light, and then you're suddenly away from the sector on the fringes fighting xenon. If you win, you return. Happens often."
SCA pirate boss: "What sort of ssstupidity are you talking abo..." *poofs in a flash of light*
SCA pirate underling: "Here we go. They never believe me. Back to the base, guys? .... GUYS? Hello?... Heck, here we go again."

Similar thoughts immediately went through my mind too...

I've got to assume there are some rules around what can safely be borrowed to avoid that sort of situation!