Intermittent freezes in TC, worse in AP - Resolved by system reset.

[zhaarteth]

I got the X3 GoldBox on Steam in early December, and have yet to have a decent playthrough of either TC or AP due to lag spikes which occur completely at random, though are worse in high-density sectors and they render any attempts to engage in combat rather pointless. This issue is equally pervasive when using minimal graphics settings as it is when using full. X3:R seems fine though, even if wildly different from the other two.

Firstly, here's my DxDiag. Of course the first thing I tried was a Steam Verify. I doubt that I have any troublesome FFDShow, Haali, or MainConcept codecs as I can't find any, nor can tools like Codec Sniper. I also doubt this is the infamous BBB issue since it also happens with a new game and has an equal tendency to get worse in the above criteria regardless of how many minutes or hours have passed in-game. This was first noticed in Vanilla (TC/AP), and for the first week I was precariously avoiding any combat whatsoever in TC and not even touching AP in order to completely verify that it was the amount of objects in a sector which tend to make the issue worse. As such, ever since I began using the Cheat Package (in both TC and AP) so that I could comfortably correlate it with any system- or process-related occurrences, I have not been able to verify whether or not using mods affects the issue (for better or worse), but I suspect not.

I have spent a few weeks tweaking different system settings, doing a good clean-out with tools like CCleaner and Defraggler, culling resource usage (such as shutting down any non-essential programs, setting in-game graphics to their lowest) and temporarily disconnected my secondary monitor as well as temporarily setting my visual theme to Classic. I've tried temporarily resetting BIOS to defaults, checked for BIOS updates, done boot-time virus scans, and even tried Razer Game Booster. I've also tried temporarily disabling different power-related BIOS features such as SpeedStep and C1E Support. Other attempted solutions listed elsewhere in the forums include using WinXP/SP3 Compatility mode (I've tried others, including Vista) as well as booting my machine without a present sound card. I've already had onboard audio disabled in BIOS since I've got an X-fi. I've also tried disconnecting non-essential USB devices. I've also been able to verify that other games which lack multi-core support run just fine; it is only X3 which has this particular issue. One other thing which may be prudent to mention is that I use a PAE-enabled kernel, which allows 32-bit Windows to fully utilise my fitted RAM. However, I've booted with the original kernel and was able to verify the issue persists regardless, nor does it worsen. One last thing I tried was to temporarily disable Avast anti-virus. Nothing I've tried has worked to address this issue so far.

To be clear, when I say "lag spikes" I mean complete yet momentary freezes in the game as covered by this FAQ entry. Everything takes a break; image, sound, everything. Any open in-game windows close as if I ticked their X buttons, and parts of the interface (sidebar, dynamic readouts, etc.) disappear. All other programs are fine (e.g. Steam and Fraps, etc.), but during the spikes, the game is no longer responsive, stops using up the graphics card, and the core which X3 is running on has a steady 80% usage rating (contrary to normal activity, which is labile). These lag spikes I am experiencing are always correlative to when the game's RBD skyrockets (clarification below*) and, in between these freezes, the FPS momentarily goes back up into the normal range**.

I am posting this in advance of pending hardware changes. I was going to post after them, but another thread caught my attention as I thought another person was having the same issue, but it turned out not to be the case after some verification with Process Explorer. As I am rather skint I don't have an ETA for these hardware changes yet (current guess is late Q1 or early Q2), but they only consist of getting a new HDD and changing to a 64-bit version of Win7Pro. While I have a distinct suspicion to the contrary, I am not willing to rule out that a complete OS do-over might help, if not completely solve the issue. I am also anxiously awaiting X3 arriving on Linux as well, especially given that it'll use SDL (something I am also fond of). I run a dual-boot system (Win7 and Kubuntu) so it'd be nice to verify whether the issue's culprit is hardware or software. However I am posting this thread in advance due to my aforementioned suspicion, and that I've already tried everything which I can think of short of a complete system wipe.

*When monitoring the game process's activity using Process Explorer I have been able to confidently verify that the game's I/O Read Bytes Delta skyrockets anywhere from 20MB/s to 40MB/s every time these lag spikes occur. Normally this would mean that it's loading something from the disk, but Disk I/O Bytes Delta shows hardly any activity when this occurs. So whatever is going on, there is an issue with something in active data, not on the disk. In any other circumstance (if similar issues were happening in other games) I'd say that something's amiss with either my RAM (which I've replaced earlier today) or SATA3 controller. But since X3 is literally the only game which I am currently having abnormal issues with (ArmA2, Saints Row 3, Warhammer 40k DoW2, Darksiders (and DS2), Orange Box games and even Black Mesa Source, as well as various Bethesda games including Skyrim and Rage, et cetera... they all run just fine) I highly doubt that this is the case. Moreover, high I/O RBD usually correlates with high Disk RBD - especially during loading screens; this is a general fact. So, again, everything points to something in active data which is amiss and being re-read or perhaps re-allocated to the point where technical madness sets in. I don't necessarily blame something within X3 itself as the culprit but whatever is going on is only affecting X3.

Apropos for those unfamiliar with ProcEx: Don't confuse total Disk Read Bytes on the second image with Bytes Delta. Delta means "happening now" whereas the former either can mean "average" or "total so far."

**I have a Salvage Insurance save right now (whose playthrough is the sole purpose of hunting the cause of this issue) where I am just about to aid some friendly Teladi to engage an entire Pirate squadron consisting of roughly 10 or so M2, M3, and M4 class fighters (along with a pirate cruiser dispatching some 3 or so Blastclaws and a Blastclaw Prototype just before it enters firing range -- but this is entirely an estimate, I don't have a chance to look at the sector screen in order to properly verify) which puts my initial FPS somewhere around 40, dropping to the lower 20s when everyone opens fire. Just under 10 seconds after they open fire, the entire game begins grinding to a halt, the RBD goes to 40MB/s, and the symptoms I mention above all come into play - rendering any combat without cheats suicide by default. That's why I am using the Cheat Package, otherwise I would likely have never been able to so confidently confirm the correlation between the lag spikes and high RBD count. All that said, the save itself has been since overwritten to just after the pirates are engaged. Clumsiness on my part. Also I'm not sure what the rules are about posting cheat-enabled savegames. If it's alright I'll post the savegame so others will see what sort of situation has the most potential to put these lag spikes at their absolute worst.

Conclusion: Either I should wait until I've completed my new HDD and OS installation before I attempt any further solution candidates, or I am a complete idiot and the culprit/solution has been staring me in the face the entire time. Both are possible at this point, I'm stumped.

Also I feel like I should apologise for my unremitting usage of parenthesis. I need to seriously work on my phrasing.

[Alan Phipps]

I still think that your 32 bit OS is likely to be limitiing your gameplay available RAM below the 4 Gb optimum available to it and so possibly requiring (or at least requesting) HDD pagefile usage such as when you get the glitches. Your graphics card will still require VRAM addressing and actual shared graphics memory on top of OS/driver/services requirements and I am unsure if PAE moves any of this into your additional 2 Gb RAM when it is all servicing the same single primary exe.

There are also some issues about PAE that leave me unconvinced of its efficiency even should more than 4 Gb RAM be fitted (as in your case) when used with a cpu/RAM-hungry single-thread 32 bit exe such as with X3:

PAE Wiki:

The 32-bit size of the virtual address is not changed, so regular application software continues to use instructions with 32-bit addresses and (in a flat memory model) is limited to 4 gigabytes of virtual address space. The operating system uses page tables to map this 4-GB address space into the 64 GB of physical memory. The mapping is typically applied differently for each process. In this way, the extra memory is useful even though no single regular application can access it all simultaneously.

...

However, "client" versions of 32-bit Windows (Windows XP SP2 and later, Windows Vista, Windows 7) limit physical address space to the first 4 GB for driver compatibility [5] and licensing[4] reasons, even though these versions do run in PAE mode if NX support is enabled.

[zhaarteth]

I still think that your 32 bit OS is likely to be limitiing your gameplay available RAM below the 4 Gb optimum available to it and so possibly requiring (or at least requesting) HDD pagefile usage such as when you get the glitches. Your graphics card will still require VRAM addressing and actual shared graphics memory on top of OS/driver/services requirements and I am unsure if PAE moves any of this into your additional 2 Gb RAM when it is all servicing the same single primary exe.

While I understand your scepticism, Alan, I must ask that you also understand mine.

Firstly, if there were any hard page file usage by the game (or anything else) there'd be corresponding HDD activity specified not only by the IDE indicator LED on the case itself, but also by Process Explorer. The bottom of the main ProcEx window indicates Physical Memory usage as well as swap usage separately, and the former is the only percentage which goes up when running X3 or any other game. Keep in mind that just because the page file has a usage rating, this doesn't mean that the page file is only being used on the disk as swap. Active data is allocated into the page file, whereby the OS decides where that active data goes - either RAM or swap.

Second, if the issue was being caused by a lack of available physical RAM, then it would not be the case solely for one game. I run quite a few other memory-hogging games and apps, and none of them experience any slowdowns. Next time I find myself using one of them for longer than an hour I'll take another ProcEx shot and post it.

There are also some issues about PAE that leave me unconvinced of its efficiency even should more than 4 Gb RAM be fitted (as in your case) when used with a cpu/RAM-hungry single-thread 32 bit exe such as with X3:

PAE Wiki:

The 32-bit size of the virtual address is not changed, so regular application software continues to use instructions with 32-bit addresses and (in a flat memory model) is limited to 4 gigabytes of virtual address space. The operating system uses page tables to map this 4-GB address space into the 64 GB of physical memory. The mapping is typically applied differently for each process. In this way, the extra memory is useful even though no single regular application can access it all simultaneously.

...

However, "client" versions of 32-bit Windows (Windows XP SP2 and later, Windows Vista, Windows 7) limit physical address space to the first 4 GB for driver compatibility [5] and licensing[4] reasons, even though these versions do run in PAE mode if NX support is enabled.

I'm not sure what wiki you're getting this from but you might want to re-read that first paragraph, as well as this article for insight on how Windows handles memory allocation. I am not looking to make 32-bit programs LAA on a 32-bit OS (that would be... odd...), I'm just giving Windows more breathing room, so to speak, before it starts relegating the page file to swap. But when does X3 ever use more than 4GiB of memory, even on 64-bit systems with an LAA flag? If it's supposed to be using more than that, I'll need some proof before I go and write this off as an abnormal recursive page fault issue.

I must reiterate that if this issue was indeed somehow being caused by the PAE kernel itself, or if the game wasn't being allotted more than it needs (thereby using swap) then it would not be only X3 having problems. Moreover, the issue would likely worsen when running the regular non-PAE kernel. I just tested this again before posting. Not only do I often have quite a few programs running simultaneously (not while gaming, of course) which thereby collectively utilises the advantage of x86 PAE, going to the System window shows all 6GiB available as opposed to specifying only 3.25 usable. ProcEx confirms this as mentioned above, and the game experiences the same issues and showing the same exact symptoms, the total available Physical Usage only goes to ~70% when not running the PAE kernel as opposed to only ~50% when using it, and there is still no major swap activity in either case. As such I'd rather put this one aside (but not rule it out) for now and try other possibilities until I can install Win7Pro x64. I've got a copy now, the only thing stopping me from installing it straight away is that I need a new HDD in order to do proper backups.

On the other hand, if you want to put this thread on a temporary hold I won't argue, given why I posted. If you're dead-set on verifying memory allocation as the culprit, then this can wait until I have an x64 version of Win7 installed and at the ready for further testing. It'll take roughly two weeks minimum, including ship time. Either way, I appreciate your time.

[Alan Phipps]

Yes, see how the OS x64 goes.

I am not at all saying that PAE is ineffective or a bad thing. What I meant was that I did not know whether it allowed a full 4 Gb RAM as an exe exclusive and always reallocated other requirements into the remaining 2 Gb you have, or whether it suffered the same penalties of co-allocation as a 32 bit OS from graphics card (VRAM and shared memory), OS, services and drivers requirements where they were all associated with support for the same exe. You, as the PAE user, are better placed to confirm that one way or the other through process monitoring and VRAM address allocation, so it does not really become a matter for scepticism on either part.

All I have to go on is experience that 32 bit OSs requiring pagefile access in-game exhibit stutters and slowdowns as described in the symptoms. I am aware though that similar symptoms do come from different causes and also that my knowledge of modern OS/cpu architecture is lagging well behind the curve. I do have an open mind though and appreciate being further educated by new situations.

[cakes]

I've updated djnoob's thread with some further testing of DPC (delayed procedure calls) to detemine whether an unrelated device driver is causing unnecessary CPU interrupts.

PAE was turned off in WinXP with SP2, quoted as instability with device drivers. With Win7, you're at the mercy of device driver developers as to whether this issue still exists and whether they actively comply with MS's guidelines.

Alan, from what I've read (and hopefully understood) each maximum 32-bit addressable segment is turned into a seperate page table. So if you had 16GB of RAM, this would be addressable as different tables in clusters of 3.xx GB. Processes are still limited to 2 or 3 GB per process, although they can run in individual tables.

[zhaarteth]

I need to stop hanging around the type of people who mean other things when they say "I don't know if..." about certain matters. My interpretation gets rather skewed that way, sorry for that bit of awkwardness Alan.

Yes, cakes, I've already had LatencyMon (somewhat like DPC checker but more specific) installed for quite a while now. Upon running it when you mentioned it in the other thread, I immediately noticed three drivers that are having very high/ongoing ISR and DPC ratings going over 1m within minutes: ataport.sys usbport.sys and dxgkrnl.sys

The DirectX driver, specifically, seems to make sense of another thing I've noticed during testing; sometimes planet textures (and only planets) go entirely grey after the first 5 or so major stalls in the game. My google-fu is failing me at figuring out what exactly would cause ataport.sys to go haywire like it is, and I'd have thought this sort of thing would give far more than one game such issues. That bit still has me puzzled more than anything, that only X3 out of all the other games I play and stuff I do is having any issues. Still, every result I can nab off a web search regarding "ATAPI driver extension latency" and "USBPORT.SYS latency" is either inconclusive or resolves completely unrelated problems (such as major differences in hardware and the like).

I've spent the past seven hours disabling different hardware components to see if the ISR and DPC ratings on those three drivers would stop climbing. I've systematically disconnected or at least disabled in BIOS, one by one, every device/function that I even suspected might be related to the three troublesome drivers -- and even did a hail mary and tried disabling every single non-essential function/device possible. Nothing seems to calm these little guys down. I may have a better idea of what's going on, but I'm still very much puzzled at X3 being the only thing having any trouble with... whatever it is. But since DirectX is included in the list of troublemakers, and given the apparent correlation with grey planets, the last thing I tried right before posting was switching out my nVidia with an older ATi 4670 1GB card. No joy. It's like no matter what I do, these freezes are like a curse. Rawr.

Something's been bugging me for a while now, though: If X3 requires more runtime memory than a 32bit OS can allocate to it exclusively in order to run smoothly, then why is it built on 32bit architecture? Colour me confused on that one. This is the first time I've seen a game (or any program) not only lack multi-thread capability but also require a 64bit OS in order to run smoothly despite only being available as a 32bit runtime.

As for driver incompatibilities regarding PAE, I've already stated that this issue is present (and precisely the same) regardless of which kernel I boot with; vanilla or PAE. I keep the non-PAE kernel separate for this very reason; to verify if the issue I am experiencing is the fault of an incompatibility there. As for understanding Windows memory addressing, you're absolutely right, cakes.

So while I'm still confident that this is not a swap-affecting-performance sort of issue, and I now suspect a driver or perhaps even... a very, eh, unique... hardware fault, I am going to put getting Win7Px64 installed (& HDD get) at the top of my agenda for this issue. While I most certainly hope the solution is as simple as doing so... I've many suspicions to the contrary. Here's to hoping those suspicions are nothing but mule muffins. <let's just pretend there's a cider toast emote here>

As a last note before I surrender to sleep, some deja vu: For the past few times I've booted my test-save in X3, the stalls are beginning to appear less extreme/frequent in that battle with the pirates. I was even able to count the number of hostile ships, and correct myself regarding which race I was helping; 13 pirate fighters and a Brigantine, in Elana's Fortune, an Argon sector. Maybe I'm just getting used to working with the stalls after spending so much time trying to be rid of them. Heh.

[Alan Phipps]

Hey, one I can answer I think!

"If X3 requires more runtime memory than a 32bit OS can allocate to it exclusively in order to run smoothly, then why is it built on 32bit architecture? " No, X3 runs smoothly at less than 4 Gb RAM as it is indeed a 32 bit exe - the X3 engine was designed in the XP era. In fact the exe can usefully employ up to about 3.5 Gb RAM on a 64 bit system with the RAM before it decides to flush its cache.

The issue as previously debated is that on a normal 32 bit OS, especially with a high end graphics card with up to 2 Gb VRAM addressing already withheld from the available 4 Gb addresses, the game exe gets no chance of access to that 3.5 Gb RAM after necessary deductions. On some systems, or with the odd background application going as well, the game exe may be lucky to get much more than 1 Gb available RAM. That is when slow pagefile use becomes a real and frequent pain in-game, especially in busy scenarios.

[cakes]

As X3 is a process, this will communicate directly with the CPU dispatcher, as opposed to kernel mode drivers being queued by DPC prior to the dispatcher executing them in order. Combine this with the high CPU utilisation and small read/writes genetated by the X3 process, we can then start to see that any latency in the DPC queue or the dispatcher might only present itself via the X3 process. Other applications might not meet these criteria and therefore don't present a visible issue to the user, although are still present with a lower impact.

Some basics really I may have missed you discussing:
- Run a system file check in Windows.
- Reset your BIOS to defaults as a test.
- In Device Manager delete all your USB Hubs and Root Hubs, then rescan for changes, incase there's a USB enumeration issue causing conflicts.

[cakes]

Hey, one I can answer I think!

"If X3 requires more runtime memory than a 32bit OS can allocate to it exclusively in order to run smoothly, then why is it built on 32bit architecture? " No, X3 runs smoothly at less than 4 Gb RAM as it is indeed a 32 bit exe - the X3 engine was designed in the XP era. In fact the exe can usefully employ up to about 3.5 Gb RAM on a 64 bit system with the RAM before it decides to flush its cache.

The issue as previously debated is that on a normal 32 bit OS, especially with a high end graphics card with up to 2 Gb VRAM addressing already withheld from the available 4 Gb addresses, the game exe gets no chance of access to that 3.5 Gb RAM after necessary deductions. On some systems, or with the odd background application going as well, the game exe may be lucky to get much more than 1 Gb available RAM. That is when slow pagefile use becomes a real and frequent pain in-game, especially in busy scenarios.

Alan, are certain sounds loaded into memory on demand in X3, then flushed? Or are they often cached. I'm still thinking that sounds are being loaded in, and queued, causing the latency. Just a theory. The same would apply to textures.

[Alan Phipps]

@ cakes: You would have to ask that on either DevNet or S&M to get a good answer - I'm just a player. I do know that some relatively rarely-used video effects (eg ship explosions) seem to have something loaded from the game files on HDD every time they are used.

You could also try a PM asking the question to Arsaneus of Egosoft with a link to these recent threads. He might even occasionally view these Tech Sp threads and so be familiar with the issue background.

[cakes]

Something you've said previously excited a few dead brain cells

Yeah, explosions was what I was trying to remember in this case.

[zhaarteth]

After days of fiddling with hardware configs, Windows, and BIOS settings, (I hate .NET, seriously) I finally was able to resolve the issue after extensive testing. There was one last thing I wanted to test, and intentionally so.

As I suspected, when switching from Enhanced IDE SATA mode to AHCI I was only able to boot into Linux whereby I discovered that in doing so I had caused classpnp.sys to freak, resulting in an infinite startup loop.

So instead of pulling an "Okay..." sadface, I bit the bullet and installed my copy of x64 Win7 Pro rather than waiting on my new HDD... 28 hours later, booted up the game and ERMAGHERDNOMOARFREEZEZ!

Now I can finally dispose of the cheat package and do this thing proper.

I'm still not entirely certain what the issue was (most everything looks the same in Device Manager), and at this point I'd rather not care. I'm just glad my suspicions about a "pointless" system reset were wrong.

Thanks for giving me somewhere to keep track of what I'd done so far, feedback, and for your patience. Now I can give those pirates a good what-for.

Appending thread title with solved tag.

[Alan Phipps]

Good to hear, and at least you are now using all your fitted RAM optimally. Enjoy the game.