[Official] Technical Engine Development faq / blog

[Tamina]

I'll make an update about dx9/11/12/opengl linux/mac a bit later.

I would like to ask, since XR has gone 64bit which propably means that 99% of the users are theoretically at minimum DX10 standard (-how am I kidding, that would be Vista: minimum DX11 standard) would XR benefit being ported to newer API versions?

I tried to look it up, but there are many different sources which all say that Dx11 has less performance then Dx9 and vice versa.

Although all state that Dx11 is way more beautiful
(don't know what D3D actually covers but I doubt it gets 'automatically' more beautiful by changing the D3D version, or does it?)
the benachmarks about performance are really mixed, while Microsoft states that Dx11 should be faster.

So would XR benefit from a higher DX-Version and is this a goal for the future of XR?
(Again I am assuming that all users (I guess Steam gives Egosoft a chart with the computer specs of their players?) have to be beyond Dx9)

[rboerdijk]

1) Slightly controversial so I'll choose my words carefully and respectfully...

We experienced performance and scaling issues on high end hardware at release, clearly some problems have been identified and fixed because things have improved well done... So what did you find, what was causing some of the performance problems beyond hardware drivers etc I am talking about the game specifically?

We performed quite a few of tasks, mostly to
1. smartly "skip" work
2. fix threading issues ( threads blocking each other ).
3. figure out why something fast wasn't working as intended.
4. change the scenario / level.
5. new features "stealing" performance.

An example of #1 was the cpu-render side, trying to predict which objects to discard - since they wouldn't end up on the screen anyway. For example we had culling code for asteroids which worked quite well - but at one point we ended up testing 20k-30k asteroids individually each frame. Since we have a nice grid with "big groups of asteroids" this could be easily improved, which dropped the number of tests needed by roughly 70%.

An example of #2 was in our way of threading, which ended up causing many thread-switches when loading directX textures/meshes (this was fixed quite early). We had some problems where worker-threads could end up blocking our main-thread.

An example of #3 was in asteroid rendering, more specifically instancing. We can't instance models with an alpha - but at one point some asteroid-models got that. Basically resulting in all asteroids being drawn individually with thousands of draw-calls - not good.

An example of #4 was certain regions with ice-asteroids, where the level-setup was such that there were way too many - even after all other optimizations. We ended up putting the counts down to reasonable levels.

Same for "choke-points" where too many ships ended up.
We used several approaches to spread the ships out more in these cases.
- from the AI-side trying to avoid ships going somewhere where "too much" activity is
- having more locations for certain asteroids-fields.
- add certain stations so ships will spread more nicely between them.

An example of #5 were lensflares. They could end up doing many "long-distance" raycast ( > 50km). We optimized those by doing a short 1-km raycasts (factors faster) because roughly 1/2 of those flares would be blocked by its own ship.

We did way more optimizations, but I hope this gives some insight.

[CBJ]

Just to add to that, performance improvements in several updates were obtained by improving the LOD setup of some of the graphical assets.

[ezra-r]

note that I know nothing about programming but the simplest coding, even less how to make use of the memory available in the system.

But question is about memory usage:

Since the move to 64 bits binary more than the usual 2GB of ram can be used and we clearly can see that, when alt+tabbing game most time uses 3.2 gigabttes sometimes I recall it even using near 4 gigabytes.

Sometimes when I turn around things get "re-painted" and sometimes when I use highways or in other situations I notice hiccups, which I assume it is the game loading resources.

In my case I have 16gb of ram and a video card with 3gb of ram.

* Can you guys "make" the game use more memory when available to pre-cache stuff that will be used later to avoid hiccups at most times or this is unavoidable?

Like, why don't I see the game using 6 gigabytes when I clearly have them spare.

PS: hope I made some sense.
PS2: Thanks for this thread.

[wwdragon]

* Can you guys "make" the game use more memory when available to pre-cache stuff that will be used later to avoid hiccups at most times or this is unavoidable?

Like, why don't I see the game using 6 gigabytes when I clearly have them spare.

I like that idea.
I also have 16 gig on my machine; 1 gig or so for os+ proggies and that leaves 14 gig that it could use!
Maybe a graphics options checkbox for MOOOOAAARR ram usage?

[vadiolive]

Q: how bad/god effect to X-rebirth if i call Windows API to CLEAN Workspace in memory each 20s?

[timon37]

Sometimes when I turn around things get "re-painted" and sometimes when I use highways or in other situations I notice hiccups, which I assume it is the game loading resources.

Actually surprisingly often the hiccups aren't due to asset loading, but scripts/AI, pathing, or other universe management code.
Loading and rendering assets is quite well decoupled so it usually doesn't stall but of course then you get popping/repainting.

* Can you guys "make" the game use more memory when available to pre-cache stuff that will be used later to avoid hiccups at most times or this is unavoidable?

In general yes that is a good idea, and the game isn't doing that very well at the moment. It's mostly a case of do you "improve this" or "fix that other broken thing".
Besides the OS normally caches the file system so technically all of the game files should already be in memory, and when we read them it's just mem-copy/decompress.

Q: how bad/god effect to X-rebirth if i call Windows API to CLEAN Workspace in memory each 20s?

Honestly I have no idea what that does, but I don't think there's much good it could do. If you're running low on memory your best bet is making sure less stuff is running in the background, maybe doing that clean once at the beginning and then just play without alt+tabbing out.

[edqe]

Could you tell us a bit what tools you are using; gcc/clang, git, etc. ?
I have read that some developers prefers using clang during the development process because of faster compiling times but uses gcc for production code.

I can imagine that Linux game development tools are still far behind Windows because Windows' 30 year head start. What kind of tools Linux needs most and has things changed better since Valve jumped to support Linux gaming?

[timon37]

@edqe (post above)

We use svn, though I locally use git. Right now gcc, but actually thanks for reminding me about trying clang;p

On the subject of compilation time, it's horrible... C++ is horrible... and it'll never get any better. I had to change my habits since I'm used to about 2-7 sec compilation-link-launch-start_testing time.

For general programming gnu/linux is much better, build tools and general methods for the development process are (most of the time) simply superior by design. VS project configuration is... beyond words. When I started with opengl on windows I simply said to myself "nope, not doin' that" and used cmake, which helped me a lot:)

But of course the main thing is graphics debugging, there it gets tricky, and actually I don't entirely know yet. I have really high hopes for vogl which I couldn't compile on windows and *spoiler alert* I'm not yet rendering on linux:( But I have vogl compiled on linux so it's prepared for the glorious day;)

Up to now for opengl debugging I've mostly used gdebugger/codexl, that mostly worked on linux same as on windows, which is so-so, crashes often, fails to display many things etc.
Intel's gpa: is kinda good, but not great, in general it's been quite helpful for DX9 debugging.
Apitrace: didn't use it much.
Gpu perfstudio: also hasn't been the most useful though not useless, used it mostly for debugging the command stream.
Finally as painful as it is, I have to admit nvidia nsight is currently the best for profiling or debugging a frame when there aren't any basic opengl errors.
There's a version for eclipse and linux but I'm guessing it's only for CUDA.

I'll be able to say more once things start rendering on linux;)

P.S. I also have really high hopes for valgrind, mostly for performance profiling ironically;p

[ezra-r]

@timon37 about the linux compiling and linux version of Rebirth, I'd love to have any kind of updates in here too..

Also, why don't you guys just "STICKY THIS THREAD" and the "Behind the Scenes" thread so it is not going up and down and people know where to look for specific insight info instead of gluing the two threads together which imo would just confuse things more, at least I see the behind the scenes thing as a graphics insight, and this is clearly programming insight.

Really enjoying this thread!

[edqe]

@timon37: Thanks for the information.

As you mentioned the GPU debugging is one of the biggest problems at the moment. Good to see Valve has started to work tackle that problem as well.

How do you see Linux GPU driver situation at the moment? Worries ahead?

Since containers (systemd-nspawn, Docker, ...) are quite hot topic in Linux field at the moment I was wondering if you are using them on your development work. I guess Valve SDK supports/requires some sort of sandboxing like chroot at least.

It would be great to hear time to time how things are progressing. Exciting times

Edit: Oh, for some reason I misread your post and you had mentioned VOGL already. Comments about it removed

[Sam L.R. Griffiths]

@timon: With all due respect, I have developed on both *NIX and Windows and IMO although EMACS is probably still my favourite text editor (and IDE for some programming languages) I must bow to the superiority of the Visual Studio IDE for C/C++/C# programming. I would say though that IMO Microsoft's IDE (and Office Tools) has got worse as it got newer, my favourite version of Visual Studio has to be either 6 or .NET 2003 - the latter versions are too quirky in some respects IMO.

In my early days as a software engineer, I heavily relied upon real-time debugging but over the years I have come to realise there is no real substitute for using runtime logging to track down issues with modern software. The latter has a lower impact on runtime timing issues and real-time debugging often can cause problems tracking down the root cause of issues IME (even if the software is designed/written appropriately).

@CBJ: My personal point was primarily to do with being unable to script AI to respond to key/controller presses (c/f MCSI scripting in the X-Trilogy) - may be possible with certain modder developed libraries but it should be part of the vanilla capability IMO. The other side of it is being able to do custom menu screens for input, perhaps just for in-game configuration of mods/scripts.

I actually agree with your approach of using the XML based scripting language, but IMO we are missing key functionality from it (or sufficient documentation for how LUA is to be used by modders in tandem with it).

[rboerdijk]

@timon: With all due respect, I have developed on both *NIX and Windows and IMO although EMACS is probably still my favourite text editor (and IDE for some programming languages) I must bow to the superiority of the Visual Studio IDE for C/C++/C# programming. I would say though that IMO Microsoft's IDE (and Office Tools) has got worse as it got newer, my favourite version of Visual Studio has to be either 6 or .NET 2003 - the latter versions are too quirky in some respects IMO.

Concerning Timons comment on the msvc-"project configuration"-dialog, I do have to agree with him, it is kind of annoying - at least for a project our size. We have 11 configurations (debug, release, master, master-profile, viewer, ... ) and each of those in both 32 bit and 64 bit. It's way too easy to mess up something. I personally like msvc, especially the debugger is quite good. Linux guys just like screens of scrolling text, Timon is no different

I do have to say that I like the newer Xcode versions quite a lot, though.

[CBJ]

My personal point was primarily to do with being unable to script AI to respond to key/controller presses (c/f MCSI scripting in the X-Trilogy) - may be possible with certain modder developed libraries but it should be part of the vanilla capability IMO. The other side of it is being able to do custom menu screens for input, perhaps just for in-game configuration of mods/scripts.

Those are jobs that would, and should, be done using the UI-related languages and other configuration tools, i.e. Lua and the XML files used to define controls, because they lie outside the domains of the domain-specific XML languages. Cross-domain "leakage" is a really great way to destroy the advantages brought to the table by using domain-specific languages in the first place!

I actually agree with your approach of using the XML based scripting language, but IMO we are missing key functionality from it (or sufficient documentation for how LUA is to be used by modders in tandem with it).

We are well aware that the ability for modders to change or add to the UI easily via Lua and Anark is a weakness of the modding capability of the game at the moment. However, the solution to this is not to work around it by adding UI modding capability to some other part of the system; clearly what is needed is better accessibility to the existing Lua and Anark tools, so that modders can work on it in the same way that our own UI developers do it.

[hourheroyes]

Where'd Matthias get that sweater from last year's release photo? And will one be made available on the Egosoft store?

edit: referencing this post http://forum.egosoft.com/viewtopic.php? ... sc&start=0

[rboerdijk]

Sometimes when I turn around things get "re-painted" and sometimes when I use highways or in other situations I notice hiccups, which I assume it is the game loading resources.

* Can you guys "make" the game use more memory when available to pre-cache stuff that will be used later to avoid hiccups at most times or this is unavoidable?

Asset-loading runs on a separate thread and is unlikely to cause framerate-hickups. Occasionally people here in the office people come to me after they observed some stuttering with the same suspicion "it's probably asset loading" - and my usual reply is "no, it's not".

If an object isn't loaded in time, you might see it popping (the moment it is loaded) - but generally asset-loading will not cause framerate hickups.

In 3.0 rc3 we fixed several causes of jitter. As an example, we had an expensive physics-update in highways (ironically introduced in 3.0 rc 1 - to fix a bug related to mass-traffic), which we replaced by a much cheaper partial update.

[dmk]

Small jitter (0.1 sec) still there, when new zone appears, while skunk in highway, it's enough to bump to some slow traffic.

question is: when gate view becomes beautiful as rest of game?
i.e. like portal(see though) and without "black" 1 sec after passing (it's enough to bump on boost to some "invisible for 1 sec" ship outside of the gate)

i.e. I understand performance issues, but they could be handled using same way as was in gothic 1 with caves, while you X meters away only static texture is rendered, less - it's becomes transparently and appears world behind it.
for simplicity sake any weapon fire should not pass though gate (but may cause water like splash ( http://madebyevan.com/webgl-water/ ).

try to ride balor or lepton from omicron to albion though TO space and watch for gates .

[BigBANGtheory]

Egosoft obviously receives an awful lot of feedback and requests for new features and changes (in an often very unorganised way), I'm sure the development team have their own opinions on this also... How do you decide which ones to take forward vs. which ones to shelve?

I guess from a technical point of view you have to make some kind of assessment as to how many man hours such a change will consume and if it will incur a performance hit after some kind of shortlist..

and thanks for the earlier answers btw

[Meme Turtle]

Egosoft obviously receives an awful lot of feedback and requests for new features and changes (in an often very unorganised way), I'm sure the development team have their own opinions on this also... How do you decide which ones to take forward vs. which ones to shelve?

I guess from a technical point of view you have to make some kind of assessment as to how many man hours such a change will consume and if it will incur a performance hit after some kind of shortlist..

and thanks for the earlier answers btw

I would like to hear about Egosoft's roadmap or development plan as well. It'd be a nice idea to understand how players can influence developers' decisions. Some of the new features in 3.0(crafting for example) are though useful and add to the gameplay level are not as important as some really critical fixes/features.

What I really want to see is an organized sticky thread with priority sorted list of new features/extensions based on players' feedback.

[BigBANGtheory]

Egosoft obviously receives an awful lot of feedback and requests for new features and changes (in an often very unorganised way), I'm sure the development team have their own opinions on this also... How do you decide which ones to take forward vs. which ones to shelve?

I guess from a technical point of view you have to make some kind of assessment as to how many man hours such a change will consume and if it will incur a performance hit after some kind of shortlist..

and thanks for the earlier answers btw

I would like to hear about Egosoft's roadmap or development plan as well. It'd be a nice idea to understand how players can influence developers' decisions. Some of the new features in 3.0(crafting for example) are though useful and add to the gameplay level are not as important as some really critical fixes/features.

What I really want to see is an organized sticky thread with priority sorted list of new features/extensions based on players' feedback.

I'd like that too (as would most I'm sure) though in fairness I think the purpose of this thread wasn't to go there so I was asking more about the how i.e. the decision making process rather than the what if that makes sense. Somethings are tough to do, others totally impractical someone somewhere along the way has to say what is technically achievable with the game engine with a good degree of success.

There is no point for example the community raging on about a feature lets say planet landing for arguments sake if it is wholly impractical with the current engine or would require 1000's of development hrs to achieve.

btw on a separate matter there is quite a bit of press emerging on dx12 no doubt ahead of the Win10 release, looks like it has some functions potentially very useful to a game like XR (more so than dx11) with the user-mode GPU driver being multi-threaded and taking on more of the work:
http://blogs.msdn.com/b/directx/archive ... tx-12.aspx