[Official] Technical Engine Development faq / blog

General discussions about X Rebirth.

Moderator: Moderators for English X Forum

Post Reply
timon37
EGOSOFT
EGOSOFT
Posts: 485
Joined: Fri, 14. Dec 12, 11:02
x4

[Official] Technical Engine Development faq / blog

Post by timon37 » Wed, 17. Dec 14, 12:14

Welcome, we'll be sharing some of the technical nitty-gritty stuff we're working on here.
So don't enter unless you have the stomach for it;)

Feel free to ask anything you might be interested in!
We've got very limited esper skills so we don't know what to talk about unless you ask (also I'm lazy).

Since I'm the lucky one responsible for the linux/mac port, I'll probably be discussing that along with opengl/dx the most. But anything is welcome, maybe? We'll see how things progress.

Topics covered so far:

The thread will probably get chaotic fast but that's fine... gives you the taste of life as a game dev...
Last edited by timon37 on Wed, 17. Dec 14, 12:54, edited 1 time in total.

User avatar
wysiwyg
Posts: 585
Joined: Thu, 26. Feb 04, 00:08
x4

Post by wysiwyg » Wed, 17. Dec 14, 12:30

Don't all post at once now - please form an orderly queue :D

Hi timon37 - I know Stefan has already said recently that there won't be any modding support for the UI anytime soon but are you able to give us a flavour of what might be on the cards if/when you do share anything with us?

Also just out of interest - Is the decision to script the UI in Lua primarily because of the Anark system used and is there likely to be a move from the current xml format used for MD/AI scripting over to a Lua (or other slightly more intuitive) scripting language?

Great work so far in getting XR to where it is now and also nice to see more dev interaction following on from Lino's efforts.

Thanks
Wysi :)

CBJ
EGOSOFT
EGOSOFT
Posts: 51740
Joined: Tue, 29. Apr 03, 00:56
x4

Post by CBJ » Wed, 17. Dec 14, 12:31

I'll kick things off with a discussion started by forum users interested in the Linux/Mac port in another thread:
karlwatson wrote:Is your AZDO renderer transition for the game in general or just specifically for the ports? I did look it up @Nvidia but I'm still unsure what it means regards the goals for Rebirth.
etonbears wrote:I think Timon is talking about an OpenGL ADZO renderer for Linux and maybe the Mac, if Apple pull their finger out on modern OpenGL support.

OpenGL is a very long-standing collection of APIs that allow you to write many different styles of renderer.

If you use the oldest OpenGL APIs, you get very little direct access to the graphics hardware, as the driver does a lot of "standard" graphics work for you. The advantage of these old APIs is that you need write less code yourself, but the disadvantage is that you get less control/flexibility over exactly how the code operates. The OpenGL driver can introduce a very large overhead as it carefully checks and validates data, and doesn't really understand your application's data layout, so it has a hard time optimising performance.

The newer OpenGL APIs, particularly since version 3.x have gradually moved to exposing the hardware more directly, so you write shaders that compile and run on the graphics card and access resources on the graphics card through manipulating memory buffers. If you write a renderer using only the newest OpenGL APIs, the OpenGL driver does much less work ( low overhead ), expecting you to manage and marshal your data into the form that will run shaders at their optimal speed.

This trend in OpenGL coding has been called AZDO - Approaching Zero Driver Overhead. However OpenGL is not alone; all graphics APIs are moving in that direction at the moment. D3D12 will also aim to lower Windows' graphics overhead, Apple will introduce their Metal API, and AMD's hardware-specific low-overhead Mantle drivers are already in use in several Windows games.

It is unlikely that the Windows code for Rebirth will be converted to use OpenGL, since the company is small, and all the low-level coders except Timon are probably Microsoft-only, or Microsoft-mainly. It may also become more difficult to produce Apple OpenGL code if they choose to focus on their own API, so I expect Timon is likely to focus on getting something working well under Linux.

It doesn't mean it is impossible for Timon to write a cross-platform, OpenGL based version of Rebirth, but it's going to be a lot of work to get it working well just in Linux.

Personally, I think a small company like Egosoft would be better off developing a platform-neutral codebase as much as possible as a hedge against future platform-limiting behaviour, and simply because it would be overall more time-efficient, as well as find more bugs, more quickly. But that is not where they have come from, and not where the bulk of their expertise probably lies; so the inertia for this is very great...

...unless, of course, Timon manages a miracle port that runs on all platforms, and faster than the original :)

linolafett
EGOSOFT
EGOSOFT
Posts: 3321
Joined: Mon, 26. Mar 12, 14:57
x4

Post by linolafett » Wed, 17. Dec 14, 12:33

We may merge this thread with the behind the scenes thread, as we have an overview about all covered topics (with categories) in the first posting there. Multiple threads may confuse users and spread the nice to know postings in the forums.

User avatar
edqe
Posts: 73
Joined: Tue, 19. Nov 13, 15:30

Post by edqe » Wed, 17. Dec 14, 12:35

Since I'm the lucky one responsible for the linux/mac port, I'll probably be discussing that along with opengl/dx the most.
This is going to be extremely interesting thread. Thanks for keeping us up to date and sharing technical information.

Can't wait to be able to play XR on Linux.

CBJ
EGOSOFT
EGOSOFT
Posts: 51740
Joined: Tue, 29. Apr 03, 00:56
x4

Post by CBJ » Wed, 17. Dec 14, 13:10

wysiwyg wrote:...is there likely to be a move from the current xml format used for MD/AI scripting over to a Lua (or other slightly more intuitive) scripting language?
I can answer this part for you: no.

Lua is a general purpose language and, while that gives it the potential to do a wide variety of things and perhaps be more accessible in some ways because it is "standard", it also makes it much less closely aligned to the task in hand. The point of using domain-specific languages for our scripting is to keep the syntax closely bound to the language's function. This allows mission authors to focus on the mission, not the language, and in our experience makes it easier for non-programmers to become mission authors relatively quickly and easily.

With XML we can set this up and extend it very easily, and also provide context-specific documentation via the schema, which authors can than make use of in any decent XML editor. There are other benefits too, such as the ability to perform script patching when we provide updates to the game, something that would be much harder to do with a general purpose language. While in principle we could have created a domain-specific language with all these features but without using XML, in practice XML and the tools that are readily available to work with it reduce the amount of work required on our side significantly, and as a small team that is an advantage we simply cannot afford to ignore.

It's probably also worth mentioning that there are a number of misconceptions about our use of XML that have circulated in the past, which may be affecting people's opinion of it. One of these is that XML is inherently "slow". This is predicated on the false assumption that we are constantly parsing and reading the script XML as the script runs. Scripts are loaded and parsed once, after which their functionality is implemented by a series of functions and classes that are, again, closely aligned to the features of the game. This makes for good run-time performance, and the small load-time overhead of reading the source XML is minimal, especially when compared to all the other things that the game has to do in order to start up.

User avatar
Sam L.R. Griffiths
Posts: 10522
Joined: Fri, 12. Mar 04, 19:47
x4

Post by Sam L.R. Griffiths » Wed, 17. Dec 14, 13:24

@CBJ: Currently, AFAIK there appears to be some missing features from the baseline X-Rebirth XML Scripting Language that some of us may have been looking for (e.g. ability to set-up keyboard shortcuts, and affect the UI in certain non-mission related ways).

Are there plans to expand the scripting language to include such features? If so, is there a target release date for these features.

If I am mistaken and these features are in already, is there any documentation available?
Lenna (aka [SRK] The_Rabbit)

"Understanding is a three edged sword... your side, their side... and the Truth!" - J.J. Sheriden, Babylon 5 S4E6 T28:55

"May god stand between you and harm in all the dark places you must walk." - Ancient Egyption Proverb

"When eating an elephant take one bite at a time" - Creighton Abrams

CBJ
EGOSOFT
EGOSOFT
Posts: 51740
Joined: Tue, 29. Apr 03, 00:56
x4

Post by CBJ » Wed, 17. Dec 14, 13:32

Roger L.S. Griffiths wrote:@CBJ: Currently, AFAIK there appears to be some missing features from the baseline X-Rebirth XML Scripting Language that some of us may have been looking for (e.g. ability to set-up keyboard shortcuts, and affect the UI in other ways).

Are there plans to expand the scripting language to include such features? If so, is there a target release date for these features.
I'm not sure what you're getting at here. The UI is, as wysiwyg mentioned, implemented primarily in Lua rather than XML, largely because it requires more of the features of a general purpose language than scripting does, and of course also because it provides easy Anark integration. While it would have been nice to have a domain-specific UI language, the balance of benefits in this case pushed us in the other direction.

The corollary to this, of course, is that you would not therefore be influencing the UI or controls directly from any of the XML domain-specific scripting languages we have, because those things are not part of the domains covered by those languages. But maybe I have missed something you should be able to do from, say, a mission or an AI script; if you have more specific examples that are part of those domains then perhaps I could give you a more specific answer. :)

User avatar
BigBANGtheory
Posts: 3167
Joined: Sun, 23. Oct 05, 12:13
x4

Post by BigBANGtheory » Wed, 17. Dec 14, 14:52

1) Slightly controversial so I'll choose my words carefully and respectfully...

We experienced performance and scaling issues on high end hardware at release, clearly some problems have been identified and fixed because things have improved well done... So what did you find, what was causing some of the performance problems beyond hardware drivers etc I am talking about the game specifically?

2) I've always been a fan of the visual results from parallax and parallax occlusion mapping, games like Stalker and Crysis used it to great effect. Is there a reason why that wouldn't work well in XR I think it can be done with Dx9?

3) Lino I think mentioned in another post recently that having a global AI was an issue for XR, what's the story there?

4) Without naming names 8), how many of your colleagues have played or have seen being played Homeworld2?

krapht
Posts: 113
Joined: Sat, 11. Oct 08, 01:16
x4

Post by krapht » Wed, 17. Dec 14, 15:03

As someone coming from a more traditional branch of "high-level" development (think enterprise software), and my experience with gaming programming extends to Minecraft modding I got a couple of questions

I have always thought that the UI felt very tightly coupled to the underlying game logic. Almost as the if some of the logic was executed in the UI layer. Is this the case?
How SOLID is the core game? (is it even object oriented? )
Is it easy to work with or is it starting to be uncomfortably complicated when you want to change / add things?
Do you do unit testing?
How much of readability / maintainability do you have to give up for performance?

Many questions just off the top of my head =)
(Sidenote: I love the recent quick iterations of the beta)

Sparky Sparkycorp
Moderator (English)
Moderator (English)
Posts: 8074
Joined: Tue, 30. Mar 04, 12:28
x4

Post by Sparky Sparkycorp » Wed, 17. Dec 14, 15:12

Hi timon,

Thanks for the invitation :)

Gun-wise, we currently have either front-mounted small weapons on S and M ships only or, with the exception of the Sucellus, turret-mounted large guns on L and XL ships only.

Are their any plans being worked on to develop turret-mounted small guns? If not, any there any particular limitations needed to be overcome before such a thing could arise?

A pea-shooter on the back of M craft would be a dream come true. Ideally with the option of adding one to the Skunk in exchange for a drawback (e.g. loss of weapon slots or increased overheating).

Artean
Posts: 1094
Joined: Tue, 14. Feb 06, 17:41
x4

Post by Artean » Wed, 17. Dec 14, 15:18

I'm a total n00b, but let me try a lill Q anyways:

Would moving XR to DX11 potentially improve performance and graphic quality?
"In the beginning the Universe was created. This has made a lot of people very angry and has been widely regarded as a bad move." - D.N.A

rboerdijk
EGOSOFT
EGOSOFT
Posts: 1672
Joined: Fri, 11. Jan 13, 14:26
x4

Post by rboerdijk » Wed, 17. Dec 14, 15:49

BigBANGtheory wrote: 4) Without naming names 8), how many of your colleagues have played or have seen being played Homeworld2?
Since you explicitely requested it, I won't tell my name, but yes I did play both of the homeworld games.

Loved the game - especially capturing enemy ships. I did that every level and my fleet-size steadily increased. What I didn't know at the time was the the enemy-strength depends on your fleet-strength at the start of the level, so I ended up in the last level being absolutely unplayable due to the crazy amounts of ships. When I got a faster machine a year or so later, I actually got back to it and finished the game.

Also played the 2nd part but I got stuck quite early, if I remember correctly ships were being infected and I didn't get past that level.

timon37
EGOSOFT
EGOSOFT
Posts: 485
Joined: Fri, 14. Dec 12, 11:02
x4

Post by timon37 » Wed, 17. Dec 14, 16:02

I'll make an update about dx9/11/12/opengl linux/mac a bit later.
I'll also be answering a bit selectively about what I know, since others will answer other stuff.
Order may also vary depending on how much I have to think about the question;p

1) Regarding performance:

It's an endless topic, and honestly it probably won't be very constructive since it comes down to specific things in the code:(
But in general: the single threaded performance of the cpu was and still is the most common/significant bottleneck, and the difference between a high-end and a low-end PC in this regard is fairly minor.
I'll maybe try to add a bit more on this topic later, it's hard to say something really meaningful on such short notice;p
BigBANGtheory wrote:2) I've always been a fan of the visual results from parallax and parallax occlusion mapping, games like Stalker and Crysis used it to great effect. Is there a reason why that wouldn't work well in XR I think it can be done with Dx9?
Agreed it's awesome. I can't be sure without actually seeing but I can think of a few reasons why it wouldn't work great:
- first of all you have to be fairly close to the surface,
- the angle at which you see it has to be changing to see the effect well,
- the surface indentation can't be too deep or too shallow, otherwise it'd look distorted, or it wouldn't provide enough benefit over simple normal mapping,
- it's heavy so you need to do it only for a few things close by, which makes "LOD" granularity much more important which might be a problem for us,
- it works best for large flat surfaces e.g. paved road, brick walls

Basically we probably don't have enough good use-cases for it other than interiors, which makes it lower on the bang-for-the-buck list.

CBJ
EGOSOFT
EGOSOFT
Posts: 51740
Joined: Tue, 29. Apr 03, 00:56
x4

Post by CBJ » Wed, 17. Dec 14, 16:04

BigBANGtheory wrote:3) Lino I think mentioned in another post recently that having a global AI was an issue for XR, what's the story there?
Perhaps there was a slight misunderstanding. There is no "issue" with having a "global AI" to implement functionality along the lines of the "GOD module" in X3; it's just that we don't have one at the moment. Adding one would be non-trivial for various reasons but not an "issue" in principle. If or when that might happen is not something I can answer at the moment.

User avatar
Tamina
Moderator (Deutsch)
Moderator (Deutsch)
Posts: 4543
Joined: Sun, 26. Jan 14, 09:56

Post by Tamina » Wed, 17. Dec 14, 16:32

timon37 wrote:I'll make an update about dx9/11/12/opengl linux/mac a bit later.
I would like to ask, since XR has gone 64bit which propably means that 99% of the users are theoretically at minimum DX10 standard (-how am I kidding, that would be Vista: minimum DX11 standard) would XR benefit being ported to newer API versions?

I tried to look it up, but there are many different sources which all say that Dx11 has less performance then Dx9 and vice versa.

Although all state that Dx11 is way more beautiful
(don't know what D3D actually covers but I doubt it gets 'automatically' more beautiful by changing the D3D version, or does it?)
the benachmarks about performance are really mixed, while Microsoft states that Dx11 should be faster.

So would XR benefit from a higher DX-Version and is this a goal for the future of XR?
(Again I am assuming that all users (I guess Steam gives Egosoft a chart with the computer specs of their players?) have to be beyond Dx9)

Code: Select all

Und wenn ein Forenbösewicht, was Ungezogenes spricht, dann hol' ich meinen Kaktus und der sticht sticht sticht.
  /l、 
゙(゚、 。 7 
 l、゙ ~ヽ   / 
 じしf_, )ノ 

rboerdijk
EGOSOFT
EGOSOFT
Posts: 1672
Joined: Fri, 11. Jan 13, 14:26
x4

Post by rboerdijk » Wed, 17. Dec 14, 16:48

BigBANGtheory wrote:1) Slightly controversial so I'll choose my words carefully and respectfully...

We experienced performance and scaling issues on high end hardware at release, clearly some problems have been identified and fixed because things have improved well done... So what did you find, what was causing some of the performance problems beyond hardware drivers etc I am talking about the game specifically?
We performed quite a few of tasks, mostly to
1. smartly "skip" work
2. fix threading issues ( threads blocking each other ).
3. figure out why something fast wasn't working as intended.
4. change the scenario / level.
5. new features "stealing" performance.

An example of #1 was the cpu-render side, trying to predict which objects to discard - since they wouldn't end up on the screen anyway. For example we had culling code for asteroids which worked quite well - but at one point we ended up testing 20k-30k asteroids individually each frame. Since we have a nice grid with "big groups of asteroids" this could be easily improved, which dropped the number of tests needed by roughly 70%.

An example of #2 was in our way of threading, which ended up causing many thread-switches when loading directX textures/meshes (this was fixed quite early). We had some problems where worker-threads could end up blocking our main-thread.

An example of #3 was in asteroid rendering, more specifically instancing. We can't instance models with an alpha - but at one point some asteroid-models got that. Basically resulting in all asteroids being drawn individually with thousands of draw-calls - not good.

An example of #4 was certain regions with ice-asteroids, where the level-setup was such that there were way too many - even after all other optimizations. We ended up putting the counts down to reasonable levels.

Same for "choke-points" where too many ships ended up.
We used several approaches to spread the ships out more in these cases.
- from the AI-side trying to avoid ships going somewhere where "too much" activity is
- having more locations for certain asteroids-fields.
- add certain stations so ships will spread more nicely between them.

An example of #5 were lensflares. They could end up doing many "long-distance" raycast ( > 50km). We optimized those by doing a short 1-km raycasts (factors faster) because roughly 1/2 of those flares would be blocked by its own ship.

We did way more optimizations, but I hope this gives some insight.
Last edited by rboerdijk on Wed, 17. Dec 14, 16:53, edited 1 time in total.

CBJ
EGOSOFT
EGOSOFT
Posts: 51740
Joined: Tue, 29. Apr 03, 00:56
x4

Post by CBJ » Wed, 17. Dec 14, 16:53

Just to add to that, performance improvements in several updates were obtained by improving the LOD setup of some of the graphical assets.

User avatar
ezra-r
Posts: 3420
Joined: Fri, 14. Oct 05, 21:04
x4

About memory usage

Post by ezra-r » Wed, 17. Dec 14, 17:03

note that I know nothing about programming but the simplest coding, even less how to make use of the memory available in the system.

But question is about memory usage:

Since the move to 64 bits binary more than the usual 2GB of ram can be used and we clearly can see that, when alt+tabbing game most time uses 3.2 gigabttes sometimes I recall it even using near 4 gigabytes.

Sometimes when I turn around things get "re-painted" and sometimes when I use highways or in other situations I notice hiccups, which I assume it is the game loading resources.

In my case I have 16gb of ram and a video card with 3gb of ram.

* Can you guys "make" the game use more memory when available to pre-cache stuff that will be used later to avoid hiccups at most times or this is unavoidable?

Like, why don't I see the game using 6 gigabytes when I clearly have them spare.

PS: hope I made some sense.
PS2: Thanks for this thread.

wwdragon
Posts: 3746
Joined: Mon, 1. Oct 07, 02:18
x4

Re: About memory usage

Post by wwdragon » Wed, 17. Dec 14, 18:02

ezra-r wrote:* Can you guys "make" the game use more memory when available to pre-cache stuff that will be used later to avoid hiccups at most times or this is unavoidable?

Like, why don't I see the game using 6 gigabytes when I clearly have them spare.
I like that idea. 8)
I also have 16 gig on my machine; 1 gig or so for os+ proggies and that leaves 14 gig that it could use!
Maybe a graphics options checkbox for MOOOOAAARR ram usage?
Editing posts since long before I remember.

Post Reply

Return to “X Rebirth Universe”