[Official] Technical Engine Development faq / blog

General discussions about X Rebirth.

Moderator: Moderators for English X Forum

Post Reply
timon37
EGOSOFT
EGOSOFT
Posts: 369
Joined: Fri, 14. Dec 12, 12:02
x4

[Official] Technical Engine Development faq / blog

Post by timon37 » Wed, 17. Dec 14, 13:14

Welcome, we'll be sharing some of the technical nitty-gritty stuff we're working on here.
So don't enter unless you have the stomach for it;)

Feel free to ask anything you might be interested in!
We've got very limited esper skills so we don't know what to talk about unless you ask (also I'm lazy).

Since I'm the lucky one responsible for the linux/mac port, I'll probably be discussing that along with opengl/dx the most. But anything is welcome, maybe? We'll see how things progress.

Topics covered so far:

The thread will probably get chaotic fast but that's fine... gives you the taste of life as a game dev...
Last edited by timon37 on Wed, 17. Dec 14, 13:54, edited 1 time in total.

User avatar
wysiwyg
Posts: 584
Joined: Thu, 26. Feb 04, 01:08
x4

Post by wysiwyg » Wed, 17. Dec 14, 13:30

Don't all post at once now - please form an orderly queue :D

Hi timon37 - I know Stefan has already said recently that there won't be any modding support for the UI anytime soon but are you able to give us a flavour of what might be on the cards if/when you do share anything with us?

Also just out of interest - Is the decision to script the UI in Lua primarily because of the Anark system used and is there likely to be a move from the current xml format used for MD/AI scripting over to a Lua (or other slightly more intuitive) scripting language?

Great work so far in getting XR to where it is now and also nice to see more dev interaction following on from Lino's efforts.

Thanks
Wysi :)

CBJ
EGOSOFT
EGOSOFT
Posts: 37464
Joined: Tue, 29. Apr 03, 00:56
x4

Post by CBJ » Wed, 17. Dec 14, 13:31

I'll kick things off with a discussion started by forum users interested in the Linux/Mac port in another thread:
karlwatson wrote:Is your AZDO renderer transition for the game in general or just specifically for the ports? I did look it up @Nvidia but I'm still unsure what it means regards the goals for Rebirth.
etonbears wrote:I think Timon is talking about an OpenGL ADZO renderer for Linux and maybe the Mac, if Apple pull their finger out on modern OpenGL support.

OpenGL is a very long-standing collection of APIs that allow you to write many different styles of renderer.

If you use the oldest OpenGL APIs, you get very little direct access to the graphics hardware, as the driver does a lot of "standard" graphics work for you. The advantage of these old APIs is that you need write less code yourself, but the disadvantage is that you get less control/flexibility over exactly how the code operates. The OpenGL driver can introduce a very large overhead as it carefully checks and validates data, and doesn't really understand your application's data layout, so it has a hard time optimising performance.

The newer OpenGL APIs, particularly since version 3.x have gradually moved to exposing the hardware more directly, so you write shaders that compile and run on the graphics card and access resources on the graphics card through manipulating memory buffers. If you write a renderer using only the newest OpenGL APIs, the OpenGL driver does much less work ( low overhead ), expecting you to manage and marshal your data into the form that will run shaders at their optimal speed.

This trend in OpenGL coding has been called AZDO - Approaching Zero Driver Overhead. However OpenGL is not alone; all graphics APIs are moving in that direction at the moment. D3D12 will also aim to lower Windows' graphics overhead, Apple will introduce their Metal API, and AMD's hardware-specific low-overhead Mantle drivers are already in use in several Windows games.

It is unlikely that the Windows code for Rebirth will be converted to use OpenGL, since the company is small, and all the low-level coders except Timon are probably Microsoft-only, or Microsoft-mainly. It may also become more difficult to produce Apple OpenGL code if they choose to focus on their own API, so I expect Timon is likely to focus on getting something working well under Linux.

It doesn't mean it is impossible for Timon to write a cross-platform, OpenGL based version of Rebirth, but it's going to be a lot of work to get it working well just in Linux.

Personally, I think a small company like Egosoft would be better off developing a platform-neutral codebase as much as possible as a hedge against future platform-limiting behaviour, and simply because it would be overall more time-efficient, as well as find more bugs, more quickly. But that is not where they have come from, and not where the bulk of their expertise probably lies; so the inertia for this is very great...

...unless, of course, Timon manages a miracle port that runs on all platforms, and faster than the original :)

linolafett
EGOSOFT
EGOSOFT
Posts: 2328
Joined: Mon, 26. Mar 12, 14:57
x4

Post by linolafett » Wed, 17. Dec 14, 13:33

We may merge this thread with the behind the scenes thread, as we have an overview about all covered topics (with categories) in the first posting there. Multiple threads may confuse users and spread the nice to know postings in the forums.

User avatar
edqe
Posts: 55
Joined: Tue, 19. Nov 13, 16:30

Post by edqe » Wed, 17. Dec 14, 13:35

Since I'm the lucky one responsible for the linux/mac port, I'll probably be discussing that along with opengl/dx the most.
This is going to be extremely interesting thread. Thanks for keeping us up to date and sharing technical information.

Can't wait to be able to play XR on Linux.

CBJ
EGOSOFT
EGOSOFT
Posts: 37464
Joined: Tue, 29. Apr 03, 00:56
x4

Post by CBJ » Wed, 17. Dec 14, 14:10

wysiwyg wrote:...is there likely to be a move from the current xml format used for MD/AI scripting over to a Lua (or other slightly more intuitive) scripting language?
I can answer this part for you: no.

Lua is a general purpose language and, while that gives it the potential to do a wide variety of things and perhaps be more accessible in some ways because it is "standard", it also makes it much less closely aligned to the task in hand. The point of using domain-specific languages for our scripting is to keep the syntax closely bound to the language's function. This allows mission authors to focus on the mission, not the language, and in our experience makes it easier for non-programmers to become mission authors relatively quickly and easily.

With XML we can set this up and extend it very easily, and also provide context-specific documentation via the schema, which authors can than make use of in any decent XML editor. There are other benefits too, such as the ability to perform script patching when we provide updates to the game, something that would be much harder to do with a general purpose language. While in principle we could have created a domain-specific language with all these features but without using XML, in practice XML and the tools that are readily available to work with it reduce the amount of work required on our side significantly, and as a small team that is an advantage we simply cannot afford to ignore.

It's probably also worth mentioning that there are a number of misconceptions about our use of XML that have circulated in the past, which may be affecting people's opinion of it. One of these is that XML is inherently "slow". This is predicated on the false assumption that we are constantly parsing and reading the script XML as the script runs. Scripts are loaded and parsed once, after which their functionality is implemented by a series of functions and classes that are, again, closely aligned to the features of the game. This makes for good run-time performance, and the small load-time overhead of reading the source XML is minimal, especially when compared to all the other things that the game has to do in order to start up.

User avatar
Roger L.S. Griffiths
Posts: 9913
Joined: Fri, 12. Mar 04, 20:47
x4

Post by Roger L.S. Griffiths » Wed, 17. Dec 14, 14:24

@CBJ: Currently, AFAIK there appears to be some missing features from the baseline X-Rebirth XML Scripting Language that some of us may have been looking for (e.g. ability to set-up keyboard shortcuts, and affect the UI in certain non-mission related ways).

Are there plans to expand the scripting language to include such features? If so, is there a target release date for these features.

If I am mistaken and these features are in already, is there any documentation available?
Roger (aka [SRK] Mr^Rabbit)

"Understanding is a three edged sword... your side, their side... and the Truth!" - J.J. Sheriden, Babylon 5 S4E6 T28:55

"May god stand between you and harm in all the dark places you must walk." - Ancient Egyption Proverb

"When eating an elephant take one bite at a time" - Creighton Abrams

CBJ
EGOSOFT
EGOSOFT
Posts: 37464
Joined: Tue, 29. Apr 03, 00:56
x4

Post by CBJ » Wed, 17. Dec 14, 14:32

Roger L.S. Griffiths wrote:@CBJ: Currently, AFAIK there appears to be some missing features from the baseline X-Rebirth XML Scripting Language that some of us may have been looking for (e.g. ability to set-up keyboard shortcuts, and affect the UI in other ways).

Are there plans to expand the scripting language to include such features? If so, is there a target release date for these features.
I'm not sure what you're getting at here. The UI is, as wysiwyg mentioned, implemented primarily in Lua rather than XML, largely because it requires more of the features of a general purpose language than scripting does, and of course also because it provides easy Anark integration. While it would have been nice to have a domain-specific UI language, the balance of benefits in this case pushed us in the other direction.

The corollary to this, of course, is that you would not therefore be influencing the UI or controls directly from any of the XML domain-specific scripting languages we have, because those things are not part of the domains covered by those languages. But maybe I have missed something you should be able to do from, say, a mission or an AI script; if you have more specific examples that are part of those domains then perhaps I could give you a more specific answer. :)

User avatar
BigBANGtheory
Posts: 2469
Joined: Sun, 23. Oct 05, 12:13
x4

Post by BigBANGtheory » Wed, 17. Dec 14, 15:52

1) Slightly controversial so I'll choose my words carefully and respectfully...

We experienced performance and scaling issues on high end hardware at release, clearly some problems have been identified and fixed because things have improved well done... So what did you find, what was causing some of the performance problems beyond hardware drivers etc I am talking about the game specifically?

2) I've always been a fan of the visual results from parallax and parallax occlusion mapping, games like Stalker and Crysis used it to great effect. Is there a reason why that wouldn't work well in XR I think it can be done with Dx9?

3) Lino I think mentioned in another post recently that having a global AI was an issue for XR, what's the story there?

4) Without naming names 8), how many of your colleagues have played or have seen being played Homeworld2?

krapht
Posts: 108
Joined: Sat, 11. Oct 08, 01:16
x4

Post by krapht » Wed, 17. Dec 14, 16:03

As someone coming from a more traditional branch of "high-level" development (think enterprise software), and my experience with gaming programming extends to Minecraft modding I got a couple of questions

I have always thought that the UI felt very tightly coupled to the underlying game logic. Almost as the if some of the logic was executed in the UI layer. Is this the case?
How SOLID is the core game? (is it even object oriented? )
Is it easy to work with or is it starting to be uncomfortably complicated when you want to change / add things?
Do you do unit testing?
How much of readability / maintainability do you have to give up for performance?

Many questions just off the top of my head =)
(Sidenote: I love the recent quick iterations of the beta)

Sparky Sparkycorp
Moderator (English)
Moderator (English)
Posts: 7482
Joined: Tue, 30. Mar 04, 12:28
x4

Post by Sparky Sparkycorp » Wed, 17. Dec 14, 16:12

Hi timon,

Thanks for the invitation :)

Gun-wise, we currently have either front-mounted small weapons on S and M ships only or, with the exception of the Sucellus, turret-mounted large guns on L and XL ships only.

Are their any plans being worked on to develop turret-mounted small guns? If not, any there any particular limitations needed to be overcome before such a thing could arise?

A pea-shooter on the back of M craft would be a dream come true. Ideally with the option of adding one to the Skunk in exchange for a drawback (e.g. loss of weapon slots or increased overheating).

Artean
Posts: 691
Joined: Tue, 14. Feb 06, 18:41
x4

Post by Artean » Wed, 17. Dec 14, 16:18

I'm a total n00b, but let me try a lill Q anyways:

Would moving XR to DX11 potentially improve performance and graphic quality?
"In the beginning the Universe was created. This has made a lot of people very angry and has been widely regarded as a bad move." - D.N.A

rboerdijk
EGOSOFT
EGOSOFT
Posts: 1212
Joined: Fri, 11. Jan 13, 15:26
x4

Post by rboerdijk » Wed, 17. Dec 14, 16:49

BigBANGtheory wrote: 4) Without naming names 8), how many of your colleagues have played or have seen being played Homeworld2?
Since you explicitely requested it, I won't tell my name, but yes I did play both of the homeworld games.

Loved the game - especially capturing enemy ships. I did that every level and my fleet-size steadily increased. What I didn't know at the time was the the enemy-strength depends on your fleet-strength at the start of the level, so I ended up in the last level being absolutely unplayable due to the crazy amounts of ships. When I got a faster machine a year or so later, I actually got back to it and finished the game.

Also played the 2nd part but I got stuck quite early, if I remember correctly ships were being infected and I didn't get past that level.

timon37
EGOSOFT
EGOSOFT
Posts: 369
Joined: Fri, 14. Dec 12, 12:02
x4

Post by timon37 » Wed, 17. Dec 14, 17:02

I'll make an update about dx9/11/12/opengl linux/mac a bit later.
I'll also be answering a bit selectively about what I know, since others will answer other stuff.
Order may also vary depending on how much I have to think about the question;p

1) Regarding performance:

It's an endless topic, and honestly it probably won't be very constructive since it comes down to specific things in the code:(
But in general: the single threaded performance of the cpu was and still is the most common/significant bottleneck, and the difference between a high-end and a low-end PC in this regard is fairly minor.
I'll maybe try to add a bit more on this topic later, it's hard to say something really meaningful on such short notice;p
BigBANGtheory wrote:2) I've always been a fan of the visual results from parallax and parallax occlusion mapping, games like Stalker and Crysis used it to great effect. Is there a reason why that wouldn't work well in XR I think it can be done with Dx9?
Agreed it's awesome. I can't be sure without actually seeing but I can think of a few reasons why it wouldn't work great:
- first of all you have to be fairly close to the surface,
- the angle at which you see it has to be changing to see the effect well,
- the surface indentation can't be too deep or too shallow, otherwise it'd look distorted, or it wouldn't provide enough benefit over simple normal mapping,
- it's heavy so you need to do it only for a few things close by, which makes "LOD" granularity much more important which might be a problem for us,
- it works best for large flat surfaces e.g. paved road, brick walls

Basically we probably don't have enough good use-cases for it other than interiors, which makes it lower on the bang-for-the-buck list.

CBJ
EGOSOFT
EGOSOFT
Posts: 37464
Joined: Tue, 29. Apr 03, 00:56
x4

Post by CBJ » Wed, 17. Dec 14, 17:04

BigBANGtheory wrote:3) Lino I think mentioned in another post recently that having a global AI was an issue for XR, what's the story there?
Perhaps there was a slight misunderstanding. There is no "issue" with having a "global AI" to implement functionality along the lines of the "GOD module" in X3; it's just that we don't have one at the moment. Adding one would be non-trivial for various reasons but not an "issue" in principle. If or when that might happen is not something I can answer at the moment.

Post Reply

Return to “X Rebirth Universe”