[TOOL] XPL (LuaJIT) files decompiler

The place to discuss scripting and game modifications for X Rebirth.

Moderators: Moderators for English X Forum, Scripting / Modding Moderators

Night Nord
Posts: 1002
Joined: Fri, 13. Jan 12, 19:09
x4

[TOOL] XPL (LuaJIT) files decompiler

Post by Night Nord » Sun, 26. Jan 14, 15:49

Tool: LuaJIT raw-bytecode decompiler. Can decompile .xpl files into lua sources
Version: 0.1.3
URL: https://github.com/NightNord/ljd/

Requirements: Python version 3+: http://www.python.org/download/releases/3.3.3/
Usage:

Code: Select all

python main.py path/to/file.xpl > source.lua
Screenshots: A simple "Hello, World!" modification of the Event Monitor.xpl script

[ external image ]

Current status:
All files in ui/core/Lua and ui/addons could be decompiled into compilable lua sources. With all UI recompiled game works without any obvious bugs.

Help needed:
  • Hunting decompiler bugs. Please create an issue at GitHub page if you'll find any nonsense in files that you believe aren't by EgoSoft (i.e. there is no such bugs with original files).
  • Engine research: we need to find a way to make UI mods. Currently I need to pack modified scripts into 08.cat/dat package. But I think it should be possible to modify a messagemap.xml via the ordinary mod interface.
  • Engine research: to make something useful (a button to rename ships, for instance) we need to find all accessible C functions and their meaning. I think there should be some way to spawn an MD task or something. At least there is a way to start a trading run via specific function call.

    Update: It seems that this is impossible. All info available for UI is read-only. But maybe it is possible to make some weird hack, who knows.
  • Engine research: while most of the UI is made dynamically by Lua code, some visual portions are taken from presentations files in some proprietary format. How much are they needed and what can we do without them is an interesting topic for a research.
Question to EgoSoft representatives: May we post the sources at a public repository (read: github) for ease of an access and modification?
Last edited by Night Nord on Wed, 29. Jan 14, 23:21, edited 4 times in total.

User avatar
YorrickVander
Posts: 2689
Joined: Tue, 29. Oct 13, 21:59
x4

Post by YorrickVander » Sun, 26. Jan 14, 16:14

Good luck Nord :)
X Rebirth - A Sirius Cybernetics Corporation Product

Split irritate visiting pilot with strange vocal patterns.

Ginger470
Posts: 158
Joined: Sun, 20. Jun 04, 03:24
x2

Post by Ginger470 » Mon, 27. Jan 14, 06:02

Now this is exciting :)

Night Nord
Posts: 1002
Joined: Fri, 13. Jan 12, 19:09
x4

Post by Night Nord » Mon, 27. Jan 14, 06:43

It seems that I forgot the most important link "Download". Fixed now.

User avatar
BigBANGtheory
Posts: 3167
Joined: Sun, 23. Oct 05, 12:13
x4

Post by BigBANGtheory » Mon, 27. Jan 14, 14:44

LuaDec is a decompiler it might be a little out of date but thought it worth mentioning:

http://luadec.luaforge.net/

Night Nord
Posts: 1002
Joined: Fri, 13. Jan 12, 19:09
x4

Post by Night Nord » Mon, 27. Jan 14, 15:33

LuaDec is unable to decompile LuaJIT bytecode efficiently (even if you'll teach it of a new bytecode and all additional opcodes) because LuaJIT provides a substantionally more information via it's specialized opcodes which could be used for a much more precise control flow decompilation.

I think most of bugs one may found in the decompiled source are to be rather simple. There are big things that are not yet supported by the decompiler at all, right now. But hopefully they aren't used in the X:R's code.

User avatar
nonnex
Posts: 366
Joined: Wed, 28. Aug 13, 12:42
x4

Post by nonnex » Mon, 27. Jan 14, 15:57

Awsome work Nord.

User avatar
YorrickVander
Posts: 2689
Joined: Tue, 29. Oct 13, 21:59
x4

Post by YorrickVander » Mon, 27. Jan 14, 19:23

Very very nice - made some time to try this today. Great results :)
X Rebirth - A Sirius Cybernetics Corporation Product

Split irritate visiting pilot with strange vocal patterns.

ICO_hr
Posts: 415
Joined: Sat, 31. Aug 13, 17:56

Post by ICO_hr » Mon, 27. Jan 14, 22:05

SO, when we can expect better UI? :)

User avatar
arc_
Posts: 62
Joined: Sat, 7. Dec 13, 20:30

Post by arc_ » Mon, 27. Jan 14, 22:28

Extracted Lua function names and signatures: https://dl.dropboxusercontent.com/u/60681258/xr/lua.txt

These are all the global functions offered by XR itself. The builtin Anark member functions are not included.

User avatar
nonnex
Posts: 366
Joined: Wed, 28. Aug 13, 12:42
x4

Post by nonnex » Tue, 28. Jan 14, 09:00

Engine research: while ost of the UI is made dynamically by Lua code, some visual portions are taken from presentations files in some proprietary format. How much are they needed and what can we do without them is an interesting topic for a research.
1.) The visuals (meshes, component tree, textures and so on) are part of the compiled presentation files which are anark proprietary binaries.
2.) The Lua files defines the behavior of a presentation.

What can we do now without the presentation files? Well, with your decompiler we are able to modify the behavior of existing presentations, but we're not able yet to change the visuals (meshes) itself or add new visuals to it.

Beside that we're now able to change (hardcoded) things inside the behaviors. for example: the "property view":
If a modder wants to use the property view for some reason he will be faced with some problems.
1) The content of the property view cant be filtered.
2) There are hadcoded context modes defined for the property view. 'main', assign commander', 'shiptrader'. The modes defines the context of the conversation. So it was not possible to add a new context beside the existing ones. Which is possible now and great, because now a mod can use the property view to select ships, stations and so on and use the passed object for further work. Which was not possible before.

Another possibility could be to use the property view for other listings like ships in sector, ship commands, NPC selecting, mod settings, etc etc

User avatar
BigBANGtheory
Posts: 3167
Joined: Sun, 23. Oct 05, 12:13
x4

Post by BigBANGtheory » Tue, 28. Jan 14, 10:03

Good work, this has some real break through potential.

Night Nord
Posts: 1002
Joined: Fri, 13. Jan 12, 19:09
x4

Post by Night Nord » Tue, 28. Jan 14, 11:52

IF anyone is going to try it out - be sure to take the topmost master. I've found a few bugs with table indices and fixed them.

Plus there is a bug breaking main menu if widget_fullscreen.xpl is recompiled - in the "initializeTable" function a scrollbar component initialization is broken - part of a table constructor is trying to access the component itself which is not yet defined. I'll fix it at the evening (MSK).

What we can do right now? A lot, but also not so much at same time. Presentations are mostly used only for core components, such as window underlayer graphics, button/widget visuals and other such stuff.

On the other hand, table layouts and contents, all buttons, captions and colors are dynamically created. For instance - ALL the targeting system is done in the lua code - object boxes, arrows and icons, rings in the center of the screen... I think it should be possible to implement a software-projected gravidar using the lua code only.

But we can't add a new functionality, at least I haven't found a way yet (maybe the SetAttribute/GetAttribute functions could help) - most of functions exposed into the lua are read-only accessors or checkers, like "IsShip", "GetTradingShipsList".

Maybe it's possible to somehow modify the contents affecting state of objects inside, but I haven'd tried it yet.

In short - there no know way to change a ship's name for instance. So we can add a button or even a new dialog, maybe implement a name generation (a gamepad friendly alternative to a virtual keyboard), but I don't know if it's even possible to assign a name to a ship. Nor I saw a way to dispatch an MD event (so we can implement an MD-accessible logic via MD and use it from the UI code).

If there is no way to change object's state from the UI somehow (I think there isn't) then nothing will help except maybe a complete SDK - all the logic is hardcoded and without the SDK any modding is restricted to simple/useless tricks only.

@arc_
Extracted Lua function names and signatures: https://dl.dropboxusercontent.com/u/60681258/xr/lua.txt
These are only the _G members? Because it seems that they've switched from ordinary lua to luajit some time ago and a lot of functions is exported via the FFI interface only. Do you know a way to list their signatures too (we can extract a symbol table with names, but it isn't very helpful - plus structure layous are still unknown)? Other than simply grep all the sources and extract all ffi.cdata calls, that is.

User avatar
arc_
Posts: 62
Joined: Sat, 7. Dec 13, 20:30

Post by arc_ » Tue, 28. Jan 14, 12:43

These are just _G functions yes; I hadn't heard about FFI before... But looking at it, it's probably like you said: the .exe will only contain the mapping between function names and addresses, while the signatures have to be extracted from the declarations in the Lua scripts.

User avatar
euclid
Moderator (Script&Mod)
Moderator (Script&Mod)
Posts: 13289
Joined: Sun, 15. Feb 04, 20:12
x4

Post by euclid » Tue, 28. Jan 14, 18:37

Excellent job Night Nord - gives rise to hope :-)

Cheers Euclid
"In any special doctrine of nature there can be only as much proper science as there is mathematics therein.”
- Immanuel Kant (1724-1804), Metaphysical Foundations of the Science of Nature, 4:470, 1786

Night Nord
Posts: 1002
Joined: Fri, 13. Jan 12, 19:09
x4

Post by Night Nord » Tue, 28. Jan 14, 21:09

Ok, there is a new version (v0.1.2, the Download link is updated) with most of problems fixed - at least now you may start a new or continue a game with all the UI recompiled. It's even playable to some degree, but some problems are still here: A broken map, A broken reticle and finally broken buttons. The most crucial problem is obviously a targeting system - there is no selection boxes too, the second problem seems to do something with sized.

P.S. And, BTW, there are scrips I'm using to automate the process on Windows (on Linux I have a cool Unix shell to do the same, but much easier. Sadly I have to iterate on Windows too, because constantly rebooting will be too much)

Update: NVM the targeting system. I was trying to recompile old xpls (1.22) and use the result with 1.23. EgoSoft had changed some stuff, including a few global functions removed and presentations changed, so it wasn't working. But the size issue still sticks.

Night Nord
Posts: 1002
Joined: Fri, 13. Jan 12, 19:09
x4

Post by Night Nord » Wed, 29. Jan 14, 23:22

Version updated to v0.1.3. Button scaling is normal now and i haven't found any other obvious bugs in my simple test-run so far.

User avatar
Fumblesneeze
Posts: 75
Joined: Tue, 26. Nov 13, 10:07
x3tc

Post by Fumblesneeze » Thu, 30. Jan 14, 05:40

Great stuff there. Can't wait to see what people can come up with using this.

Did you get any info from egosoft whether you can put the sources into github?
Streaming while writing mods: twitch.tv/fumblesneeze

Ginger470
Posts: 158
Joined: Sun, 20. Jun 04, 03:24
x2

Post by Ginger470 » Sun, 9. Mar 14, 06:26

Can we get a step-by-step on how to use this :) That would be uber-useful ;)

Mad_Joker
Posts: 274
Joined: Sun, 14. May 06, 11:21
x3

Post by Mad_Joker » Mon, 21. Apr 14, 08:06

Thanks for this, man. I can now mod UIs (see here for a WIP of my UI modding adventures).

Post Reply

Return to “X Rebirth - Scripts and Modding”