[WIP] XR database

The place to discuss scripting and game modifications for X Rebirth.

Moderators: Scripting / Modding Moderators, Moderators for English X Forum

malolotja
Posts: 141
Joined: Thu, 7. Nov 13, 13:11

[WIP] XR database

Post by malolotja »

Hi all :)

For my XR pleasure I am quite often consulting http://roguey.co.uk/xrebirth/wares/
I don't know what I would do without that resource.

But unfortunately there were some things I was missing. Missing links between wares and stations for instance.

So I set out to create some tools to import the data from XMLs into an RDBMS. The first tool to import text is finished. Now I am working on the wares. The ultimate goal will be to create a webservice (which can also be run locally) for all our enjoyment.

Feedback and help will be greatly appreciated.

You can find the sources here:

https://github.com/wongak/xrguide

I will upload the binaries now (as soon as I can get the cross-compilation stuff sorted out).

Cheers

Edit: compiled the windows binary on my windows machine: https://github.com/wongak/xrguide/relea ... port_t.zip

Importing texts should be working now. Wares are almost done. Next steps: Create a webapp to display the data and import the starmap to link the wares to stations.
User avatar
YorrickVander
Posts: 2774
Joined: Tue, 29. Oct 13, 21:59
x4

Post by YorrickVander »

Lovely job. With some flagrent abuse of the debug log file, could even work on a live trade feed into the db.
X Rebirth - A Sirius Cybernetics Corporation Product

Split irritate visiting pilot with strange vocal patterns.
Deleted User

Post by Deleted User »

good idea.
User avatar
Tanvaras
Posts: 170
Joined: Thu, 27. Nov 03, 10:45
x4

Post by Tanvaras »

Well done, Very nice and handy :)
brammie
Posts: 100
Joined: Thu, 19. Dec 13, 23:26

Post by brammie »

Hmm, most of the wares can be ripped out of the save game xml file's
I was looking at the storage os the shipyyards to see what was missing and noticed
cargobayy one:
<component class="storage" macro="struct_econ_prod_stor_container_macro" ....>
....
<cargo>
<summary connection="connection_cargobay01">
<ware ware="bioopticwiring" amount="497"/>
....
<ware ware="machinegunturret" amount="163"/>
</summary>
</cargo>
....
</component>

and cargo bay 2
<component class="storage" macro="struct_econ_prod_stor_container_macro" >
...
<cargo>
<summary connection="connection_cargobay01">
<ware ware="microchips"/>
<ware ware="plasmaflowregulator" amount="415"/>
....
</summary>
</cargo>

the ware that was missing is the
<ware ware="microchips"/>

so the factory stalls....

anyways, if your good in xslt you can get the facotry storage from a save file, and use the production modules for the input and output. (or use some xml object code generator to parse and bind the save file.

The whole state of the game is saved there.


Hopes this helps

ps

<component class="buildmodule" macro="buildmodule_ships_albion_l_macro" ...>
<build ..>
<resources>
<ware ware="bioopticwiring" amount="98"/>
......

gives you the wares needed for current production

<component class="production" macro="struct_econ_prod_energycells_oa_macro" connection="space" id="[0x383ba7]">
<production start="13200.3" end="13800.3" item="0" cycle="0" state="producing">
<efficiency product="1.3662"/>
<queue ware="energycells"/>
</production>

gives an current output

and at the end of the file ive notices some lists bound to the facility id's if more then one product is possible, mostly your basic queue's of products

now i'm thinking of it, everything is in there, even the positions rotations, and its not hard coded in the cat/dat fille's... (prolly a template in there)

if you make it that I can upload my own zipped save file (around 2 mb) and see/save the generated resource input/output excell/open office doc ill bow befor you. lol

not hard, just soem work
malolotja
Posts: 141
Joined: Thu, 7. Nov 13, 13:11

Post by malolotja »

Hey thanks. I was able to extract almost all of the data I wanted to. Next thing will be the maps and their stations. Then I could link the inputs/outputs to the wares. Even create a pathfinder for the most profitsss/jump etc.
I sent an e-mail to Egosoft to ask if I was allowed to publish the extracted data. If not you can still use the tools to extract them yourself. They can be easily converted to SQL or CSV, which in turn can be imported into Excel.

Almost done with the ware productions :)

The Windows binaries might take until tonight. I have to compile them separately.
brammie
Posts: 100
Joined: Thu, 19. Dec 13, 23:26

Post by brammie »

I have not tried to load the xml in a real xml editor but i think its one huge tree containing of somthing like

<universe>
<system id='some ref to the T file name'>
<position x="..." y="..." />
<rotation pitch="..." yawn="..." />
<sector >
<position x="..." y="..." />
<rotation pitch="..." yawn="..." />
<zone>
<component type="gate" id=".." to="some other gate id" />
<component type="structure" id=".." >
<position />
<rotation />
....
all sub components including npc's, guns and stuff
......
</component >
<component type="ship" id=".." >
....
</component >
</zone>

</sector>
...
more secors
.....
</system>
.....
more systems
...
</universe>


wirth something like

http://bl.ocks.org/mbostock/4063550

or get ur basic tree layout or better an inverted node tree layout

http://www.cg.tuwien.ac.at/courses/Info ... /guide.htm

for nice visualization and you have the basic trade flow visualized :)

I've used the edge bundles myself to visualize the traffic of a web server within a site containg 100k pages and 1M hits/day and wanted a year flow thru the site, was fun to build...

would be nice to see where you can hit PMC the hardest, eg kill some critical food factory and all traffic gets redirected thru enemy space
malolotja
Posts: 141
Joined: Thu, 7. Nov 13, 13:11

Post by malolotja »

Hi, yeah, I will definitely load the map into a graph (mathematical graph that is; the thing with the edges and vertices ;) )
Once we have this, we can do anything to traverse it. Probably starting with Dijkstra/A* for simple pathfinding.
brammie
Posts: 100
Joined: Thu, 19. Dec 13, 23:26

Post by brammie »

sax handler or an xml pull parser handler and go ...

<component class="station" ....

and start from there, only 176 in my game

that might be te easiest way, the <trade wares="${list}"
and all subnodes of that tells the price amount and if its bought or sold
some have a source="script" and that ware the trade missions currently available there

so you only need to get the following per station

<component class="galaxy" id="[0x430ff]">
<component class="cluster" id="[0x430ff]">
<component class="sector" id="[0x430ff]">
<component class="zone" connection="sector" id="[0x4122e]">
<component class="station" id="[0x412fd]">
<trade>
<offers>
<mission><trade ware="dumbfiremissile" price="3878" amount="250" ><source class="script"/></trade></mission>
<production>
<trade ware="bofu" price="70" amount="18861"><seller/><source class="production"/></trade>
<trade ware="plankton" price="12" amount="129632"><buyer/><source class="production"/></trade>
</production>
</trade>
</component>
</component>
</component>
</component>
</component>

and get the id to name mapping from somewhere or use the macro name.
and thats the trade in a save fille.
I almost get the idea that the fastest way is to use a simple xslt file that transforms the save to a minimal form i just typed out, and just use xml binding to a frontend (im not to familiar with python) but in .net or java its very easy.
malolotja
Posts: 141
Joined: Thu, 7. Nov 13, 13:11

Post by malolotja »

Yep, that would be the info at the current state of the game (save file). I am currently parsing the XML libraries to extract a general guide (vanilla stations). The ware relations are of course the way they are ;)

It's not really magic. It's just some XML. I imported them into a relational database and now I will create a small webservice to make the data accessible.

I wish I could show you what I extracted as of now. But since the data I have is based on ES-owned files, I asked them if I was allowed to publish the SQLs/CSVs. As soon as I have a go, I will put it somewhere.
malolotja
Posts: 141
Joined: Thu, 7. Nov 13, 13:11

Post by malolotja »

So, put up the new binaries here: https://github.com/wongak/xrguide/relea ... .1-alpha.2

Text and wares import should be working now. A quick intro can be found here: https://github.com/wongak/xrguide
Will have to update the documentation.
User avatar
YorrickVander
Posts: 2774
Joined: Tue, 29. Oct 13, 21:59
x4

Post by YorrickVander »

@ brammie

Much of the trade and station flow issue is the limitation of trade scope, as defines in trade.ships.xml. The save files can be handy for a snapshot of a specific case of course. For instance, my addon script to dump the market to csv shows over time the market of say devries basically stops becuase of this limit : no trades out of cluster can occur, not just because of storyline but because theyre coded that way. I guess the idea is that the player fills the holes and makes the galactic economy work.
X Rebirth - A Sirius Cybernetics Corporation Product

Split irritate visiting pilot with strange vocal patterns.
brammie
Posts: 100
Joined: Thu, 19. Dec 13, 23:26

Post by brammie »

YorrickVander wrote:@ brammie

Much of the trade and station flow issue is the limitation of trade scope, as defines in trade.ships.xml. The save files can be handy for a snapshot of a specific case of course. For instance, my addon script to dump the market to csv shows over time the market of say devries basically stops becuase of this limit : no trades out of cluster can occur, not just because of storyline but because theyre coded that way. I guess the idea is that the player fills the holes and makes the galactic economy work.
Do you have any idea if the NPC stations are fully upgraded and if not do they upgrade over time?

The reason i'm asking is cause i have no idea if the NPC market changes over time (Is there a GOD engine like in the previous games)?

If there are changes it could be useful to see if there are eg new reinforced metal plates factories build, or if there is a new resource missing in a region.

ps,
DeVries misses a beef(3600/h) and spices(3600/h), might use some ice to but i see the miner going back and forth to supply the one factory there is. when you plug those holes there is lots of ecell, water, wheat and food left to use. The shipyard has no suppliers at all, but thats our task i think.
dmk
Posts: 682
Joined: Fri, 25. Nov 05, 00:59
x4

Post by dmk »

I don't know where is used beef,
but meat is produced here http://roguey.co.uk/xrebirth/universe/c ... 99/item-8/

(Terran Module is producing different wares)

P.S. I did not see ice miner... probably it was killed, and devries don't try to rebuild it - its building only warship for sun.

Return to “X Rebirth - Scripts and Modding”