How to read info out of X3 using external scripts

[THE_WESTi]

I want to create a dashboard that is showing my property owned etc. at all times. I am wondering if anyone knows how to read out of x3 to get this information? I thought about using a hex editor to understand the save file and then parsing that to get what I need but that would have time lag issues as the save files only represent a point in time. Is there some way I can get this information live in the same way that Litcube has the Rainmeter add on?

Any ideas would be great! I will adapt which language I use etc. (i.e. whether to use Python, .Net etc.) based on how I can access this information.

Edit: One way I have just thought of is could I create an X3 script which dumps the information I need to a text file in the X directory on a periodic basis?

[sepa]

Best option would be to build on top of G15/G19/Rainmeter plugin - it already exports some stats, just alter it a bit.

That's what I've done for my little project a few months ago - plugin exports basic data, Telegraf collects them and pushes to InfluxDB, Grafana is used for displaying the stats.
Works on Mayhem. I've done that as I could not find where my ore is

I've also started on external UI for Mayhem, similar concept, plugin to export data (using parts of Export/Import scripts), a bit of Java + Angular WebUI, idea is that changes made will be imported back into game.

Example dashboard:


Another one:

[THE_WESTi]

Awesome, thanks for the suggestions. I have started poking around in the Rainmeter scripts but am still new to X3 scripting.

Saw in your post in the Mayhem thread that you have created your own dashboard - would be keen to give it a go if I can? Could even negate the need for me to do my own!

[sepa]

Hey,

I would be happy to share what I've done

You will need a few things to get it going though:

1. Download and install Telegraf: https://www.influxdata.com/time-series- ... /telegraf/
2. Download and install InfluxDB: https://www.influxdata.com/get-influxdb/
3. Download and install Grafana: https://grafana.com/
4. Get the zip with my stuff and use it: https://cdn.discordapp.com/attachments/ ... _WebUI.zip

The zip contains the docker-compose.yml that can sort out steps 1-3 for you if you got a Docker installation somewhere (I run mine in a Linux VM).
There are also example dashboards for Grafana in there (read about how to import them on the net), as well as a sample plugin for Mayhem (messy work-in-progress version).
Btw, I had these running in the cloud for a while, but as I'm not playing X3 atm I took it down (it does cost a few £ per week to run).

Now plugin is for Mayhem, and so are the Telegraf files (they map some wares used in Mayhem), but with a little bit of work it shall be suitable for any mod/vanilla.
That said, I have little idea what stats would be so important to put on dashboard in vanilla game - I mean apart from what G19 plugin does already show

Give me a shout once you get these things running, and of course if you need any help, too.

Cheers,
Sepa

[bergi9]

@sepa
I really like the idea. I tested it after some modifications, the log.txt file is given to 9005 instead of 9004 which fixed the rainmeter output. Telegraf config file.
That was smart to do check every 10s per sanctuary but that could give at end a big delay. If you have like 30 sanctuaries then one Sanctuary data would be updated each 5 min and adding new sanctuary would take a time till you see it on the WebUI.
The question for myself is: how much does the performance affects to get infos and write into file? I mean if we would fetch all ship informations in near-realtime. And I assume the string concenation is not "free" here like on normal programming languages.

EDIT: I made a seperated AL script exclusively for the telegraf, to not override the G19 AL script. And added interval option into the AL plugin by toggling the AL enable status (should match the interval in the telegraf.conf) http://bergi9.de/telegraf_webui_v0.zip

[sepa]

Hi,

Sorry to say I've completely missed your response - I've not been playing X for a while.

Anyhow, I do play it again and I have saw your post now Tomorrow I'll look into your script - I've also made some small adjustments and fixes here and there, and made a few new variations of Sanctuary View dashboards - this one is the latest:



As per timing of updates - I've changed it to 5s, and I use Grafana refresh of 30s. Decent enough, but with more sanctuaries (I got 11 atm) it will be a pain. Unfortunately exporting data is costly - I've went down to single sanctuary at a time as with more than a few the game was stuttering.
Ideas as per how to speed things up:

• Output selected wares only instead of all of them - in reality you don't care about your stack of Argnu Beef as much as you do about Crystals, and you probably don't give a toss about Passengers at all.
  Separating wares output into frequent/occasional/on demand should speed things up a lot, also on InfluxDB/Telegraf side.
  
  I.E.: Stats like on my dashboard could be outputted at every cycle, for all the sanctuaries (or for X sanctuaries at a time if we see performance hit), while less important stats like laser/shield stocks etc can be exported round robin per sanctuary - 5 minutes delay here does not matter much.
  Additionally a hotkey could trigger 'export all'. Such an 'export all' is something I was considering for a WebUI - a way to manage jobs/tresholds/production from the browser instead in-game. I've done a small POC, exporting data from game and then transforming it from XML to Json using Spring Boot + Angular 7 to display it. Not progressed much past that, but I'm planning to come back to it one day.

• Another thing to consider is a different serialization method - some testing would be needed to determine the fastest way for X to get data out. Parsing it on the other end is not that much of a problem, Telegraf can do a lot, and there is always Perl if more mangling is needed. The current way was simply the quickest way to get the concept working, it is not the most performant for sure.

I think best option would be to do both - limit exports and work on serialization method.
Let me know please if you want to play with these things together.

Cheers,
Sepa

[sepa]

@bergi9

I've checked out your AL plugin - really nice job.

Today I have implemented selective output of wares - only a few things from group 16 plus full 15, 14, 13, 11, 10, 9, 8. That reduced the output file size by ~60%.
Also changed the way Telegraf works - instead of polling the file every X seconds I switched to 'inputs.tail' plugin - it detects changes in log file. Log file itself is being appended and wiped out once the sanctuary cycle is restarted.

I run your AI plugin on 1s interval now with no sign of delays - it's still round robin through all sanctuaries though, so effective time to get all data is 'SanctuaryCount * AiPluginRunInterval'.

One small harmless issue is that Telegraf throws some parsing errors due to bug in tail plugin - it does not support comments in CSV files correctly, and I can't be bothered to deal with empty lines in output with any other way than turning them into comments

I guess now it would be a good time to pack that thing together into some easy to install form, or find a way to host it online again so ppl can easily use it.

[sepa]

I've managed to get some basic production stats out - I had to modify the Mayhem.Monitor.Shipyard.Player to get these though... I hope that Joubarbe might consider my changes in future update, or suggest a better way to get these out without altering his code.

Teaser: