Api?

[Tenlar Scarflame]

It strikes me that a very welcome tool for X4 would be some kind of API that would allow external applications to retrieve world information, and detect/trigger events, in a running instance of X4.

It's 100% understood and appreciated that this would probably not be a Release Day Feature. =b That being said, I would LOVE to see what the modding community could do with this kind of tool kit... Windows- or even browser-based sector maps, fleet management interfaces, scripting tools, all kinds of fun stuff comes to mind. Heck, if Kerbal Space Program is any indication, small-scale multiplayer would even be on the table for a dedicated and skilled modder.

TLDR - yall have any thoughts regarding an X4 API?

[Ketraar]

Not the first time this has come up.

https://forum.egosoft.com/viewtopic.php?t=399110

MFG

Ketraar

[Tenlar Scarflame]

Not the first time this has come up.

Saw that, actually that was kind of what got me thinking in this direction. The biggest draw for me in that thread is the idea of a "second monitor" application that could let you have things like a sector map, fleet management, statistics, etc. running in a second screen or on a secondary device. Not something that Egosoft would be writing themselves (obviously they're plenty busy at the moment) but that an intrepid modder/modding team could cook up.

But, thinking with a broader scope in mind, that's just one use case for the interface that would be required to get that to work. Once you're able to query game data and detect/execute scripts from outside X4, a LOT of other stuff potentially becomes possible. ^^

[Tamina]

That would be very nice. Maybe just the ability to send and receive data from scripts (mods).
Then we would be able to create (mod) our own API. Egosoft just needs to implement some form of I/O streams in scripts.

How does inter-process communication actually work? Does it work?

[SirNukes]

If you launch X4 from some other process, it is straightforward to control the standard IO pipes. Input to X4 could just be MD xml action commands sent as a text block, that X4 can parse and process like normal MD commands (maybe with a per-session temporary namespace). An MD command can be added for returning a text string with captured information to stdout, which the calling process can parse and handle as it wants.

Overall, simple and powerful. This is the sort of setup I've used when needing to integrate with some other program in a bidirectional manner, taking maybe an hour or two to get something working.

[Lord Kellmar]

I've been dreaming of something like this for ages...

Most people only see the option to view info on another monitor. But with the accecibility of Arduino-like components, an API can make things wayyy more interesting :

Like this or like that or even this (video)

/*insert spongebob "imagination" image*/

[Tenlar Scarflame]

I've been dreaming of something like this for ages...

Most people only see the option to view info on another monitor. But with the accecibility of Arduino-like components, an API can make things wayyy more interesting :

Like this or like that or even this (video)

/*insert spongebob "imagination" image*/

That would be a FANTASTIC use case. Knowing the kind of stuff I've seen people build for Kerbal Space Program, I can just imagine the crazy simpits and command centers that could potentially arise out of this. (For those of us with considerably less physical space to work with, secondary monitor / device functionality is the next best thing. ^^)

[Tamina]

this (video)

Okay! We need this! Now!

[Morkonan]

this (video)

Okay! We need this! Now!

That series of lights is exactly what I saw in my mind the last time I was turned down for a date.

UV radiation is destructive to humans and anything that promotes less exposure to the Sun would benefit humanity. If Egosoft cares about humans, they will provide this API so that its customers can construct working X4 cockpits in their basements...

[Norway174]

If the LUA is exposed to your filesystem, or a at the very least specific folders.

I can totally imagine this being quite trivial to add as a mod and a separate app.

Simply just make the LUA in-game write and read from, even simple .txt files would do.

And then have the app read and write to those same files.
From there, you could pretty much do anything with the app.

This of course assumes, there will be some method on which the scripting engine, witch I heard was LUA, do have access to at least a sandboxed filesystem.

I'd assume so. Since scripts will probably want to save various settings for example to some location.

Downside is that this method probably won't be as a real time though.

[Loader009]

Let me push this thread @Ketraar and @Tamina .

I request a feature to be able to write data to a serial interface (on windows generally a COMx interface, on linux often ttyACMx or ttyUSBx - x for a number).
The data which is output should be fully controllable by a script, so everything the game needs is serial interface support (maybe also for X: Rebirth and older?).

It would be a start to export ingame-data while playing.
An arduino can then display different stuff, depending on a script, e.g.:
- credits
- location
- warnings and alerts
- rocket types and amounts
- Shipname, hull, shields
- drone status

And I am even thinking about an automatic on/off switch for travel mode,
self-turning speed wheel (with a motor or a servo and a rotary encoder).

There are endless possibilities.


Unfortunately I am just a beginner at Arduino programming and I haven't wrote a script for X: Rebirth / X4 Foundations yet.
But I think that here are tons of people with much more knowledge and I would be excited to see anything like this.

I hope this will be read and at least talked about.
A serial interface communication is probably the easiest solution to provide data for external hardware.

[SirNukes]

I wrote a core inter-process api using pipes 1-2 months ago, and just have been lazy about posting it to the forum.
https://github.com/bvbohnen/X4_Named_Pipes_API

[Tamina]

@Loader009
Yes, a serial connection would be nice but a general "stream/pipe API" is better (basically a serial connection between programs). You could write an external program that maps streams to a physical serial connection.
The problem with your approach is that it is not very straight forward nor standardized or reliable to communicate via a serial connection. On Windows' standard drivers the DTS pin is always pulsed, which would i.e. reset an arduino everytime.
Then there is the missing standard protocol. And even if one would be chosen by Egosoft, it might not be compatible with your Arduino.

I wrote a core inter-process api using pipes 1-2 months ago, and just have been lazy about posting it to the forum.
https://github.com/bvbohnen/X4_Named_Pipes_API

That's pretty cool Tell us more *-* Is Python used to create the Pipes on X4 site or do you need Python on the receiving site as well?

[Loader009]

I wrote a core inter-process api using pipes 1-2 months ago, and just have been lazy about posting it to the forum.
https://github.com/bvbohnen/X4_Named_Pipes_API

That's nice!
But... I have absolutely no experience in Windows programming/compiling and only basic experience on linux for python and arduino c++ code.
So I have no idea right now, how to use Win pipe, but that's a reason to find out, how I could use that.

@Loader009
Yes, a serial connection would be nice but a general "stream/pipe API" is better (basically a serial connection between programs). You could write an external program that maps streams to a physical serial connection.
The problem with your approach is that it is not very straight forward nor standardized or reliable to communicate via a serial connection. On Windows' standard drivers the DTS pin is always pulsed, which would i.e. reset an arduino everytime.
Then there is the missing standard protocol. And even if one would be chosen by Egosoft, it might not be compatible with your Arduino.

Oh, it is way more complicated than I imagined.

I don't know of "stream/pipe", I don't program in general, just some tinkering with an arduino and a neverending try again and fail loop (because of not enough interest) on various programming languages.
That's why I rely on people, who might program that stuff. (I now have to take a look of SirNukes program, if I will be able to forward e.g. the ships name to a serial connection.)
The serial interface would be just for DIY solutions, I don't think that there is any professional display with some (hatefully proprietary) standard, which would audience a smaller group anyway.

I didn't know about the DTS pin issue (though, I am irritated right now of what pin this is), I hadn't had such issues with linux yet (or I didn't notice it).
Though, everyone who uses an arduino, should be able to circumvent the issue and I am very sure that people will do tutorials (and youtube videos) on how to set up the arduino and circumvent the issue.
So I don't think that this is a problem at all.

[SirNukes]

That's pretty cool Tell us more *-* Is Python used to create the Pipes on X4 site or do you need Python on the receiving site as well?

Python is optional. In loose summary (I think some of this is written up on git somewhere already), there are five parts:

1. dll with lua bindings for accessing windows pipes. Porting to linux would mean rewriting this C code and recompiling for the linux equivalent, eg. using mkfifo.
2. X4 lua api for reading/writing pipes directly.
3. X4 md api for reading/writing pipes from md scripts, with state handling in case of saves/reloads/etc.
4. (Optional) Python pipe server to host the pipes, and a little wrapping on pipe objects for easier access. (Wrapping is for windows pipes, though a linux version could be slotted in.)
5. X4 md api for starting up per-mod (when enabled) python pipe server modules, assuming the host is active (requires python).

Parts 1-3 would work fine if someone wants to write their pipe server in something else; they make no assumptions about the communication protocol. Parts 4-5 aim to make writing and running new pipe servers easy.

On that git repo is an example mod that uses a pipe to add new hotkeys, where md scripts can register key combinations and their callback cues, and the python server handles capturing and processing key presses (when x4 has focus).

But... I have absolutely no experience in Windows programming/compiling and only basic experience on linux for python and arduino c++ code.
So I have no idea right now, how to use Win pipe, but that's a reason to find out, how I could use that.

Linux mkfifo works almost the same way, just with different commands and such to set it up. I don't mess much with linux outside of work stuff, though, so I don't plan to add it to the tool. But if someone wants linux support, it shouldn't be too hard for them to slot it in.

[Killjaeden]

When i think api i think of the additional resources available for simlation/computation it brings. External applications can processes data (in a different threat, running on one of the many cores that twiddle their thumbs) and only shoves the results back into the game.
Arma 3 can even call custom extensions (dll's) in C/C++/C#, and it's resulted in, for example, libraries that allow avoiding the slow custom-script-language that sits normally on top and directly use the functions of the game code in a generic language. (https://forums.bohemia.net/forums/topic ... -native-c/ )

If X4 had something like this ... think of all the available libraries that can be used. AI, path planning, Machine learning...
Oh ... and you wouldnt have to touch xml any more than necessary

[AquilaRossa]

External applications can processes data

i was thinking about this the other day and wonder if out of sector stuff could be handled by a separate process that can act like a data server feeding universe data to the main process that handles the map and 3D in sector visuals etc for the player. But I looked at the map side on and you can see stuff on the other side of the universe, which explains why going through gates is seamless (i have only recently got back into X after over 10 years away). I am not sure how they could separate what the player can see from the universe's number crunching and keep it seamless. A second of sync lag going thru a gate would not worry me though if the trade off was not getting really low FPS on stations etc. Not everybody has 16 or more threads, but it's heading that way (64 core Threadripper might be happening and a 16/32 core/thread Ryzen is due very soon, while I think Intel are going with 10 for desktop and 20 or more on their mega buck HEDT stuff). I would like the game to use those all threads though. it uses 8 cores max at the moment, or 8 just threads?

[paris_england]

It's already possible to output to a custom text file from scripts ... it would be trivial to parse those externally. You'd be limited to HDD IO speed, but for non-critical events (ones which don't need constant microsecond updating), this would be a viable and simple -- albeit one way (game output only) -- solution.

That said, as I was typing this I thought maybe a symlink to the t-folder and using a specific t file for the custom output logs 'may' be readable/manipulatable in-game (hacky for sure, but...????) Could be a way to get data out and back in.

Not sure what that could be used for (or if it's possible) .... but ....

[Killjaeden]

Not sure what that could be used for

Simulating a much larger universe than actually exists, and only "displaying" the parts you need to see.
Or performing calculation intensive stuff that only needs to be updated on a universe scale (faction military/economic logic stuff)

You could export ware data for entire universe, and then perform some large number crunching /statistics/ stuff for analysis, which then can be turned into info you can feed into a combination of statemachines and neural network, to get output about how achieve <some goal> (e.g. stabilize the economy). The return information would be fairly small. Build station X there, scrap station Y there, ... liquidize assets

You could also simulate different competing companies that way - each making it's own number crunching from the same database.

[Tamina]

Python is optional. In loose summary (I think some of this is written up on git somewhere already), there are five parts:
[...]
Parts 1-3 would work fine if someone wants to write their pipe server in something else; they make no assumptions about the communication protocol. Parts 4-5 aim to make writing and running new pipe servers easy.
[...]

I see, thanks for the quick summary
One last question: X4 functions as a client? So X4 is connecting "to" pipes, created by an external server site?
I am asking because I just googled a bit and it seems easy to connect to pipes but difficult to create them, and rules out other high-level languages at this point. So it is either C++ or your Python


I wonder if it would be possible to use TCP/UDP sockets instead of pipes in your project (Not asking you to do it, but theoretically)