Enhanced ship computer via LLMs and TTS

[Anekto]

Hello dear community!

I dont know how many feel like this but i miss the old x3 ship computer voice explaining stuff and giving a sort of immersive feel to the game.

They added some more of it since launch but i still feel unsatisfied.

Now i have never programmed a X4 mod but i start to think about a mod i really would like to have in the game. As technology nowadays stands large language models and text to speech can be run on comparatively simple hardware. Depending on your graphics cards 4 to 8gb vram can be enough to run a llm and tts service. So i would really like to enhance the x4 ship computer voice read out with a large language model that runs locally on my hardware or players hardware. i am sure there could also be an api access to run it in the cloud. for now i would really like to have or make a mod that runs an llm and gives a futuristic revival of the old x3 ship computer voice creating a more immersive gameplay.

In the beginning it should simply be a readout of game stats and what not when i open ship databases or encylopedia entries but could be developed to be more interactive and integrated into the whole game.

If anyone has any thoughts about this please let me know. Also if you think this is feasible or why it is not feasible please let me know.

A constructive discussion even if opionated is very welcome.

Have a great day, thanks.

[adeine]

Hello dear community!

I dont know how many feel like this but i miss the old x3 ship computer voice explaining stuff and giving a sort of immersive feel to the game.

They added some more of it since launch but i still feel unsatisfied.

Now i have never programmed a X4 mod but i start to think about a mod i really would like to have in the game. As technology nowadays stands large language models and text to speech can be run on comparatively simple hardware. Depending on your graphics cards 4 to 8gb vram can be enough to run a llm and tts service. So i would really like to enhance the x4 ship computer voice read out with a large language model that runs locally on my hardware or players hardware. i am sure there could also be an api access to run it in the cloud. for now i would really like to have or make a mod that runs an llm and gives a futuristic revival of the old x3 ship computer voice creating a more immersive gameplay.

In the beginning it should simply be a readout of game stats and what not when i open ship databases or encylopedia entries but could be developed to be more interactive and integrated into the whole game.

If anyone has any thoughts about this please let me know. Also if you think this is feasible or why it is not feasible please let me know.

A constructive discussion even if opionated is very welcome.

Have a great day, thanks.

Running a live LLM/TTS does not sound like a good idea in X4.

A better idea would be to write a standalone tool that scans dialogue tables and pre-generates all lines outside of the game this way. Basically you run it once and it generates Betty voice lines for all the lines in your tables regardless of what mods you run. No performance impact and future proof

[Anekto]

Oh you mean like pre compiling the output or lets say building a cache beforehand instead of using real time inference?

You might be onto something that real time inference calls might take a while to get a respose. But the again llms in a usable state exist now only for 2 years and they have become much faster in that time. They are scaling incredible well with time to perfomance. So i would estimate vaguely that in another two years real time inference might be possible without the need of pre running inference for various dialogs or outputs. Not that this would not add another performance boost to use pre inference and if nothing in a database is available actually use real time inference to answer the mods call.

Yours seem like a solution but i think we could already go the full way and prepare for a full integration with a backup plan to use pre run inference on various dialogs.

It will be definitely possible to run real time inference i think ^^.

[beko]

I'd not waste any resources on "AI" just because it's the hot thing rn and people literally throw money at every project announcing "AI". As a dev I'm very opinionated about this.

Pre-generating some lines, as Adeine mentions, is the way to go.

In the beginning it should simply be a readout of game stats and what not when i open ship databases or encylopedia entries but could be developed to be more interactive and integrated into the whole game.

As for dragging out live game data: Be my guest https://github.com/bekopharm/x4-simpit

Alas I have no confirmed installation for Windows so far so it may need some tinkering there. Should get the gist though

[Anekto]

What do you mean by wasting resources?

I'd not waste any resources on "AI" just because it's the hot thing rn and people literally throw money at every project announcing "AI". As a dev I'm very opinionated about this.

Things like ollama work right out of the box. I am kinda amazed how easy it is to get a small llm to run locally on my computer. I want to explore this possibility further. I agree right now AI often seems like some marketing scam slapped on anything just to say ai is in it. Lets rather call it some sophisticated text to speech interface with bi-directional capabilities with future proof extension support.

https://ollama.com/

this is ollama which i tried and used to run a local model on my pc on an gtx 1070 ti. It did use like 6-8gb ram if i remember correctly maybe less with a 7 billion parameter model like minstral i think. It did get very slow when i played a game aswell as the gpu vram was competing with the ingame vram used. But with a gpu like 16gb vram both can be run at the same time without any hiccups. Thats what i am aiming for.

Even if it looks like slapping the AI meme on a mod request i really would like to explore a way to make it actually a cool addition instead of a waste of time !

EDIT:

Your github looks promising, i am no dev and only did some small coding and bit small linux exploring but on the first glance it looks very promising to get something working even if its just a text to speech pre compiler. it could get the project started, a good foundation. THANKS!

[thomassanford]

I agree with the above that running an LLM to act as the computer's voice would not be the best way to go in terms of computer resources or even accuracy. The computer voice could easily be expanded using in game mechanisms. You could use an AI RVC to clone the existing voice and then prerecord whatever you wanted. I would use something like https://huggingface.co/coqui/XTTS-v2 to clone the voice and then create the needed responses. The hard part and another reason an LLM wouldn't be very practical or easy is conveying game states to trigger the needed reactions. Just my opinion of course.

[Anekto]

I would use something like https://huggingface.co/coqui/XTTS-v2 to clone the voice and then create the needed responses. The hard part and another reason an LLM wouldn't be very practical or easy is conveying game states to trigger the needed reactions. Just my opinion of course.

One poster already mentioned a mod/interface for x4 which leads you access to lots of ingame information and use it however you want. So there might already be a solution for how to convey game stats to the llm. I think if at first it would be more of a passive role like a TTS informing about game stats like the messages ingame already do would be a good way to start and expand on!

I totally see how this a project again way bigger than what i can easily whip up. But no i think at least a sort of api for an LLM integration could really be something to look into. I know hardware right now constrains this but again i think in a few years according to moores law we will have way better access to a capable model and hardware to make this cost effective. UP side would be a shipcomputer that is actually really a sort of ai which it usually gets portrayed in games.

I agree probably using text to speech synthesis to prerecord lines and clone the voice would be a good start. But i would rather go the whole way and try to get a sort of api ready to interface with more of the game.

Thank you for your input !