Machine learning aspects for game AI?

This forum is the ideal place for all discussion relating to X4. You will also find additional information from developers here.

Moderator: Moderators for English X Forum

==Troy==
Posts: 61
Joined: Sat, 17. May 08, 12:36

Machine learning aspects for game AI?

Post by ==Troy== » Wed, 3. Jan 18, 22:41

First to preemptively answer common questions:

- ML : Machine Learning. RL - Reinforcement Learning (a subset of ML)

- ML AI is too CPU intensive - for training yes, a frozen trained AI is very efficient and simple to run. Can be run in parallel to the main game. Or even on GPU if needed.

- ML AI is too young to use in games. Partly correct. ML AI only won game of GO and DOTA, and those were made by large teams focusing on a single problem. ML AI still has not won in SC2, but shows great promise in the recent competition played. These projects were huge and aimed at teaching AI to play from the same input as humans - visual feed. AI is a lot more prominent in simpler environments (noise cancellation, face detection, etc).



To give an idea of what Reinforcement learning is, imagine AI agent controlling something in the game. The developer decides the reward signal for said agent (amount of credits / etc) and actions that the agent can take (such as set price of goods, or go to place, buy goods etc). The agent then explores the environment by taking the available actions and learns to maximize the reward signal over thousands of millions of simulations. It takes long to train, but once trained it does not take much to run one.


I do not see ML as an all-answer to all problems of AI in X-series games. But I do strongly believe that ML, and specifically RL agents are the only way to give life to the NPCs in the X universe.

Specifically ML could be used for individual independent ships and stations :

- Traders could train to maximize income per minute while striving to survive

- Stations learn to price their goods and demands to attract traders, while still making profit.


This could of course be applied to other aspects of the game, but the above two are the simplest ones in my mind and the most relevant ones for "feeling" the life of the game (the number of times I found stations just losing money to me is not fun and one would expect they'd go out of business pretty quick).


There are already existing implementations of the cutting-edge algorithms available and open source (https://github.com/openai/baselines , published by OpenAI (Elon Musk;s company)).So its not a matter of developers figuring out how to implement ML algorithms - more of figuring out how to apply them to the game. And as the focus here is not to "play from visual input like human does" a lot of simplifications can be made that would mean a simpler neural network would handle the task.


I'm happy to provide more references and sources if needed. And do sincerely hope my words would reach at least one dev willing to consider the application :)

User avatar
Killjaeden
Posts: 5023
Joined: Sun, 3. Sep 06, 18:19

Post by Killjaeden » Thu, 4. Jan 18, 03:56

more of figuring out how to apply them to the game
Sounds easy, is very much not so.
ML without providing all the information a human intuitively has from his knowledge about his world and the game world is imo. useless for any complex rule set, unless you spend considerable time to trial and error it.

For complex games (means: many complex rules) like X games you have to dig down very deep and analyze what information a human is processing in the specific situations. And you have to provide a method for AI to receive the same information as a human. That alone can be a big and complex thing. Especially when 3D environment is involved in this thought process. Your ML AI can never get better than the developer, if he can't provide it with all the important information - not just the one the developer thought was important or could think of at that moment.

A simple example: A military general reads a map of some land terrain with the aim to make strategic or tactical decisions.
It's incredible how much information you have to artificially provide an AI to make any valuable decision whatsoever. As developer you need to author this information, preselect what you think is important. I did some thinking on this issue, wrote a 14 page doc alone on what information would be required and it's not even comprehensive or complete. In this example, i believe i need 18-20 different maps (3d arrays basically - Position in 2D and some informational value) with different information encoded in it to give the AI the same strategic and tactical information i would be having when just looking at a normal terrain map. And this is already with my bias as developer - already having performance and practicality thoughts in the back of my head. So in reality it could easily be more information, i just havent realized or dismissed the importance yet. And that goes the same for any non-visual information as well.

Lets assume you have this information problem solved. Now you have to teach it the rules you think are important and tell it the goal it should maximize. These are very critical things.
If you dont and give just the very basics (sell, buy, maximise profit) and simply let the ML do it's thing, its optimising weight factors for something that is based on a certain state of the universe at that time. Plus, maximising profit is extremely vague. It would lead to short term thinking and min maxing. Not always is going for the short term max profit better than for long term profit. For the AI to understand this concept you have to do a lot of work on the "teach the rules" and goal setting part. And if its just the basic stuff you are after - you are quicker just hardcoding that AI in - because the work (assessing what information and rules to define, and providing them) is the same. The only difference is that you hardcode how the AI weights and rates the information. ML AI trial&errors to learn the importance it gives to information for a decision process.

If this was so easy to describe, somebody would have already done it for RL economics, and a lot of people would be out of a job.

In addition, game rules can change over time (patches or even just the dynamical nature of the future X4). Means whatever the AI has now hardcoded as decision weights into it, it could lead to bad decision making in future. So you have to relearn it everytime some rule changes or the dynamic has swung to far in one direction.

SC2 AI - as far as i know - is currently just good at either copying the current meta game of other humans, or the meta game the way it's developers understand it. It is not developing entirely new strategies on its own.
Image
XTC Mod Team Veteran. My current work:
Image

ZaphodBeeblebrox
Posts: 1460
Joined: Mon, 10. Apr 06, 20:35

Post by ZaphodBeeblebrox » Thu, 4. Jan 18, 06:07

Q) What are the hardware requirements needed to run artificial intelligence (including how a machine also learns, aka Google Deep Mind)?

An answer from a Google engineer

Andrew McGregor, Site Reliability Engineer at Google (2013-present)

A learned model can be quite small and lightweight… for example, a general English language speech recognition model is about 400MB of data, and runs just fine on a modern phone. Speaking a language is about the same size, and also runs on a phone, at the same time as the recogniser… and now you know how Google Buds do their trick.

Making those models? That takes a facility you can see from space, that uses a big town worth of electricity, employs a couple of dozen maintenance staff, and costs a few hundred million dollars.
It was a woman who drove me to drink... you know I never went back and thanked her.

Don't try to outweird me, three-eyes. I get stranger things than you free with my breakfast cereal.

==Troy==
Posts: 61
Joined: Sat, 17. May 08, 12:36

Post by ==Troy== » Thu, 4. Jan 18, 14:21

ZaphodBeeblebrox wrote:Q) What are the hardware requirements needed to run artificial intelligence (including how a machine also learns, aka Google Deep Mind)?

An answer from a Google engineer

Andrew McGregor, Site Reliability Engineer at Google (2013-present)

A learned model can be quite small and lightweight… for example, a general English language speech recognition model is about 400MB of data, and runs just fine on a modern phone. Speaking a language is about the same size, and also runs on a phone, at the same time as the recogniser… and now you know how Google Buds do their trick.

Making those models? That takes a facility you can see from space, that uses a big town worth of electricity, employs a couple of dozen maintenance staff, and costs a few hundred million dollars.
I work in this area. Language models are quite complex and take an enormous amount of expensive data to train. And has nothing to do with reinforcement learning as such.

OpenAI already provides RL models that can be used on variety of tasks with minimal modifications. The models are also quite simple in comparison.

AlphaGo Zero (read the paper if you wish), Ran on effectively a powerful desktop computer and trained in about a week.



Killjaeden wrote:
more of figuring out how to apply them to the game
Sounds easy, is very much not so.
ML without providing all the information a human intuitively has from his knowledge about his world and the game world is imo. useless for any complex rule set, unless you spend considerable time to trial and error it.

For complex games (means: many complex rules) like X games you have to dig down very deep and analyze what information a human is processing in the specific situations. And you have to provide a method for AI to receive the same information as a human. That alone can be a big and complex thing. Especially when 3D environment is involved in this thought process. Your ML AI can never get better than the developer, if he can't provide it with all the important information - not just the one the developer thought was important or could think of at that moment.

A simple example: A military general reads a map of some land terrain with the aim to make strategic or tactical decisions.
It's incredible how much information you have to artificially provide an AI to make any valuable decision whatsoever. As developer you need to author this information, preselect what you think is important. I did some thinking on this issue, wrote a 14 page doc alone on what information would be required and it's not even comprehensive or complete. In this example, i believe i need 18-20 different maps (3d arrays basically - Position in 2D and some informational value) with different information encoded in it to give the AI the same strategic and tactical information i would be having when just looking at a normal terrain map. And this is already with my bias as developer - already having performance and practicality thoughts in the back of my head. So in reality it could easily be more information, i just havent realized or dismissed the importance yet. And that goes the same for any non-visual information as well.

Lets assume you have this information problem solved. Now you have to teach it the rules you think are important and tell it the goal it should maximize. These are very critical things.
If you dont and give just the very basics (sell, buy, maximise profit) and simply let the ML do it's thing, its optimising weight factors for something that is based on a certain state of the universe at that time. Plus, maximising profit is extremely vague. It would lead to short term thinking and min maxing. Not always is going for the short term max profit better than for long term profit. For the AI to understand this concept you have to do a lot of work on the "teach the rules" and goal setting part. And if its just the basic stuff you are after - you are quicker just hardcoding that AI in - because the work (assessing what information and rules to define, and providing them) is the same. The only difference is that you hardcode how the AI weights and rates the information. ML AI trial&errors to learn the importance it gives to information for a decision process.

If this was so easy to describe, somebody would have already done it for RL economics, and a lot of people would be out of a job.

In addition, game rules can change over time (patches or even just the dynamical nature of the future X4). Means whatever the AI has now hardcoded as decision weights into it, it could lead to bad decision making in future. So you have to relearn it everytime some rule changes or the dynamic has swung to far in one direction.

SC2 AI - as far as i know - is currently just good at either copying the current meta game of other humans, or the meta game the way it's developers understand it. It is not developing entirely new strategies on its own.

This is the beauty of reinforcement learning - you do not require to do any of the points you mentioned above.

All you literally need to do is to provide the AI with :

- State information (Where you are, what you can see on the radar. etc)
- Possible actions (Dock at X, fly to Y, jump to Z)
- Reward signal (You earned 10$ good!, You earned 0$ OK! You died, REALLY BAD!)

And then you leave your simulation running as the AI learns to play the game. And learns to understand all of the complex concepts you mention.

Just look at the DOTA Machine Learning AI, imagine how much complexity did it have to learn, and all the developers had to do was to make an RL algorithm and leave the game running while the AI played against itself as it learned!

ZaphodBeeblebrox
Posts: 1460
Joined: Mon, 10. Apr 06, 20:35

Post by ZaphodBeeblebrox » Thu, 4. Jan 18, 14:38

Taken from Wikipedia

Hardware cost
The hardware cost for a single AlphaGo Zero system, including custom components, has been quoted as around $25 million.

The rules of Go are simple and easy to express. I just had a look at a web site dedicated to learning and playing GO.

A few A4 pages which was mostly diagrams, to explain the rules and expound on some strategy.

[edited 1 time]

==Troy==
Posts: 61
Joined: Sat, 17. May 08, 12:36

Post by ==Troy== » Thu, 4. Jan 18, 15:52

ZaphodBeeblebrox wrote:Taken from Wikipedia

Hardware cost
The hardware cost for a single AlphaGo Zero system, including custom components, has been quoted as around $25 million.

The rules of Go are simple and easy to express. I just had a look at a web site dedicated to learning and playing GO.

A few A4 pages which was mostly diagrams, to explain the rules and expound on some strategy.

[edited 1 time]
Correct, That is their final system used to play. If you watch the actual videos of the game you will note David Silver commenting that AG-0 can be ran on a desktop computer after optimisations that they would propose later.


I recommend to look at AI playing DOTA and SC2. And recognise that it is done from pixels, not from in-game data that would be available to X-based AI, thus simplifying it significantly.

User avatar
Killjaeden
Posts: 5023
Joined: Sun, 3. Sep 06, 18:19

Post by Killjaeden » Thu, 4. Jan 18, 17:34

And then you leave your simulation running as the AI learns to play the game.
There is a distinct difference between DOTA, SC2 and X. Dota and SC2 games have and End that is a "win". And this end goal is reached in the scope of minutes. That means iterations have a clear end result, at which AI can decide if its good or bad what it did. X games do not have a "you win" scenario. And they do not end. The maps in SC2 and Dota are static. Ressources always at the same positions. Certain situations repeat themself frequently. In X this is not the case. factories are not always at the same spot, can change dynamically from factors outside of controll of AI, or directly by controll of AI.

Its not impossible. But it will take a lot longer than SC2 or Dota before anything usable comes out. Egosofts ressources are limited. Expecting them to be fore-runners of game AI is ... slightly unrealistic.

If you think its not difficult - go ahead to create a ML AI for X3 or XR. Their universe states are much more static compared to what X4 will be, so it should much less time consuming... show Egosoft a proof of concept and offer collaboration for "intelligent AI dlc" for X4. Thats how you get things done.
And recognise that it is done from pixels
And how good would that work in a 3D environment with dynamic lighting? Completely different level of complexity. Before AI can recognize a ship by pixels in every lighting condition you can run many many iterations...
Image
XTC Mod Team Veteran. My current work:
Image

csaba
Posts: 295
Joined: Fri, 26. Aug 05, 22:39

Post by csaba » Thu, 4. Jan 18, 21:27

GCP Grey recently had a really fun video explaining how these AI learn. Showing a genetic breeding model:

https://www.youtube.com/watch?v=R9OHn5ZF4Uo&t=16s


He also details recursive neural networking in a follow up video:

https://www.youtube.com/watch?v=wvWpdrfoEv0


I would dread to think how such an AI would change station prices in an X game. It might just change the price right as you hit accept on the transaction and other nasty things.

SirNukes
Posts: 225
Joined: Sat, 31. Mar 07, 23:44

Post by SirNukes » Thu, 4. Jan 18, 22:21

I took a look at the DOTA and SC2 AI you mentioned, but they look really limited. The DOTA bot, somewhat successful, is working on a limited game scenario (1v1, single hero, etc.). I had trouble finding anything about a successful starcraft bot, but there was this from a few months ago: "DeepMind’s existing algorithms, including those that performed with super-human skill across a host of classic Atari titles, 'cannot win a single game against the easiest built-in AI,' in Starcraft II" (link).

The X games have rather complicated environments in terms of number of actors. 10-20 thousand ships, maybe 2 thousand traders, running independent AI routines, as opposed to just two actors in the above games. There might also be issues with the inherent obfuscation of a trained AI's logic, making it hard to patch or mod.

The other thing to keep in mind is that a lot of the X game AI is dumb simply because of simplistic coding. Eg. "pick random ware in my cargo hold and try to sell for average+1 price; otherwise pick random ware in my trade list and try to buy for average-1 price". A lot of improvement can often be had with a few hours of script development, if improvement of NPC AI is needed.

ZaphodBeeblebrox
Posts: 1460
Joined: Mon, 10. Apr 06, 20:35

Post by ZaphodBeeblebrox » Fri, 5. Jan 18, 08:29

The fact is, AI behaviour in X games can be reduced to a set of simple algorithms.

Which is exactly what the programmers require.

Algorithms can be refined, chained together and iterated over to produce seemingly complex autonomous actions.

Emergent properties appear, when a large number of actors interact with each other and their environment.

The "trick" in the X games is the genesis and destruction of actors, two things that (I thought) were never handled very well.

AI on the other hand, seems to me, to be a "sledgehammer to crack a nut."

However there is one area that I think might benefit. That is the FIGHT part of the game. Here I am thinking of large carriers and destroyers.
It was a woman who drove me to drink... you know I never went back and thanked her.

Don't try to outweird me, three-eyes. I get stranger things than you free with my breakfast cereal.

==Troy==
Posts: 61
Joined: Sat, 17. May 08, 12:36

Post by ==Troy== » Fri, 5. Jan 18, 09:01

Killjaeden wrote:
And then you leave your simulation running as the AI learns to play the game.
There is a distinct difference between DOTA, SC2 and X. Dota and SC2 games have and End that is a "win". And this end goal is reached in the scope of minutes. That means iterations have a clear end result, at which AI can decide if its good or bad what it did. X games do not have a "you win" scenario. And they do not end. The maps in SC2 and Dota are static. Ressources always at the same positions. Certain situations repeat themself frequently. In X this is not the case. factories are not always at the same spot, can change dynamically from factors outside of controll of AI, or directly by controll of AI.

Its not impossible. But it will take a lot longer than SC2 or Dota before anything usable comes out. Egosofts ressources are limited. Expecting them to be fore-runners of game AI is ... slightly unrealistic.

If you think its not difficult - go ahead to create a ML AI for X3 or XR. Their universe states are much more static compared to what X4 will be, so it should much less time consuming... show Egosoft a proof of concept and offer collaboration for "intelligent AI dlc" for X4. Thats how you get things done.
And recognise that it is done from pixels
And how good would that work in a 3D environment with dynamic lighting? Completely different level of complexity. Before AI can recognize a ship by pixels in every lighting condition you can run many many iterations...


Reinforcement learning handles continuous rewards without an issue. It does not require win/loss. It just needs something to maximise. I.e. credits for trader or station.

I do not suggest an AI that would learn from pixels in X3, I specifically stated that one would not need to add such complexity here, while the examples I have given did add it and were still able to beat humans, in albeit limited DOTA setting.


csaba wrote:GCP Grey recently had a really fun video explaining how these AI learn. Showing a genetic breeding model:

https://www.youtube.com/watch?v=R9OHn5ZF4Uo&t=16s


He also details recursive neural networking in a follow up video:

https://www.youtube.com/watch?v=wvWpdrfoEv0


I would dread to think how such an AI would change station prices in an X game. It might just change the price right as you hit accept on the transaction and other nasty things.

Just depends on when you allow it to change the prices... Player docked? lock them.

Also ,genetic breeding is a last-century training technique. Nowadays its all about gradient descent and variations of thereof.

SirNukes wrote:I took a look at the DOTA and SC2 AI you mentioned, but they look really limited. The DOTA bot, somewhat successful, is working on a limited game scenario (1v1, single hero, etc.). I had trouble finding anything about a successful starcraft bot, but there was this from a few months ago: "DeepMind’s existing algorithms, including those that performed with super-human skill across a host of classic Atari titles, 'cannot win a single game against the easiest built-in AI,' in Starcraft II" (link).

The X games have rather complicated environments in terms of number of actors. 10-20 thousand ships, maybe 2 thousand traders, running independent AI routines, as opposed to just two actors in the above games. There might also be issues with the inherent obfuscation of a trained AI's logic, making it hard to patch or mod.

The other thing to keep in mind is that a lot of the X game AI is dumb simply because of simplistic coding. Eg. "pick random ware in my cargo hold and try to sell for average+1 price; otherwise pick random ware in my trade list and try to buy for average-1 price". A lot of improvement can often be had with a few hours of script development, if improvement of NPC AI is needed.

https://www.technologyreview.com/s/6092 ... ftfor-now/
Facebook AI did a little better. SC2 is incredibly complex game to ask AI to solve by itself. And it is orders more complex than what I am proposing to let AI handle in X games.

You do not require to have thousands of ships in the state, nor do you need to provide the entire universe to the trader. Throw in a bit of memory and the AI will learn to explore, remmeber and decide which deals it wants to take. Which would lead to a natural behaviour, instead of a predictable scripted one. (especially because you can have many versions of the network running at different stages of trianing / seeds resulting in variable behaviour in identical situations).


-------------------------

To summarise. I do not suggest to use ML AI to solve X-games. My belief is that if simple, primitive ML AI implementations were to be used for individual, very specific circumstances, it could give the game's AI more natural and unpredictable feel, while keeping the development time low and not require huge overhaul of the game's AI in general.

"Solving" X with a single instance of ML AI will take a few more years of research.

User avatar
ubuntufreakdragon
Posts: 3965
Joined: Thu, 23. Jun 11, 14:57

Post by ubuntufreakdragon » Fri, 5. Jan 18, 09:48

What happened to the forum, I expected at least one (fun) comment expressing the fear of the Xenon becoming an actual Xenon. :P

Modern AI is still far away from Xenon, but Neuronal Networks are the right tool, they can be quite effective, and for training you only need time and power so development could be cheaper than the conservative way, and may be much better, I corrected many tradescripts for XR and all these changes become part of patches and these bugs were quite obvious to reading the code with common sense, so any change would be an improvement.

But btw any constant AI will be beaten by a learning "person" given enough time.
My X3 Mods

XRebirth, things left to patch:
In General; On Firing NPC's; In De Vries; Out Of Zone; And the Antiwishlist

User avatar
JSDD
Posts: 834
Joined: Fri, 21. Mar 14, 21:51

Post by JSDD » Fri, 5. Jan 18, 15:39

... AI scripts will be sufficient for me, there is no need for complex machine learning etc ... it will be enough if the other races choose their friends/foes by counting some "bonus value" for each possible race ,the one with the highest bonus will get an "alliance offer" or such ...the bonus value can take into account recent trade volume, accomplished missions and destroyed enemies/ protected friends etc ... as it has been in the previous games (maybe a bit better balanced/scripted). bottom line: there is a dynamic system of friend/foe relations, including another scriptt which checks that not everything gets out of hand (war everywhere with every race, extinguished races, no stable economy etc).

the rules of that dynamics dont have to be dynamic again (aka "learned") as long as they are modifiable

by the way: i too highly doubt that egosoft with its limited manpower will develop artificial machine learning system for the next game (releasd in 2018)
To err is human. To really foul things up you need a computer.
Irren ist menschlich. Aber wenn man richtig Fehler machen will, braucht man einen Computer.


Mission Director Beispiele

User avatar
Killjaeden
Posts: 5023
Joined: Sun, 3. Sep 06, 18:19

Post by Killjaeden » Fri, 5. Jan 18, 18:49

==Troy== wrote:Facebook AI did a little better. SC2 is incredibly complex game to ask AI to solve by itself. And it is orders more complex than what I am proposing to let AI handle in X games.
For simple tasks like a trader, you do not need complex or intelligent AI. You may not even want intelligent AI, because its easier to adjust a weight factor or functionality of something you wrote yourself to dial the behaviour in a better direction. Why would you want that? Because the universe is dynamic. If AI traders are trained for optimal trade runs, sub-optimal trade opportunities can get ignored which can have big impacts on the economy of an entire region.
Also, if the traders are competent in picking out good deals, then the player will have basically no way to get good deals himself.

In addition, barely anybody would notice intelligent NPC traders. Nobody follows them around for ages to observe what they do. The only thing he would propably see is the results, namely that there will be very few good deals to be had.

The only aspects where i can see a place for ML is in very complex tasks, that are hard to define conventionally. Like combat, and also faction controll (ordering aorund the military and police ships effectively, building stations at most suited locations). And these already require a lot more complex behaviour. Combat is fortunately easier to iterate on because each "test" is finished in a matter of minutes, rather than a matter of hours or days as would be the case for faction controll tasks.
Just depends on when you allow it to change the prices... Player docked? lock them.
Then it wouldnt happen to the player but all npc ships... And if you say lock price whenever ships are docked, a trader (or player) could dock a ship permanently to keep constant price.
Image
XTC Mod Team Veteran. My current work:
Image

SirNukes
Posts: 225
Joined: Sat, 31. Mar 07, 23:44

Post by SirNukes » Fri, 5. Jan 18, 20:39

ubuntufreakdragon wrote:What happened to the forum, I expected at least one (fun) comment expressing the fear of the Xenon becoming an actual Xenon. :P
It would tremendously amuse me if Egosoft used ML to program only the Xenon AI, while the other races use normal scripts.

Post Reply

Return to “X4: Foundations”