[Program] X3 Map Surgeon 1.0

[solarahawk]

X3 Map Surgeon v1.0

The Map Surgeon is a collection of Python utilities and related data files designed to make it less labor-intensive to develop and update new galaxy maps. Using an Excel spreadsheet editor, sectors can be assigned to new coordinates, and new gate networks can be mapped out, including connections between distant sectors.

The provided command-line programs process different data files that are needed for a new galaxy map. These utilities generate support files needed by X3, such as an updated Jobs file or ReadText file containing the remapped sector names and descriptions. Other utilities process your exemplar remapped galaxy file in order to generate new schema data for subsequent version updates of your map. This can save you from having to repeat any hand adjustments you make to sector designs.

Full instructions on using Map Surgeon utilities and editing the Remap Schema spreadsheet are included in the package.


What Can Map Surgeon Do?

At the most basic level, it is possible to rearrange the sectors of a galaxy map, whether that map is the vanilla X3:AP map or the LU Vanilla map*, and generate a new layout with the sectors in any arrangement you like. You can add or remove any gates in any sectors and connect those gates into a new gate network. Even without touching the in-game Galaxy Editor, the Map Surgeon is capable of generating a new map that is functional, and even when placing new gates in sectors that previously did not have gates in those positions, it will place the new gates as well as possible. (The current default placement method is very simplistic, but it will eventually be upgraded with a semi-intelligent placement routine that attempts to avoid possible object collisions.)

More advanced map design goals, like creating new sectors, still require using the in-game Galaxy Editor, but once these new sectors have been designed, the polished galaxy map can be exported and processed by the Map Surgeon. Using the custom sector data in the exported map, it will generate schema files that capture those features. These schema files can then be used to regenerate the polished map in future version updates of the map.

Automated version updates can save a tremendous amount of time for a modder who needs to maintain support for new versions of the original map their mod is based upon. For example, when new versions of Litcube's Universe are released, any map changes in the version update can be captured and included in the updated map mod.

* NOTE: The Map Surgeon should be compatible with X3:TC, X3:AP, as well as any modded version of the game, like LU. However, a new galaxy map will likely break any missions in that game. Thus, creating a new galaxy will either require updating all missions, or restricting yourself to sandbox play only.

My development focus is solely on Litcube's Universe. I can make no guarantees for reliable support outside of Litcube's Universe.


What Does Map Surgeon Not Do?

Apart from adjusting gates, the current version of Map Surgeon does not alter the contents of sectors. It is solely focused on rearranging existing sectors into new layouts and reconnecting their gates as appropriate. A modder can create a new sector design and have it added to the map, and sectors can be removed. So, if a modder wishes to change the contents of a sector, that can be achieved by dropping the old sector and adding a new sector using the old sector name and assigning it to the desired coordinates.


Where is the Graphical Interface Version of Map Surgeon?

Alas, I am not so skilled with designing GUI applications. For this task, I prefer the command-line anyway. However, if someone wants to work on building a pretty graphical interface for Map Surgeon, they are more than welcome.


What Is Planned For The Future?

The Map Surgeon has laid the groundwork for a new utility I plan on releasing in coming months: an automated map generator. I have mapped out some of the algorithms and rules this tool will need. The goal for this "Auto-Map Surgeon" is to be able to generate a semi-random galaxy map on the fly. I have been bouncing some ideas off of vukica, and I anticipate this tool will allow anyone ready to start a new game to generate a new galaxy for a fresh game experience. More details will be available once I near a testable prototype, which may be a few months down the road. (I have some other projects I want to wrap up first.)


Downloads

Want to read more? The PDF README and a Guide to using Map Surgeon are both available for direct download as well as being included in the program folder itself.

README:
[ external image ]

Guide to the Map Surgeon:
[ external image ]


Download Retracted
Critical errors found in the current version. I am retracting it until I can fix it.

Installation

Map Surgeon requires Python 2.7, plus a several addon modules which must be installed separately. The easiest way to install Python in Windows and fulfill these dependencies is by installing the free Anaconda Python Distribution.

If you choose to use the standard Python.org distribution, Map Surgeon requires the following Python modules, which can be installed via 'pip install <modulename>':

* numexpr
* xlrd
* pandas

[solarahawk]

Reserved

[Joubarbe]

"Automated Map Generator", that would be cool !

Why did you use Python exactly for this ? For the Excel stuff ?

Just asking

[solarahawk]

Python is just convenient and quite capable of the text and XML manipulation I needed. With Python, I was able to get this tool up and running in a weekend. Most of the time spent on it since then has just been for polishing, documenting, and making it more robust for public consumption.

Originally, I was exporting the worksheets into CSV and importing those files into the tools, but to make things even easier to use, I decided to skip the export step and just load the Excel file itself.

[vukica]

wooohoo! It's out!

[Tharrg]

Looks like an intersting tool, what happened?

I was thinking of building something on top of it.

[enenra]

Oooh random galaxy generation. Fancy.

I've done some research into that myself. However, I'd much rather like to generate the galaxy ingame, on the fly. I've got some old prototypes lying around somewhere in my MD folder. If there's any interest in that (considering some ideas could be reappropriated for an external generator too) I can upload those files into a dropbox or something.

On a semi-related note: LU has some text file entries suggesting an ability to create a bod file for the universe map. Is that actually the case?

[solarahawk]

Well, you can't "build" a galaxy from scratch dynamically during a game, but you could make it look like the galaxy is growing by activating gates like the Xtended team did in their mod. The sectors have to already exist at game start.

LU includes Litcube's script (also available on the forums as a download) for generating a new BOD file. I forget its name off the top of my head, but when you run it from the script console, it generates a new BOD and saves it to a text file. You just have to rename it and move it to the correct folder.

[enenra]

Well, you can't "build" a galaxy from scratch dynamically during a game, but you could make it look like the galaxy is growing by activating gates like the Xtended team did in their mod. The sectors have to already exist at game start.

That's partially true. With MD it's possible to load maps into the game to add to the current map. Thus it would technically be possible, if convoluted and messy. No, the thing I have in mind is take a default map with every sector possible created but without anything else and no objects in them. Then populate the sector with objects dynamically - either by filling all the sectors up front or adding over time. As the player cannot access a sector unless he knows it's there anyway, the additional sectors that technically exist but cannot be accessed make no difference really.

I have prototype code for populating a sector with "natural" objects like suns, planets, asteroids etc. as a MD library. The other parts are more or less planned out but not put into code yet.

What we did with XTC is different in the sense that everything exists from the beginning except for the gate connections. Upon a connection, the target sector is wiped of stations, which are afterwards rebuilt over time.

[solarahawk]

Oh, very interesting. It would be cool to see something like that. I am hoping to get back to the Map Surgeon and fix the bugs in it. Need to see if I can budget a little python debugging time around my dissertation.

[enenra]

Oh, very interesting. It would be cool to see something like that. I am hoping to get back to the Map Surgeon and fix the bugs in it. Need to see if I can budget a little python debugging time around my dissertation.

I can upload the code but to give a quick overview:

The old plan was to just have a file that creates sector clusters and then connects them with "supergates" (just simply gates that go over a bit further a distance). Then have that file call a library on each sector that fills the sector with stuff, in a more or less randomized way. The idea to make those sectors look kind of okay was to assign clusters of sectors a "mood" - pretty much simply a primary color. I sorted all the backgrounds, planets, suns, etc. by primary color and then saved those lists into t-files for easy access. When creating a sector, I'd pull from those lists to create a cohesive sector.
There are various problems with that and I haven't worked on that code in a couple years but the basic idea works.

The current plan is to part all of this up a bit more:

On gamestart, the player will spawn with nothing into an empty sector, get a message which then opens up a menu. In the menu, he can customize the map creation and gamestart - I'm thinking of this a bit alike to embarking in Dwarf Fortress.

After that, the information is fed into a master file which organizes all further galaxy generation. First it creates the clusters etc. of sectors (I still have to figure out a way to generate those clusters properly but that's just a matter of sitting down and figuring it out). Then it links the sectors. Finally it decides on which area is the starting area of each race and adjusts mood etc. according to that.

Then another part of code does a balance pass over the galaxy in terms of sun intensity and asteroid count / yield. This is all still in variables, no actual objects are created. The goal is that all races are going to start out at about even footing resource-wise.

Next the sectors are filled with objects. I want to make sure I can generate some interesting asteroid belts etc. and not place planets in front of suns and so on, so that will require some extensive tinkering but I'll use old code as a base.

Next up, stations are placed. Depending on whether I support gradual expansion or not, all sectors are filled at game start or it happens gradually with the races expanding from their home sector. We'll see.

Finally, I got some other libraries in the work right now but while those are needed for my stuff, it doesn't really relate that much to galaxy generation in vanilla / other mods.

[vukica]

Oh, very interesting. It would be cool to see something like that. I am hoping to get back to the Map Surgeon and fix the bugs in it. Need to see if I can budget a little python debugging time around my dissertation.

lost colony mod had in one of the older versions loading three additional maps.

RSM creates and adds sectors "dynamically".

one of my ideas for surgeon_remap_kind_of_mod was:

A) designa a "lib" with a large number of sectors (suns, planets, backgrounds, roids, not much else)
B) use map "loader" to randomly select pre-created sector and place it at a random free location in the galaxy... every 5-6 hours.
C) populate sector slowly (with constraints, we don't want 50 ore mines if there happen to be 50 good roids in the sector)

not place planets in front of suns and so on

why not

[enenra]

not place planets in front of suns and so on

why not

The reason is quite simply that suns have a higher render priority (or whatever that'd be called) - basically they shine through planets from what I remember. Which looks pretty silly.

Some good ideas for the mod btw. - personally I just like the idea of generating everything to create a new experience every time. Essentially, take the game from a skyrim-like approach with an open, but preset world to a dwarf fortress like approach where the basic ideas are the same but due to everything being generated, each playthrough is different. To encourage the players to play differently depending on the situation in each new game they start.

[vukica]

not place planets in front of suns and so on

why not

The reason is quite simply that suns have a higher render priority (or whatever that'd be called) - basically they shine through planets from what I remember. Which looks pretty silly.

Some good ideas for the mod btw. - personally I just like the idea of generating everything to create a new experience every time. Essentially, take the game from a skyrim-like approach with an open, but preset world to a dwarf fortress like approach where the basic ideas are the same but due to everything being generated, each playthrough is different. To encourage the players to play differently depending on the situation in each new game they start.

so don't shine. light source can be placed behind the planet. see sector "dual eclipse" (op in remap thread, i think)
all you need is a "dummy star".

generating new sector every time would be... problematic. i think i had that discussion with someone, possibly myself...
some pattern would have to be used. a "sector type" if you will. Asteroid belt belt, gas giant w moons, earth like, some stupid rock, some other stupid rock, binary planets, hot jupiter, weird binary stars.

[enenra]

so don't shine. light source can be placed behind the planet. see sector "dual eclipse" (op in remap thread, i think)
all you need is a "dummy star".

generating new sector every time would be... problematic. i think i had that discussion with someone, possibly myself...
some pattern would have to be used. a "sector type" if you will. Asteroid belt belt, gas giant w moons, earth like, some stupid rock, some other stupid rock, binary planets, hot jupiter, weird binary stars.

Yeah, but the whole point of my approach is that everything is generated (see my posts above), hence the problem with suns behind planets. Need 3D math to make sure suns aren't placed behind planets and then shine through them.

Generating everything is not hard. The hard part is to make it look good. That's what I'm working on. Or have been and will again in the future. Not a high priority for me at the moment though. Got other stuff brewing.

[vukica]

Yeah, but the whole point of my approach is that everything is generated (see my posts above), hence the problem with suns behind planets. Need 3D math to make sure suns aren't placed behind planets and then shine through them.

Generating everything is not hard. The hard part is to make it look good. That's what I'm working on. Or have been and will again in the future. Not a high priority for me at the moment though. Got other stuff brewing.

but patterns are cheaper than geometry.

perhaps, object group patterns would be even better than sector patterns. Link planets moons and stars. change their type at random, and change their relative positions randomly within given constraints. then rotate entire pattern, again randomly.

do the same with roids.

[enenra]

Definitely cheaper but my plan is to do it at gamestart for all sectors, not progressively when sectors are discovered. That way, the player might have to wait a bit longer at gamestart, but during the game, it won't make any difference. IMO acceptable.

[vukica]

Definitely cheaper but my plan is to do it at gamestart for all sectors, not progressively when sectors are discovered. That way, the player might have to wait a bit longer at gamestart, but during the game, it won't make any difference. IMO acceptable.

oh, i see.

eitherway, it shouldn't' actually block player at game start.

[enenra]

Definitely cheaper but my plan is to do it at gamestart for all sectors, not progressively when sectors are discovered. That way, the player might have to wait a bit longer at gamestart, but during the game, it won't make any difference. IMO acceptable.

oh, i see.

eitherway, it shouldn't' actually block player at game start.

Well, my plan was to change gamestart around quite a bit. Essentially, you'd get a number of tokens / credits at gamestart that you could then spend on ships / upgrades / etc. You'd also pick a race, name, affiliation and stuff like that and then also decide on the parameters for galaxy generation. Essentially, it'd allow you to dynamically "create" your own gamestart instead of them being defined ahead of time and you choosing one.

So the player would be in a menu at gamestart either way - making him wait a bit for galaxy generation shouldn't be much of an issue, especially as the initial loading in would be really quick because the map is empty at that point.

[wesley32]

Hi solarahawk,

Do you have an ETA on re-release? Can't wait to use your tool!