Any Way to easily extract JUST station data from a save file?

[Duncaroos]

I'm playing around with an empire logistic tool, and I am able to import a construction plan to pre-populate stations and module assignments. I would like to carry this over to importing stations from game files into this tool, however I don't see a way of only extracting player station data from a save file, given the size of the xml.

ANy thoughts on how to do this? For the Construction Plan import, I used Excel VBA to load a construction plan xml and map the important bits of the xml file I need and extract the module data. Would like to use the same to import save data into this tool, but the xml file would have to remove any non-player station data to not be extremely slow (I don't even think Excel VBA can open up an entire uncompressed save xml file anyways).

[Baconnaise]

Depends on the modules on the station so you would look it up via the macro name of say argon terran split boron etc. You would then look for the owner=player tag. That should get you the first line or a first line on one of the modules on the station so you can go from there. I think rogeuys x4 site still lists macro names or just look manually in the save file at first. You should be able to grab all or most it.

VS code opens the XML saves fine as does notepad++.

EDIT: Meant to say you can simply search for owner=player tags instead unless it's an old save with lots of player owned bits and bobs. Tossed a quick example for the PHQ and the owner tag.

Code: Select all

<component class="station" macro="station_pla_headquarters_base_01_macro" owner="player" 

[Duncaroos]

Thanks for the input.

Excel has only 1,048,576 rows available, but a midgame save file has 16 million+ lines in the save file (order of magnitude). Even if I try to use a ListObject Table using the Workbooks.OpenXML(...) w/ 'xlXmlLoadImportToList' setting - say there are 10 sub-levels within the XML file (so 10 ListColumns)...that is still going to result in 1.6 million+ Excel rows that will not be able to load. I tried loading my save XML to Excel, and by the gods it is way too slow for other users to even want to wait for it just for the VBA code to just load the XML file, let alone extract and manipulate the data of each player station. Example of such a manipulation is there is a lack of consistency between a module's Ware ID and the Macro ID (e.g. 'module_arg_pier_l_01' --> 'pier_arg_harbor_01_macro', or 'dockarea_arg_m_station_01_macro' --> 'module_arg_dock_m_01'), which requires additional VBA code to create a translation rule to get the Ware ID from the Macro ID (which is not always consistent for all modules either).

Looking manually into one of my midgame vanilla save files, I was hoping there would have been a [station] ... [/station] section within a save file that covers all stations and its module breakdown and sector location...but seems like the save files are the wild west and stations are just added to the bottom as it comes in, so there is no way for me to say get a start / end row and extract to greatly decrease the size of the table and/or loading times (even though I question would even still be a possibility, given Excel only can read a limited # of rows). Sector location doesn't even look like it is in the station data - it is off somewhere else that I have yet to find.

Would die to know if there is some kind of default XML save file structure that I could see, so that I can at least figure out high-level where to look and maybe find something that I can work with.

By the time I finished this message, it took like 15-20 minutes for a 738MB save file to load...and it only gave me one row of data that did not contain anything useful...so unsure what to do

[Duncaroos]

So seems like the structure that I have to 'drill down' to is as follows (hopefully Excel VBA allows me to do this...):

Code: Select all

'>>>Request Save file from User<<<

'XML Table Import
  Data Query Drill Down -> "universe" -> "component" -> "connections" -> "connection"

      'This area shows "Cluster" / System information...got to loop through each one:

      For i = 1 To nRows("connection")
      
          Data Query Drill Down -> "component"_i      'i.e. current components of cluster
          Data Query Drill Down -> "connections" -> "connection"
                    
          'This area contains information on the Region and/or Sectors....need to look through each 'Sector'
                        
          For j = 1 To nRows("connection")
          
              If "connection Attribute:connection"_j Contains "_sector" Then
              
                  Data Query Drill Down -> "component"_j      'i.e. current components of sector
                  >>>Store "component Attribute:macro"_j         'i.e. macro name for zone (to find Sector Name)
                  
                  Data Query Drill Down -> "connections"  -> "connection"
                      
                  'This area contains information on Sector 'Zones'....need to look through each
                                           
                  For k = 1 To nRows("connection")
                  
                      If "connection Attribute:connection"_k Contains "zone" Then
                      
                          Data Query Drill Down -> "component"_j          'i.e. current components of Sector's 'Zone'
                          
                          Data Query Drill Down -> "connections" -> "connection"
                                       
                          'This area contains list of build storages, ships, stations, satellies
                                                           
                          For x = 1 To nRows("connection")
                                  
                              If "connection Attribute:connection" Contains "stations" Then
                                      
                                  Data Query Drill Down -> "component"_x    'i.e. current station group

                                  If "component Attribute:owner" Equals "player" Then
                                          
                                      >>>Store "component Attribute:name"    'i.e. station name by player
                                          
                                      Data Query Drill Down -> "construction" -> "sequence" -> "entry"
                                      
                                      'This area contains list of station modules

                                      For y = 1 To nRows("entry")
                                      
                                          >>> Store "entry Attribute:macro"_y    'i.e. macro ID for module
                                          
                                      Next y
                                  End If
                              End If
                          Next x
                      End If
                  Next k
              End If
          Next j
      Next i

Hopefully I got this down pat. I had to write this down somewhere...will be good reference when I make the VBA. Anyone that has good reference on using Excel VBA with "Drilling Down" instructions for query editor prior to load, it would be appreciated. When I do this drill down using Excel, it still is loading the entire file; this is gonna be fun lol.

Using a map schema this looks accurate. Gotta check construction plots though

[Baconnaise]

I hope you're able to figure it out. One last thing to suggest is to create a new save file from scratch with a player-owned station. New saves should be easier to deal with AFAIK.

[Captain_Canard]

Do you have any experience with languages like Python? I've been writing my own Python-based XML parser specifically for the X4 save file, does all the "drilling down" for me. Python has some nice XML libraries that can handle the large save file, though I did have to hand code some missing features like tracking the "family tree" hierarchy (the relationship between parent nodes and their children). Right now I'm using it to easily extract the specific elements of interest (owner = player) and display the data in an easy-to-read format, but eventually I should be able to use it to modify that data. But for right now I just find what I'm looking for, which helps me then edit the raw XML in Notepad++

[Duncaroos]

Do you have any experience with languages like Python? I've been writing my own Python-based XML parser specifically for the X4 save file, does all the "drilling down" for me. Python has some nice XML libraries that can handle the large save file, though I did have to hand code some missing features like tracking the "family tree" hierarchy (the relationship between parent nodes and their children). Right now I'm using it to easily extract the specific elements of interest (owner = player) and display the data in an easy-to-read format, but eventually I should be able to use it to modify that data. But for right now I just find what I'm looking for, which helps me then edit the raw XML in Notepad++

I don't know Python, but I've heard of it and is really something I SHOULD learn given my field of engineering is starting to use it more and more.

I have learned though that for Excel VBA...instead of using:

Code: Select all

Dim curXMLFile As Workbook
    Set curXMLFile = Workbooks.OpenXML(FullFilePath, , xlXmlLoadImportToList)

I should instead use:

Code: Select all

Dim XMLFile As Object
    Set XMLFile = CreateObject("MSXML2.DOMDocument.6.0")
        XMLFile.async = False
        XMLFile.validateOnParse = False
    XMLFile.Load(FullFilePath)

This new code would allow me to "Drill Down" the save file's XML to the area I need and not pre-load the save file as a workbook and cause all sorts of issues. This comes to a task of just creating For Loops to get to where I need to:

Code: Select all

For Each XMLChild1 In XMLFile.DocumentElement.ChildNodes
    If XMLChild1.BaseName = "universe" Then
        For Each XMLChild2 In XMLChild1.ChildNodes
            If XMLChild2.BaseName = "component" Then
                For Each XMLChild3 In XMLChild2.ChildNodes
                    If XMLChild3.BaseName = "connections" Then
                        For Each XML_Cluster In XMLChild3.ChildNodes
                            If XML_Cluster.BaseName = "connection" Then
                                ...                            
                            End If
                        Next XML_Cluster
                        GoTo endloop
                    End If
                Next XMLChild3
                GoTo endloop
            End If
        Next XMLChild2
        GoTo endloop
    End If
Next XMLChild1

endLoop:
'...

Had to take a bit of hiatus from this to have to deal with work projects, but I think I'm on the right track . Still have to understand the parameter "XMLFile.async" and whether I want True or False for this. If True, apparently the full XML file is not fully "loaded", and the opposite for False....I have not seen much improvement in speed to complete a save file import, but I have to complete my For Loop mayhem to time it. It would be nice if I could select a specific Child Node based on its BaseName, but I have not found a way to do that yet.

[Duncaroos]

BEHOLD ... only needed 11 nested loops to get there but it works! :

Code: Select all

  For Each XMLChild1 In XMLFile.DocumentElement.ChildNodes
    If XMLChild1.BaseName = "universe" Then                                             'Want the sub-area for "Universe" info only.
      For Each XMLChild2 In XMLChild1.ChildNodes
        If XMLChild2.BaseName = "component" Then                                    'Want the various components of the universe
          For Each XMLChild3 In XMLChild2.ChildNodes
            If XMLChild3.BaseName = "connections" Then                          'Want the various connections (Clusters/Systems) of the universe
              For Each XML_Cluster In XMLChild3.ChildNodes
                If XML_Cluster.BaseName = "connection" Then                 'Want only "connection" types under "connections" - i.e. a Cluster/System
                  For Each XMLChild4 In XML_Cluster.ChildNodes
                    If XMLChild4.BaseName = "component" Then                                    'Want the various components within a Cluster
                      For Each XMLChild5 In XMLChild4.ChildNodes
                        If XMLChild5.BaseName = "connections" Then                          'Want the various connections (Sectors) of the current Cluster/System
                          For Each XML_Sector In XMLChild5.ChildNodes
                            If XML_Sector.BaseName = "connection" Then                  'Want only "connection" types under "connections" - i.e. a Sector/Region of a Cluster/System
                              On Error Resume Next
                                Err = 0
                                ErrHandleVal = Empty
                                ErrHandleVal = .Find("_sector", XML_Sector.Attributes(0).Value)                 'Sectors have unique lookup value containing "_sector"
                                If Err <> 0 Then
                                  GoTo skipXMLSector
                                Else
                                  ErrHandleVal = .Find("_region", XML_Sector.Attributes(0).Value)
                                  If Err = 0 Then GoTo skipXMLSector
                                  Err = 0
                                  For Each XMLChild6 In XML_Sector.ChildNodes
                                    If XMLChild6.BaseName = "component" Then                                'Want the various components within a Sector
                                      For Each XMLChild7 In XMLChild6.ChildNodes
                                        If XMLChild7.BaseName = "connections" Then                                          'Want the various connections (Zones) within the current Sector
                                          For Each XML_Zone In XMLChild7.ChildNodes
                                            On Error Resume Next
                                              Err = 0
                                              ErrHandleVal = Empty
                                              ErrHandleVal = .Find("zone", Left(XML_Zone.Attributes(0).Value, 4))     'Zones containing stations have unique lookup value beginning "zone" (i.e. "zone###_*" or "zones")
                                              If Err <> 0 Then
                                                GoTo skipXMLZone
                                              Else
                                                For Each XMLChild8 In XML_Zone.ChildNodes
                                                  If XMLChild8.BaseName = "component" Then                                            'Want the various "components" within the current Zone.
                                                    For Each XMLChild9 In XMLChild8.ChildNodes
                                                      If XMLChild9.BaseName = "connections" Then                                  'Want the various "connections" within the current Zone.
                                                        For Each XML_Station In XMLChild9.ChildNodes
                                                          If XML_Station.BaseName = "connection" Then                         'Want only "connection" types under "connections" - i.e. an object within the current Zone.
                                                            If XML_Station.Attributes(0).Value = "stations" Then            'Want station objects only.
                                                              For Each XML_StationData In XML_Station.ChildNodes
                                                                If XML_StationData.BaseName = "component" Then          'Want component information for current station within the current Zone.
                                                                  'Confirm station owner is player.
                                                                    For Each XMLTempAttribute In XML_StationData.Attributes
                                                                      If XMLTempAttribute.BaseName = "owner" Then
                                                                        If XMLTempAttribute.Value <> "player" Then
                                                                          GoTo skipXMLStation                             'If not player - skip station
                                                                        Else
                                                                          Exit For                                        'If player - exit owner loop & continue
                                                                        End If
                                                                      End If
                                                                    Next XMLTempAttribute

                                                                  'Store Player Station Location Info:
                                                                    ReDim Preserve Clusters(1 To UBound(Clusters) + 1)
                                                                    ReDim Preserve Sectors(1 To UBound(Sectors) + 1)
                                                                    ReDim Preserve Zones(1 To UBound(Zones) + 1)
                                                                    ReDim Preserve StationName(1 To UBound(StationName) + 1)
                                                                    ReDim Preserve BuildMethod(1 To UBound(BuildMethod) + 1)
                                                                    ReDim Preserve FillToMax(1 To UBound(FillToMax) + 1)
                                                                    ReDim Preserve Modules(1 To UBound(Modules) + 1)
                                                                    If Err <> 0 Then
                                                                      ReDim Clusters(1 To n_initial)
                                                                      ReDim Sectors(1 To n_initial)
                                                                      ReDim Zones(1 To n_initial)
                                                                      ReDim StationName(1 To n_initial)
                                                                      ReDim BuildMethod(1 To n_initial)
                                                                      ReDim FillToMax(1 To n_initial)
                                                                      ReDim Modules(1 To n_initial)
                                                                      Err = 0
                                                                    End If
                                                                    
                                                                    Clusters(UBound(Clusters)) = XMLChild4.Attributes(1).Value          'Get cluster macro name where station is located.
                                                                    Sectors(UBound(Sectors)) = XMLChild6.Attributes(1).Value            'Get sector macro name where station is located.
                                                                    Zones(UBound(Zones)) = XMLChild8.Attributes(1).Value                'Get zone macro name where station is located.
                                                                  
                                                                  'Store Player Station Name
                                                                    For Each XMLTempAttribute In XML_StationData.Attributes
                                                                      If XMLTempAttribute.BaseName = "name" Then
                                                                        StationName(UBound(StationName)) = XMLTempAttribute.Value
                                                                        Exit For                                                    'Exit loop once Station Name is obtained.
                                                                      End If
                                                                    Next XMLTempAttribute
                                                                    
                                                                  'Get Station Build Method Setting
                                                                    For Each XMLChild10 In XML_StationData.ChildNodes
                                                                      If XMLChild10.BaseName = "build" Then
                                                                        BuildMethod(UBound(BuildMethod)) = XMLChild10.Attributes(0).Value
                                                                        Exit For                                                    'Exit loop once Station's Custom Build Method is obtained.
                                                                      End If
                                                                    Next XMLChild10
                                                                  
                                                                  'Get Habitat Fill Setting
                                                                    For Each XMLChild10 In XML_StationData.ChildNodes
                                                                      If XMLChild10.BaseName = "workforces" Then
                                                                        For Each XMLTempAttribute In XMLChild10.Attribute
                                                                          If XMLTempAttribute.BaseName = "fillcapacity" Then
                                                                            If XMLTempAttribute.Value = 1 Then FillToMax(UBound(FillToMax)) = True
                                                                            Exit For
                                                                          End If
                                                                        Next XMLTempAttribute
                                                                        Exit For
                                                                      End If
                                                                    Next XMLChild10
                                                                  
                                                                  'Get Module Data
                                                                    For Each XMLChild10 In XML_StationData.ChildNodes
                                                                      If XMLChild10.BaseName = "construction" Then                    'Want construction data of current station.
                                                                      
                                                                        For Each XMLChild11 In XMLChild10.ChildNodes
                                                                          If XMLChild11.BaseName = "sequence" Then                'Want module sequence data of current station.
                                                                          
                                                                            'Clear & Reset TempModules array
                                                                              Erase TempModules
                                                                              ReDim TempModules(1 To n_initial)
                                                                              
                                                                            'Get Data
                                                                              For Each XML_Module In XMLChild11.ChildNodes
                                                                                If TempModules(UBound(TempModules)) <> "" Then ReDim Preserve TempModules(1 To UBound(TempModules) + 1)
                                                                                TempModules(UBound(TempModules)) = XML_Module.Attributes(2).Value
                                                                              Next XML_Module
                                                                              
                                                                            'Exit
                                                                              Exit For
                                                                          End If
                                                                        Next XMLChild11
                                                                        Exit For
                                                                      End If
                                                                    Next XMLChild10
                                                                    
                                                                    If UBound(TempModules) >= 1 Then Modules(UBound(Modules)) = TempModules
                                                                  
                                                                  'Exit
                                                                    Exit For                    'Leave XML_StationData - Finished collecting data.
                                                                End If
                                                              Next XML_StationData
                                                            End If
                                                          End If
                                                          'Continue                                       'Continue looping through all connections (Stations) until complete.
skipXMLStation:                                         Next XML_Station
                                                        Exit For            'Leave XMLChild9 - no longer within "connnections" area of current Zone.
                                                      End If
                                                    Next XMLChild9
                                                    Exit For                    'Leave XMLChild8 - no longer within "components" area of current Zone.
                                                  End If
                                                Next XMLChild8
                                              End If
                                            On Error GoTo 0
                                            'Continue                                   'Continue looping through all connections (Zones) until complete.
skipXMLZone:                              Next XML_Zone
                                          Exit For            'Leave XMLChild7 - no longer within "connections" area of current Sector.
                                        End If
                                      Next XMLChild7
                                      Exit For                    'Leave XMLChild6 - no longer within "component" area of current Sector.
                                    End If
                                  Next XMLChild6
                                End If
                              On Error GoTo 0
                            End If
                            'Continue                                       'Continue looping through all connections (Sectors) until complete.
skipXMLSector:            Next XML_Sector
                          Exit For                'Leave XMLChild5 - no longer within "connections" area of current Cluster/System.
                        End If
                      Next XMLChild5
                      Exit For                        'Leave XMLChild4 - no longer within "component" area of current Cluster/System.
                    End If
                  Next XMLChild4
                End If
                'Continue                                   'Continue looping through all connections (Clusters) until complete.
              Next XML_Cluster
              GoTo endloop        'Leave XMLChild3/2/1 - no longer within "connections" area of Universe.
            End If
          Next XMLChild3
          GoTo endloop                'Leave XMLChild2/1 - no longer within "component" area of Universe.
        End If
      Next XMLChild2
      GoTo endloop                        'Leave XMLChild1 - no longer within "Universe" section of save file.
    End If
  Next XMLChild1
endloop:
'...

[Captain_Canard]

BEHOLD ... only needed 11 nested loops to get there but it works!

Or just one recursive function.

[Duncaroos]

True, but I could never get my head around recursive stuff . I was able to find a way to reference the ChildNodes directly based on BaseName...helped a lot in making that mayhem much more easier to handle:

Code: Select all

 For Each XML_Cluster In XMLFile.DocumentElement.SelectSingleNode("universe").SelectSingleNode("component").SelectSingleNode("connections").ChildNodes
  For Each XML_Sector In XML_Cluster.SelectSingleNode("component").SelectSingleNode("connections").ChildNodes
    If Not (IsError(.Find("sector", XML_Sector.GetAttribute("connection")))) And IsError(.Find("region", XML_Sector.GetAttribute("connection"))) Then
      For Each XML_Zone In XML_Sector.SelectSingleNode("component").SelectSingleNode("connections").ChildNodes
        If Not (IsError(.Find("zone", Left(XML_Zone.GetAttribute("connection"), 4)))) Then
          If XML_Zone.SelectSingleNode("component").SelectSingleNode("connections") Is Nothing Then GoTo skipZone
          For Each XML_Station In XML_Zone.SelectSingleNode("component").SelectSingleNode("connections").ChildNodes
            If XML_Station.GetAttribute("connection") = "stations" Then
              If XML_Station.SelectSingleNode("component").GetAttribute("owner") = "player" Then
                'Store Player Station Location Info:
                  On Error Resume Next
                    ReDim Preserve Clusters(1 To UBound(Clusters) + 1)
                    ReDim Preserve Sectors(1 To UBound(Sectors) + 1)
                    ReDim Preserve Zones(1 To UBound(Zones) + 1)
                    ReDim Preserve StationName(1 To UBound(StationName) + 1)
                    ReDim Preserve OvrBuildMethod(1 To UBound(OvrBuildMethod) + 1)
                    ReDim Preserve FillToMax(1 To UBound(FillToMax) + 1)
                    ReDim Preserve Modules(1 To UBound(Modules) + 1)
                    If Err <> 0 Then
                      ReDim Clusters(1 To n_initial)
                      ReDim Sectors(1 To n_initial)
                      ReDim Zones(1 To n_initial)
                      ReDim StationName(1 To n_initial)
                      ReDim OvrBuildMethod(1 To n_initial)
                      ReDim FillToMax(1 To n_initial)
                      ReDim Modules(1 To n_initial)
                      Err = 0
                    End If
                  On Error GoTo 0
                
                'Get Station General Info / Settings
                  'Location
                    Clusters(UBound(Clusters)) = XML_Cluster.SelectSingleNode("component").GetAttribute("macro")            'Get cluster macro name where station is located.
                    Sectors(UBound(Sectors)) = XML_Sector.SelectSingleNode("component").GetAttribute("macro")               'Get sector macro name where station is located.
                    Zones(UBound(Zones)) = XML_Zone.SelectSingleNode("component").GetAttribute("macro")                     'Get zone macro name where station is located.
                  
                  'Station Name
                    If IsNull(XML_Station.SelectSingleNode("component").GetAttribute("name")) Then
                      'Debug how to get auto name
                      StationName(UBound(StationName)) = XML_Station.SelectSingleNode("component").GetAttribute("code")   '??? How does game determine name then ???
                    Else
                      StationName(UBound(StationName)) = XML_Station.SelectSingleNode("component").GetAttribute("name")
                    End If
                    
                  'Station Settings
                    On Error Resume Next
                      OvrBuildMethod(UBound(OvrBuildMethod)) = XML_Station.SelectSingleNode("component") _
                                            .SelectSingleNode("build").GetAttributes("method")      'Get station's build method override.
        
                      FillToMax(UBound(FillToMax)) = XML_Station.SelectSingleNode("component") _
                                      .SelectSingleNode("workforces").GetAttribute("fillcapacity")        'Get station's Habitat 'Fill to Max' setting.
                    On Error GoTo 0
              
                  'Get Module Data
                    Erase TempModules
                    ReDim TempModules(1 To n_initial)
                    
                    If XML_Station.SelectSingleNode("component").SelectSingleNode("construction") Is Nothing Then GoTo skipModules
                    For Each XML_Module In XML_Station.SelectSingleNode("component").SelectSingleNode("construction").SelectSingleNode("sequence").ChildNodes
                      If TempModules(UBound(TempModules)) <> "" Then ReDim Preserve TempModules(1 To UBound(TempModules) + 1)
                      TempModules(UBound(TempModules)) = XML_Module.GetAttribute("macro")
                    Next XML_Module          
skipModules:
                    Modules(UBound(Modules)) = TempModules
              End If
            End If
          Next XML_Station
        End If
skipZone:
        Next XML_Zone
    End If
  Next XML_Sector
Next XML_Cluster