[TOOL] X4LogAnalyzer - Help analyse the trade log

The place to discuss scripting and game modifications for X4: Foundations.

Moderators: Moderators for English X Forum, Scripting / Modding Moderators

P.Hastig
Posts: 20
Joined: Thu, 14. Jun 07, 14:19
xr

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by P.Hastig » Tue, 8. Jan 19, 20:44

magictrip wrote:
Mon, 7. Jan 19, 01:34
Hello P.Hastig. I was able to find a bug that was introduced when I added support for TaterTrade. Please download the new version and it should work for you now.

Please let me know if you are still having problems.

PS: I added your files in the new version so other people can use it if needed :-)
Thank you,

the new version now can handle german savegames, but there is still a little issue.
It seems, that product-strings with special characters like ä, ö, ü can't be parsed.
That's strange, because destination-strings like "ARG Hüllenteilefabrik I" where parsed correctly.

See here..:
Image

..and the corresponding rows in the savegame file:

Code: Select all

<entry time="147056.314" title="Handel abgeschlossen" text="Hermes Angreifer QRT-394 verkaufte 50 Sensorenbündel an ARG Argon Raumdock JCB-827 in Argon Prime für 104074 Cr." faction="{20203,201}" money="10407450"/>
<entry time="150377.804" title="Handel abgeschlossen" text="Hermes Angreifer QRT-394 verkaufte 10 Sensorenbündel an ARG Geschützturmkomponenten-Fabrik I DBX-715 in Argon Prime für 18861 Cr." faction="{20203,201}" money="1886130"/>
It's a newer savegame (save_005.xml.gz), i upoaded it to the same directory.

Also i made a little mistake translating wares.json in the first row. The char 'e' at the end was missing: "Name": "Fortschrittliche Verbundwerkstoffe".

magictrip
Posts: 247
Joined: Tue, 25. Dec 18, 01:04

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by magictrip » Tue, 8. Jan 19, 21:30

P.Hastig wrote:
Tue, 8. Jan 19, 20:44
Thank you,

the new version now can handle german savegames, but there is still a little issue.
It seems, that product-strings with special characters like ä, ö, ü can't be parsed.
That's strange, because destination-strings like "ARG Hüllenteilefabrik I" where parsed correctly.

See here..:
Image

..and the corresponding rows in the savegame file:

Code: Select all

<entry time="147056.314" title="Handel abgeschlossen" text="Hermes Angreifer QRT-394 verkaufte 50 Sensorenbündel an ARG Argon Raumdock JCB-827 in Argon Prime für 104074 Cr." faction="{20203,201}" money="10407450"/>
<entry time="150377.804" title="Handel abgeschlossen" text="Hermes Angreifer QRT-394 verkaufte 10 Sensorenbündel an ARG Geschützturmkomponenten-Fabrik I DBX-715 in Argon Prime für 18861 Cr." faction="{20203,201}" money="1886130"/>
It's a newer savegame (save_005.xml.gz), i upoaded it to the same directory.

Also i made a little mistake translating wares.json in the first row. The char 'e' at the end was missing: "Name": "Fortschrittliche Verbundwerkstoffe".
Well, I believe the difference there is that for ship names I use the identifier (like QRT-394) to do the queries and searches, while the wares I use the name, I may have to remove special characters or something like that for them to behave correctly. So, I will need to check that.

PS: Thanks for the fixed wares.json, I will upload to the new version as well.

User avatar
nonnex
Posts: 366
Joined: Wed, 28. Aug 13, 12:42
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by nonnex » Tue, 8. Jan 19, 22:02

convert haystack and needle strings to to UTF-8 before comparing. Or maybe easier-> https://docs.microsoft.com/en-us/dotnet ... work-4.7.2

magictrip
Posts: 247
Joined: Tue, 25. Dec 18, 01:04

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by magictrip » Tue, 8. Jan 19, 22:34

nonnex wrote:
Tue, 8. Jan 19, 22:02
convert haystack and needle strings to to UTF-8 before comparing. Or maybe easier-> https://docs.microsoft.com/en-us/dotnet ... work-4.7.2
Thanks for the tip, I will look into that.

Baldamundo
Posts: 99
Joined: Sat, 21. Jun 08, 18:46
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by Baldamundo » Thu, 10. Jan 19, 07:09

This looks fantastic, but I'm getting a lot of error spam saying "Error while trying to summarize the operations - Object reference not set to an instance of an object.", and a lot of ships aren't showing up in the list, and the graphs aren't displaying properly.

Could it be some sort of mod conflict? And if so, any ideas what type of mod's likely to be the culprit?

magictrip
Posts: 247
Joined: Tue, 25. Dec 18, 01:04

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by magictrip » Fri, 11. Jan 19, 02:58

Baldamundo wrote:
Thu, 10. Jan 19, 07:09
This looks fantastic, but I'm getting a lot of error spam saying "Error while trying to summarize the operations - Object reference not set to an instance of an object.", and a lot of ships aren't showing up in the list, and the graphs aren't displaying properly.

Could it be some sort of mod conflict? And if so, any ideas what type of mod's likely to be the culprit?
It looks like it's a mod conflict or a language problem. Can you provide me one of your save games that are causing that so I can take a look?

Baldamundo
Posts: 99
Joined: Sat, 21. Jun 08, 18:46
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by Baldamundo » Fri, 11. Jan 19, 08:36

magictrip wrote:
Fri, 11. Jan 19, 02:58
It looks like it's a mod conflict or a language problem. Can you provide me one of your save games that are causing that so I can take a look?
Am running the game in English, but have lots of mods installed, so likely the former. Off the top of my head, the only ones that should affect the trade log are Urgent Messages and Tradeship Buy Notifications.

Save uploaded here: https://www.dropbox.com/s/42wjxvn7nipq4 ... ml.gz?dl=0

magictrip
Posts: 247
Joined: Tue, 25. Dec 18, 01:04

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by magictrip » Fri, 11. Jan 19, 18:43

Baldamundo wrote:
Fri, 11. Jan 19, 08:36
Am running the game in English, but have lots of mods installed, so likely the former. Off the top of my head, the only ones that should affect the trade log are Urgent Messages and Tradeship Buy Notifications.

Save uploaded here: https://www.dropbox.com/s/42wjxvn7nipq4 ... ml.gz?dl=0
Yes, it was a mod. It was adding a (xxx Cr) after the name of the ware and because of that I was not finding the ware. I fixed that so please try the new version, it should work fine for you now.

One thing that I noticed is that you have lots of negative trader, I guess is because the mod is logging the traders where you are purchasing something? Like the wares to build stations and that kind of thing?

P.Hastig
Posts: 20
Joined: Thu, 14. Jun 07, 14:19
xr

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by P.Hastig » Sat, 12. Jan 19, 18:13

I finally managed to debug the application and found a workaround regarding the issue with special characters.

In MainWindow.xaml.cs, line 149 i replaced

Code: Select all

using (StreamReader r = new StreamReader(waresFileName))
with

Code: Select all

using (StreamReader r = new StreamReader(waresFileName, System.Text.Encoding.Default))
(System.Text.Encoding.UTF7 also works for me.)

The problem is, that this is not necessarily a solution for all environments or all possible translated versions of wares.json.

Baldamundo
Posts: 99
Joined: Sat, 21. Jun 08, 18:46
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by Baldamundo » Sun, 13. Jan 19, 11:53

magictrip wrote:
Fri, 11. Jan 19, 18:43
Yes, it was a mod. It was adding a (xxx Cr) after the name of the ware and because of that I was not finding the ware. I fixed that so please try the new version, it should work fine for you now.
Tried the new version - error messages are gone, and now ship information and ware analysis seem to work fine, but the ship analysis graphs are all blank.
One thing that I noticed is that you have lots of negative trader, I guess is because the mod is logging the traders where you are purchasing something? Like the wares to build stations and that kind of thing?
Yeah, that's the Tradeship Buy Notifications mod - it's very handy.

BTW, does this only count ships that aren't fleet members? (thus not working with the Mining/Trading Fleet mod)

And have you thought about adding some aggregated entries, so e.g. you could easily compare the combined profit of all freighters vs all gas miners vs all ore miners vs all factories (although I guess the ware analysis gives you some indication of this)

magictrip
Posts: 247
Joined: Tue, 25. Dec 18, 01:04

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by magictrip » Sun, 13. Jan 19, 22:26

Baldamundo wrote:
Sun, 13. Jan 19, 11:53
Tried the new version - error messages are gone, and now ship information and ware analysis seem to work fine, but the ship analysis graphs are all blank.
You just need to select one of the options on top and then double click on one of the ships to add them. As we can have multiple ships to have all them added would become a mess.
Baldamundo wrote:
Sun, 13. Jan 19, 11:53
Yeah, that's the Tradeship Buy Notifications mod - it's very handy.
That is a very good info, I wish they put that in the vanilla game.
Baldamundo wrote:
Sun, 13. Jan 19, 11:53
BTW, does this only count ships that aren't fleet members? (thus not working with the Mining/Trading Fleet mod)
Sadly the log for the fleet ships are incomplete and not set in the same place. They only log "Trade Complete" and I can discover how much money they made considering the station account, but there's no way in the log to discover which ware they sold and to whom. I am thinking in adding a "Station Overview" but it would basically bring the same information we already have in game, it would probably just be a little better UI and maybe bring the information of which ship is performing better, but now how/why they are doing.
Baldamundo wrote:
Sun, 13. Jan 19, 11:53
And have you thought about adding some aggregated entries, so e.g. you could easily compare the combined profit of all freighters vs all gas miners vs all ore miners vs all factories (although I guess the ware analysis gives you some indication of this)
In a consistent way the only thing I could do was to aggregate the wares because that is what is available in the logs. Now for the ships I will need to deal with their part of the XML (not the log itself, but still inside the save file) and get the ships info and probably check the type of inventory they have and maybe their size. That is a little bit more complicated than just parsing the log but it's doable.

My thought on doing that was more to track ships stuck with cargo and maybe show a grid with a fleet overview, but after I have that information is easy to consolidate and do what you mentioned.

magictrip
Posts: 247
Joined: Tue, 25. Dec 18, 01:04

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by magictrip » Sun, 13. Jan 19, 22:28

P.Hastig wrote:
Sat, 12. Jan 19, 18:13
I finally managed to debug the application and found a workaround regarding the issue with special characters.

In MainWindow.xaml.cs, line 149 i replaced

Code: Select all

using (StreamReader r = new StreamReader(waresFileName))
with

Code: Select all

using (StreamReader r = new StreamReader(waresFileName, System.Text.Encoding.Default))
(System.Text.Encoding.UTF7 also works for me.)

The problem is, that this is not necessarily a solution for all environments or all possible translated versions of wares.json.
Thanks man. I believe if we use System.Text.Encoding.UTF8 it should work for everyone, I will take a look and test with the save you provided before (I think I still have it).

realShakran
Posts: 8
Joined: Mon, 10. Dec 18, 20:08
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by realShakran » Mon, 14. Jan 19, 11:05

could it be that your analyzer doesnt like to ahve any mods installed? i have stuff like player owned shipyard, early seta, decreased buildtimes and "learningallthethings" installed, and when i try to load any save it just shows blank, as if loading completly nothing, and i cant say much as of why ebcause i understand not really anything about programming :)
i really would like to use it but seems like one of the mods is changing abit too much stuff?

seePyou
Posts: 30
Joined: Thu, 26. Jan 06, 01:02
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by seePyou » Tue, 22. Jan 19, 22:41

Hello!
I love this and it's potential, but I cannot get it to show the graph for the ships. I can load it, there is no error and it shows the ware analysis and the ship data, but when I go to the ship analysis tab, regardless of what I press and what I select of deselect, I cannot get a graph to show up.
Any ideas?

drJulien
Posts: 19
Joined: Mon, 4. Jan 21, 18:49
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by drJulien » Mon, 4. Jan 21, 19:13

I hope people in this forum can comment if there's another solution to this problem:

Manage your fleet of traders and miners to identify the best and worst performers. This is something basic for strategy games!
But I'm not angry at X4 - except that it took me so long to discover this incredible game, that I've been waiting for 24 years in my life. No joke.

@MagicTrip are you still active? I tried to ping you on github with an Issue. I'm a developer. I can contribute! But I have missing references.

So if there's a better tool already, please point me to that.
If this is the best thing to start from - this is still great! I'd like to add features and convenience. Maybe I can branch out without your permission or get permission to contribute?

If anyone is interested in helping with the project - just by helping me understand the log file, so I can code around it - that's also great
Last edited by X2-Illuminatus on Mon, 4. Jan 21, 22:51, edited 1 time in total.
Reason: removed full quote from post to ensure readability of the whole thread

Roeleveld
Posts: 403
Joined: Tue, 17. Feb 04, 23:34
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by Roeleveld » Wed, 6. Jan 21, 09:59

drJulien wrote:
Mon, 4. Jan 21, 19:13
I hope people in this forum can comment if there's another solution to this problem:

Manage your fleet of traders and miners to identify the best and worst performers. This is something basic for strategy games!
But I'm not angry at X4 - except that it took me so long to discover this incredible game, that I've been waiting for 24 years in my life. No joke.

@MagicTrip are you still active? I tried to ping you on github with an Issue. I'm a developer. I can contribute! But I have missing references.

So if there's a better tool already, please point me to that.
If this is the best thing to start from - this is still great! I'd like to add features and convenience. Maybe I can branch out without your permission or get permission to contribute?

If anyone is interested in helping with the project - just by helping me understand the log file, so I can code around it - that's also great
I would be willing to assist, but only if we can get this tool ported and supported on different platforms.

drJulien
Posts: 19
Joined: Mon, 4. Jan 21, 18:49
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by drJulien » Wed, 6. Jan 21, 12:51

Roeleveld wrote:
Wed, 6. Jan 21, 09:59
I would be willing to assist, but only if we can get this tool ported and supported on different platforms.
You're on Linux?
This application is currently .NET Framework 4.6.1
If I convert it to .Net Core, it can run as a standalone web application on any platform. One click and it opens a browser. you could even be using your mobile or to access the application while you're playing the game!
Visual Studio Code is also available on Linux so some limited help with the coding can be made by developers on Linux.

Well that's a lot of work. And already I can't compile now because the project uses something called Livecharts and 3 other components that are not on Nuget and probably licensed. They're not particularly good, the graphs and menus. But basically, I have to rewrite and rethink everything except the classes and log import code in C# (which are priceless btw).

I'm an experienced Developer with .Net and I think I can start a brand new project based on my personal work on a proper web application with security, charts and grids using Kendo. Kendo is not free but a trial version has some "you're using a trial version" once per session and u just refresh the page to use the application. The grids and charts are out of this world, though. One "problem" is my application uses MySQL database so it's another installation step for a poor user... But I'd like a database to store the log file data so it's easier to work with. Maybe it can be contained in the application but I never did this.

On the other hand if a good samaritan hosts the website for everyone - they register and upload their save file, update their save file and see their results. we could even share our empire's performance with other players! Each save file is 50MB though...

edit: another decision considering a web application is how to access the log file. ultimately the web application can be hosted on the internet. the user then has to upload his file. it can be stored on the server or locally but it has to be processed on the server. the application could also run on the X4 machine and have direct access to the save files on the disk.
So, what do you think? Why different platforms? The community here is multiplatform? I'm new!

Roeleveld
Posts: 403
Joined: Tue, 17. Feb 04, 23:34
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by Roeleveld » Wed, 6. Jan 21, 13:23

drJulien wrote:
Wed, 6. Jan 21, 12:51
Roeleveld wrote:
Wed, 6. Jan 21, 09:59
I would be willing to assist, but only if we can get this tool ported and supported on different platforms.
You're on Linux?
Yes, and there are several others.
A tool like this should be available on the same platforms that is supported by X4. (In my opinion)
This application is currently .NET Framework 4.6.1
If I convert it to .Net Core, it can run as a standalone web application on any platform. One click and it opens a browser. you could even be using your mobile or to access the application while you're playing the game!
Visual Studio Code is also available on Linux so some limited help with the coding can be made by developers on Linux.
My preference would be to split the code into 2 sections:
1) Log/savegame parser
2) Display

The parser should be written in standard C/C++ using non-platform specific libraries or using libraries that are available on all platforms.
The display-part should use a framework per platform to make it look nice.
As the display only shows tables and graphs, it shouldn't change too often, the parser will need updating along with the savegame format.
Well that's a lot of work. And already I can't compile now because the project uses something called Livecharts and 3 other components that are not on Nuget and probably licensed. They're not particularly good, the graphs and menus. But basically, I have to rewrite and rethink everything except the classes and log import code in C# (which are priceless btw).

I'm an experienced Developer with .Net and I think I can start a brand new project based on my personal work on a proper web application with security, charts and grids using Kendo. Kendo is not free but a trial version has some "you're using a trial version" once per session and u just refresh the page to use the application. The grids and charts are out of this world, though. One "problem" is my application uses MySQL database so it's another installation step for a poor user... But I'd like a database to store the log file data so it's easier to work with. Maybe it can be contained in the application but I never did this.

On the other hand if a good samaritan hosts the website for everyone - they register and upload their save file, update their save file and see their results. we could even share our empire's performance with other players! Each save file is 50MB though...

edit: another decision considering a web application is how to access the log file. ultimately the web application can be hosted on the internet. the user then has to upload his file. it can be stored on the server or locally but it has to be processed on the server. the application could also run on the X4 machine and have direct access to the save files on the disk.
So, what do you think? Why different platforms? The community here is multiplatform? I'm new!
My preference, and I think quite a few others as well, would be something that doesn't require an internet connection. And if it can be pointed at a newly created savegame without having to upload it first would be my preference.

drJulien
Posts: 19
Joined: Mon, 4. Jan 21, 18:49
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by drJulien » Wed, 6. Jan 21, 13:39

Roeleveld wrote:
Wed, 6. Jan 21, 13:23
drJulien wrote:
Wed, 6. Jan 21, 12:51
Roeleveld wrote:
Wed, 6. Jan 21, 09:59
I would be willing to assist, but only if we can get this tool ported and supported on different platforms.
You're on Linux?
Yes, and there are several others.
A tool like this should be available on the same platforms that is supported by X4. (In my opinion)
This application is currently .NET Framework 4.6.1
If I convert it to .Net Core, it can run as a standalone web application on any platform. One click and it opens a browser. you could even be using your mobile or to access the application while you're playing the game!
Visual Studio Code is also available on Linux so some limited help with the coding can be made by developers on Linux.
My preference would be to split the code into 2 sections:
1) Log/savegame parser
2) Display

The parser should be written in standard C/C++ using non-platform specific libraries or using libraries that are available on all platforms.
The display-part should use a framework per platform to make it look nice.
As the display only shows tables and graphs, it shouldn't change too often, the parser will need updating along with the savegame format.
Well that's a lot of work. And already I can't compile now because the project uses something called Livecharts and 3 other components that are not on Nuget and probably licensed. They're not particularly good, the graphs and menus. But basically, I have to rewrite and rethink everything except the classes and log import code in C# (which are priceless btw).

I'm an experienced Developer with .Net and I think I can start a brand new project based on my personal work on a proper web application with security, charts and grids using Kendo. Kendo is not free but a trial version has some "you're using a trial version" once per session and u just refresh the page to use the application. The grids and charts are out of this world, though. One "problem" is my application uses MySQL database so it's another installation step for a poor user... But I'd like a database to store the log file data so it's easier to work with. Maybe it can be contained in the application but I never did this.

On the other hand if a good samaritan hosts the website for everyone - they register and upload their save file, update their save file and see their results. we could even share our empire's performance with other players! Each save file is 50MB though...

edit: another decision considering a web application is how to access the log file. ultimately the web application can be hosted on the internet. the user then has to upload his file. it can be stored on the server or locally but it has to be processed on the server. the application could also run on the X4 machine and have direct access to the save files on the disk.
So, what do you think? Why different platforms? The community here is multiplatform? I'm new!
My preference, and I think quite a few others as well, would be something that doesn't require an internet connection. And if it can be pointed at a newly created savegame without having to upload it first would be my preference.
I don't know any way to downgrade to C++ and be multiplatform. There are tools like Xamarin (linux gtk) to create ports in various platforms but it's probably licensed?
.net core is not platform specific except it can only run web applications in a browser. using a browser is the modern way to be accessible to anyone. And to be clear it doesnt need to be on the internet.
It's always a good idea to separate the project like you say into the business logic that processes the log file and the UI. Multiple UI projects for each platform can connect to that one. In visual studio terms, that's a solution with multiple projects.

Roeleveld
Posts: 403
Joined: Tue, 17. Feb 04, 23:34
x4

Re: [TOOL] X4LogAnalyzer - Help analyse the trade log

Post by Roeleveld » Wed, 6. Jan 21, 14:22

drJulien wrote:
Wed, 6. Jan 21, 13:39


I don't know any way to downgrade to C++ and be multiplatform. There are tools like Xamarin (linux gtk) to create ports in various platforms but it's probably licensed?
.net core is not platform specific except it can only run web applications in a browser. using a browser is the modern way to be accessible to anyone. And to be clear it doesnt need to be on the internet.
It's always a good idea to separate the project like you say into the business logic that processes the log file and the UI. Multiple UI projects for each platform can connect to that one. In visual studio terms, that's a solution with multiple projects.
Isn't C# the same as C++ with a whole bunch of additional libraries?
I was thinking of going through the savegame myself to see what I can find.

GTK is a graphical toolkit, sticking to something opensource like "Boost" should help with keeping the parser portable.
For the graphical display part, QT (is available on different platforms), GTK (Not sure if it exists for MS Windows) or something similar should easily be applicable.
But the graphical part can be seperated per platform as long as all the data can be made available.

Post Reply

Return to “X4: Foundations - Scripts and Modding”