[TOOL] X4LogAnalyzer - Help analyse the trade log

[magictrip]

This looks fantastic, but I'm getting a lot of error spam saying "Error while trying to summarize the operations - Object reference not set to an instance of an object.", and a lot of ships aren't showing up in the list, and the graphs aren't displaying properly.

Could it be some sort of mod conflict? And if so, any ideas what type of mod's likely to be the culprit?

It looks like it's a mod conflict or a language problem. Can you provide me one of your save games that are causing that so I can take a look?

[Baldamundo]

It looks like it's a mod conflict or a language problem. Can you provide me one of your save games that are causing that so I can take a look?

Am running the game in English, but have lots of mods installed, so likely the former. Off the top of my head, the only ones that should affect the trade log are Urgent Messages and Tradeship Buy Notifications.

Save uploaded here: https://www.dropbox.com/s/42wjxvn7nipq4 ... ml.gz?dl=0

[magictrip]

Am running the game in English, but have lots of mods installed, so likely the former. Off the top of my head, the only ones that should affect the trade log are Urgent Messages and Tradeship Buy Notifications.

Save uploaded here: https://www.dropbox.com/s/42wjxvn7nipq4 ... ml.gz?dl=0

Yes, it was a mod. It was adding a (xxx Cr) after the name of the ware and because of that I was not finding the ware. I fixed that so please try the new version, it should work fine for you now.

One thing that I noticed is that you have lots of negative trader, I guess is because the mod is logging the traders where you are purchasing something? Like the wares to build stations and that kind of thing?

[P.Hastig]

I finally managed to debug the application and found a workaround regarding the issue with special characters.

In MainWindow.xaml.cs, line 149 i replaced

Code: Select all

using (StreamReader r = new StreamReader(waresFileName))

with

Code: Select all

using (StreamReader r = new StreamReader(waresFileName, System.Text.Encoding.Default))

(System.Text.Encoding.UTF7 also works for me.)

The problem is, that this is not necessarily a solution for all environments or all possible translated versions of wares.json.

[Baldamundo]

Yes, it was a mod. It was adding a (xxx Cr) after the name of the ware and because of that I was not finding the ware. I fixed that so please try the new version, it should work fine for you now.

Tried the new version - error messages are gone, and now ship information and ware analysis seem to work fine, but the ship analysis graphs are all blank.

One thing that I noticed is that you have lots of negative trader, I guess is because the mod is logging the traders where you are purchasing something? Like the wares to build stations and that kind of thing?

Yeah, that's the Tradeship Buy Notifications mod - it's very handy.

BTW, does this only count ships that aren't fleet members? (thus not working with the Mining/Trading Fleet mod)

And have you thought about adding some aggregated entries, so e.g. you could easily compare the combined profit of all freighters vs all gas miners vs all ore miners vs all factories (although I guess the ware analysis gives you some indication of this)

[magictrip]

Tried the new version - error messages are gone, and now ship information and ware analysis seem to work fine, but the ship analysis graphs are all blank.

You just need to select one of the options on top and then double click on one of the ships to add them. As we can have multiple ships to have all them added would become a mess.

Yeah, that's the Tradeship Buy Notifications mod - it's very handy.

That is a very good info, I wish they put that in the vanilla game.

BTW, does this only count ships that aren't fleet members? (thus not working with the Mining/Trading Fleet mod)

Sadly the log for the fleet ships are incomplete and not set in the same place. They only log "Trade Complete" and I can discover how much money they made considering the station account, but there's no way in the log to discover which ware they sold and to whom. I am thinking in adding a "Station Overview" but it would basically bring the same information we already have in game, it would probably just be a little better UI and maybe bring the information of which ship is performing better, but now how/why they are doing.

And have you thought about adding some aggregated entries, so e.g. you could easily compare the combined profit of all freighters vs all gas miners vs all ore miners vs all factories (although I guess the ware analysis gives you some indication of this)

In a consistent way the only thing I could do was to aggregate the wares because that is what is available in the logs. Now for the ships I will need to deal with their part of the XML (not the log itself, but still inside the save file) and get the ships info and probably check the type of inventory they have and maybe their size. That is a little bit more complicated than just parsing the log but it's doable.

My thought on doing that was more to track ships stuck with cargo and maybe show a grid with a fleet overview, but after I have that information is easy to consolidate and do what you mentioned.

[magictrip]

I finally managed to debug the application and found a workaround regarding the issue with special characters.

In MainWindow.xaml.cs, line 149 i replaced

Code: Select all

using (StreamReader r = new StreamReader(waresFileName))

with

Code: Select all

using (StreamReader r = new StreamReader(waresFileName, System.Text.Encoding.Default))

(System.Text.Encoding.UTF7 also works for me.)

The problem is, that this is not necessarily a solution for all environments or all possible translated versions of wares.json.

Thanks man. I believe if we use System.Text.Encoding.UTF8 it should work for everyone, I will take a look and test with the save you provided before (I think I still have it).

[realShakran]

could it be that your analyzer doesnt like to ahve any mods installed? i have stuff like player owned shipyard, early seta, decreased buildtimes and "learningallthethings" installed, and when i try to load any save it just shows blank, as if loading completly nothing, and i cant say much as of why ebcause i understand not really anything about programming
i really would like to use it but seems like one of the mods is changing abit too much stuff?

[seePyou]

Hello!
I love this and it's potential, but I cannot get it to show the graph for the ships. I can load it, there is no error and it shows the ware analysis and the ship data, but when I go to the ship analysis tab, regardless of what I press and what I select of deselect, I cannot get a graph to show up.
Any ideas?

[drJulien]

I hope people in this forum can comment if there's another solution to this problem:

Manage your fleet of traders and miners to identify the best and worst performers. This is something basic for strategy games!
But I'm not angry at X4 - except that it took me so long to discover this incredible game, that I've been waiting for 24 years in my life. No joke.

@MagicTrip are you still active? I tried to ping you on github with an Issue. I'm a developer. I can contribute! But I have missing references.

So if there's a better tool already, please point me to that.
If this is the best thing to start from - this is still great! I'd like to add features and convenience. Maybe I can branch out without your permission or get permission to contribute?

If anyone is interested in helping with the project - just by helping me understand the log file, so I can code around it - that's also great

[Roeleveld]

I hope people in this forum can comment if there's another solution to this problem:

Manage your fleet of traders and miners to identify the best and worst performers. This is something basic for strategy games!
But I'm not angry at X4 - except that it took me so long to discover this incredible game, that I've been waiting for 24 years in my life. No joke.

@MagicTrip are you still active? I tried to ping you on github with an Issue. I'm a developer. I can contribute! But I have missing references.

So if there's a better tool already, please point me to that.
If this is the best thing to start from - this is still great! I'd like to add features and convenience. Maybe I can branch out without your permission or get permission to contribute?

If anyone is interested in helping with the project - just by helping me understand the log file, so I can code around it - that's also great

I would be willing to assist, but only if we can get this tool ported and supported on different platforms.

[drJulien]

I would be willing to assist, but only if we can get this tool ported and supported on different platforms.

You're on Linux?
This application is currently .NET Framework 4.6.1
If I convert it to .Net Core, it can run as a standalone web application on any platform. One click and it opens a browser. you could even be using your mobile or to access the application while you're playing the game!
Visual Studio Code is also available on Linux so some limited help with the coding can be made by developers on Linux.

Well that's a lot of work. And already I can't compile now because the project uses something called Livecharts and 3 other components that are not on Nuget and probably licensed. They're not particularly good, the graphs and menus. But basically, I have to rewrite and rethink everything except the classes and log import code in C# (which are priceless btw).

I'm an experienced Developer with .Net and I think I can start a brand new project based on my personal work on a proper web application with security, charts and grids using Kendo. Kendo is not free but a trial version has some "you're using a trial version" once per session and u just refresh the page to use the application. The grids and charts are out of this world, though. One "problem" is my application uses MySQL database so it's another installation step for a poor user... But I'd like a database to store the log file data so it's easier to work with. Maybe it can be contained in the application but I never did this.

On the other hand if a good samaritan hosts the website for everyone - they register and upload their save file, update their save file and see their results. we could even share our empire's performance with other players! Each save file is 50MB though...

edit: another decision considering a web application is how to access the log file. ultimately the web application can be hosted on the internet. the user then has to upload his file. it can be stored on the server or locally but it has to be processed on the server. the application could also run on the X4 machine and have direct access to the save files on the disk.
So, what do you think? Why different platforms? The community here is multiplatform? I'm new!

[Roeleveld]

I would be willing to assist, but only if we can get this tool ported and supported on different platforms.

You're on Linux?

Yes, and there are several others.
A tool like this should be available on the same platforms that is supported by X4. (In my opinion)

This application is currently .NET Framework 4.6.1
If I convert it to .Net Core, it can run as a standalone web application on any platform. One click and it opens a browser. you could even be using your mobile or to access the application while you're playing the game!
Visual Studio Code is also available on Linux so some limited help with the coding can be made by developers on Linux.

My preference would be to split the code into 2 sections:
1) Log/savegame parser
2) Display

The parser should be written in standard C/C++ using non-platform specific libraries or using libraries that are available on all platforms.
The display-part should use a framework per platform to make it look nice.
As the display only shows tables and graphs, it shouldn't change too often, the parser will need updating along with the savegame format.

Well that's a lot of work. And already I can't compile now because the project uses something called Livecharts and 3 other components that are not on Nuget and probably licensed. They're not particularly good, the graphs and menus. But basically, I have to rewrite and rethink everything except the classes and log import code in C# (which are priceless btw).

I'm an experienced Developer with .Net and I think I can start a brand new project based on my personal work on a proper web application with security, charts and grids using Kendo. Kendo is not free but a trial version has some "you're using a trial version" once per session and u just refresh the page to use the application. The grids and charts are out of this world, though. One "problem" is my application uses MySQL database so it's another installation step for a poor user... But I'd like a database to store the log file data so it's easier to work with. Maybe it can be contained in the application but I never did this.

On the other hand if a good samaritan hosts the website for everyone - they register and upload their save file, update their save file and see their results. we could even share our empire's performance with other players! Each save file is 50MB though...

edit: another decision considering a web application is how to access the log file. ultimately the web application can be hosted on the internet. the user then has to upload his file. it can be stored on the server or locally but it has to be processed on the server. the application could also run on the X4 machine and have direct access to the save files on the disk.
So, what do you think? Why different platforms? The community here is multiplatform? I'm new!

My preference, and I think quite a few others as well, would be something that doesn't require an internet connection. And if it can be pointed at a newly created savegame without having to upload it first would be my preference.

[drJulien]

I would be willing to assist, but only if we can get this tool ported and supported on different platforms.

You're on Linux?

Yes, and there are several others.
A tool like this should be available on the same platforms that is supported by X4. (In my opinion)

This application is currently .NET Framework 4.6.1
If I convert it to .Net Core, it can run as a standalone web application on any platform. One click and it opens a browser. you could even be using your mobile or to access the application while you're playing the game!
Visual Studio Code is also available on Linux so some limited help with the coding can be made by developers on Linux.

My preference would be to split the code into 2 sections:
1) Log/savegame parser
2) Display

The parser should be written in standard C/C++ using non-platform specific libraries or using libraries that are available on all platforms.
The display-part should use a framework per platform to make it look nice.
As the display only shows tables and graphs, it shouldn't change too often, the parser will need updating along with the savegame format.

Well that's a lot of work. And already I can't compile now because the project uses something called Livecharts and 3 other components that are not on Nuget and probably licensed. They're not particularly good, the graphs and menus. But basically, I have to rewrite and rethink everything except the classes and log import code in C# (which are priceless btw).

I'm an experienced Developer with .Net and I think I can start a brand new project based on my personal work on a proper web application with security, charts and grids using Kendo. Kendo is not free but a trial version has some "you're using a trial version" once per session and u just refresh the page to use the application. The grids and charts are out of this world, though. One "problem" is my application uses MySQL database so it's another installation step for a poor user... But I'd like a database to store the log file data so it's easier to work with. Maybe it can be contained in the application but I never did this.

On the other hand if a good samaritan hosts the website for everyone - they register and upload their save file, update their save file and see their results. we could even share our empire's performance with other players! Each save file is 50MB though...

edit: another decision considering a web application is how to access the log file. ultimately the web application can be hosted on the internet. the user then has to upload his file. it can be stored on the server or locally but it has to be processed on the server. the application could also run on the X4 machine and have direct access to the save files on the disk.
So, what do you think? Why different platforms? The community here is multiplatform? I'm new!

My preference, and I think quite a few others as well, would be something that doesn't require an internet connection. And if it can be pointed at a newly created savegame without having to upload it first would be my preference.

I don't know any way to downgrade to C++ and be multiplatform. There are tools like Xamarin (linux gtk) to create ports in various platforms but it's probably licensed?
.net core is not platform specific except it can only run web applications in a browser. using a browser is the modern way to be accessible to anyone. And to be clear it doesnt need to be on the internet.
It's always a good idea to separate the project like you say into the business logic that processes the log file and the UI. Multiple UI projects for each platform can connect to that one. In visual studio terms, that's a solution with multiple projects.

[Roeleveld]

I don't know any way to downgrade to C++ and be multiplatform. There are tools like Xamarin (linux gtk) to create ports in various platforms but it's probably licensed?
.net core is not platform specific except it can only run web applications in a browser. using a browser is the modern way to be accessible to anyone. And to be clear it doesnt need to be on the internet.
It's always a good idea to separate the project like you say into the business logic that processes the log file and the UI. Multiple UI projects for each platform can connect to that one. In visual studio terms, that's a solution with multiple projects.

Isn't C# the same as C++ with a whole bunch of additional libraries?
I was thinking of going through the savegame myself to see what I can find.

GTK is a graphical toolkit, sticking to something opensource like "Boost" should help with keeping the parser portable.
For the graphical display part, QT (is available on different platforms), GTK (Not sure if it exists for MS Windows) or something similar should easily be applicable.
But the graphical part can be seperated per platform as long as all the data can be made available.

[drJulien]

C# also has useful shortcuts
.net is used in the code we have now with Linq for example
You'd have to go line by line and rebuild it.

I can't help you with all the stuff you listed to get to the point of drawing graphs and information grids with c++ open source libraries it would take me weeks just to get started. I'm too busy to learn new (and especially old ) stuff

the magic guy doesn't reply to my messages im sure it's legal/appropriate to "be inspired" by his project, the classes and the writeLogEntry function. If you progress on your side let us know here. My timeline is 2 days if I get started. I dont want to start yet if I step one someone's toes or double-work.
https://github.com/magictripgames/X4Mag ... ogAnalyzer

keep in touch

[Roeleveld]

C# also has useful shortcuts
.net is used in the code we have now with Linq for example
You'd have to go line by line and rebuild it.

I can't help you with all the stuff you listed to get to the point of drawing graphs and information grids with c++ open source libraries it would take me weeks just to get started. I'm too busy to learn new (and especially old ) stuff

the magic guy doesn't reply to my messages im sure it's legal/appropriate to "be inspired" by his project, the classes and the writeLogEntry function. If you progress on your side let us know here. My timeline is 2 days if I get started. I dont want to start yet if I step one someone's toes or double-work.
https://github.com/magictripgames/X4Mag ... ogAnalyzer

keep in touch

I don't see any license mentioned, but as long as you credit the source (eg. "magictrip" and a link to the github repo) it should be fine.
It's also not been touched for nearly 2 years, which points to abandonware in my book.

My timeline will be longer than a few days.

[drJulien]

Ok so I was stressed today so I started working on it.
I wanted to have fun so I used my familiar tools. Sorry it's in evil .net core
A lot of things to fix but once it's somewhat cleaner I'll put this on github if anyone is interested.






PS: Integrating the other project's code was not pleasant. Some bits are brilliant but it's a bit messy overall.

EDIT: Wait what the frick is happening to my profit the past 2 hours???

[Roeleveld]

Please share the code (and a compiled version) for others to use, if you don't mind.

.net-code is not (necessarily) evil, it's just not easy to run elsewhere.
Am going to see if I can get it running using proton or wine. If it does, I'll simply write up those instructions and we have a "portable" tool.

I'm not overly concerned about open/closed source, I simply want software to be available on all platforms with acceptable performance.
For games and work-related software that means: native or using a really good emulation layer.
For tools like this: Anything will do.

[drJulien]

Please share the code (and a compiled version) for others to use, if you don't mind.

.net-code is not (necessarily) evil, it's just not easy to run elsewhere.
Am going to see if I can get it running using proton or wine. If it does, I'll simply write up those instructions and we have a "portable" tool.

I'm not overly concerned about open/closed source, I simply want software to be available on all platforms with acceptable performance.
For games and work-related software that means: native or using a really good emulation layer.
For tools like this: Anything will do.

Thanks for your interest.
This is extremely rushed. And there's a lot of what some might consider useless fat around the bone. But I like to think big. if it's on a browser it's very portable and I've deployed this on linux mysql. Wait mysql? why do you need a database? the json file currently used is not ideal - I've seen how many ships you guys have!
Sorry I hijacked this thread, maybe it's the exact same project just a new version so I vote to keep discussing here.
https://github.com/docJulien/X4APS
see the readme for installation notes. very rushed. there is no "binary" gotta build it and research how dotnet standalone can be executed. im curious if visual studio code on linux can build this. otherwise use VS Community.