EnglishGermanFrenchRussianItalianSpanish
Log inRegister
 
[X3LU/MAYHEM][MOD] Assorted technical questions
Post new topic Reply to topic Goto page 1, 2  Next
View previous topic :: View next topic
Author Message
lostProfitssssArrgh





Joined: 22 Aug 2017



PostPosted: Wed, 13. Sep 17, 10:47    Post subject: [X3LU/MAYHEM][MOD] Assorted technical questions Reply with quote Print

Hello!

I'm exploring an idea for a mod and am in need of technical feedback from knowledgeable modders about file structures, engine limitations etc... More questions are likely to arise as the concept is explored, so expect to see me posting here for a bit; or not if it ends up being completely impossible.

Thanks in advance,
-lpa

-------

Questions, round 1:

1. How high a number can an arbitrary pageId (ie not accessed by existing code) reach in a Tfile? (I'm somewhat expecting to get 2^31 as a reply Very Happy ) The answer to that ties in to #3 and #4.

2. Same as above wr/t stringIds per pageId.

3. What is the maximum file size accessible by the X3AP binary? (likely tied to 32-bit but idk if it's 2gb or 4gb)

4. Does the script engine have access to a binary file format for I/O? If not, what would the be best way to store normal ship traffic XYZ coords and associated short strings? (xml parse speed versus tab separated tokenization etc...) Disk through-output considerations can be ignored for now.

5. Can the script engine start an independent external process?

6. Is there a 'tick' speed driving OOS ship movement, and how is it accessed?

7. How fast is the script engine compared to binary? An approximate answer will do.

8. Are the x-y minimum and maximum sector coordinates hard-coded or is just a map object limitation?

9. Does the map coord system handle a Z axis? (just making sure Razz )

10. Can a sector host more than 4 gates, aside from the Hub?

11. What data is routinely purged vs. constantly loaded objects?
---

...That's all I can think of for now. Thanks again!

Back to top
View user's profile Send private message
Cycrow
Moderator (Script&Mod)
Moderator (Script&Mod)



Joined: 15 Nov 2004
Posts: 20469 on topic
Location: London
Thank you for registering your game
PostPosted: Wed, 13. Sep 17, 11:04    Post subject: Reply with quote Print

Ill start by answering a few of these

1. a page id can be a maximum of 9999. The game will only use 4 digits for page ids, the extra digits are used to define the game

so, 0007, 300007, 350007 and 380007 are all the same page id of 7

2. Not sure of the limit on the t ids, most likely a 32bit int limit. There are some pretty big ids in the menus page (35)

8. This is limited to 32bit ints, but its done in game units not m, so in m its abit less, 500 units in a m

9. yes, same restrictions as above

10. the game can handle 6 gates, technically you can add more, but any more than 6 and you will run into issues.

11. Data, like the models, textures, etc are all cached until the game gets close to its memory limit, which is when it purges. It wont purge anything thats currently in the active sector


_________________
My Scripts | MY X3TC Scripts | X3 Plugin Manager | Custom Gui
Back to top
View user's profile Send private message Visit poster's website MSN Messenger
lostProfitssssArrgh





Joined: 22 Aug 2017



PostPosted: Wed, 13. Sep 17, 11:15    Post subject: Reply with quote Print

Thanks for 1, 2, 10 and 11!

Quote:

8. This is limited to 32bit ints, but its done in game units not m, so in m its abit less, 500 units in a m

9. yes, same restrictions as above


I was actually referring to the galaxy map space (ie kingdom end x0,y0 etc)... Sorry :/

Back to top
View user's profile Send private message
lostProfitssssArrgh





Joined: 22 Aug 2017



PostPosted: Wed, 13. Sep 17, 11:18    Post subject: Reply with quote Print

Damn, forgot about this:

12. Aside from script commands that need to reside in memory by nature, how are the rest accessed, purged etc...?

Back to top
View user's profile Send private message
Joubarbe





Joined: 31 Oct 2006
Posts: 2647 on topic
Location: France
Thank you for registering your game
PostPosted: Wed, 13. Sep 17, 11:35    Post subject: Reply with quote Print

lostProfitssssArrgh wrote:
Thanks for 1, 2, 10 and 11!

Quote:

8. This is limited to 32bit ints, but its done in game units not m, so in m its abit less, 500 units in a m

9. yes, same restrictions as above


I was actually referring to the galaxy map space (ie kingdom end x0,y0 etc)... Sorry :/


Maximum is 25*20 sectors. That's x:0,24 and y:0,19. Filling that space is really huge and will certainly lead to serious problems. Also, you need to have a 0,0 sector or you will experienced freezes over time. If I remember correctly, Jack once said to me that all your sectors need to be connected in LU if you want your AI ships to work properly.

And if you want to have a correct in-game map of your new galaxy, you need to use this.

Also note that if I never really achieved something with my universe generator tool, that's because the game acted really strange about my random universe: some commands were not working, sectors appeared with graphic glitches, AI ships were not doing what they're supposed to do... In short: expect bugs.


_________________
Mayhem - Satellites Monitoring - Internal Distribution Network - ADS (Joubarbian Edition) - Bounty Board - The Joubarbian Toolkit - EasyConnect
Back to top
View user's profile Send private message
lostProfitssssArrgh





Joined: 22 Aug 2017



PostPosted: Wed, 13. Sep 17, 11:40    Post subject: Reply with quote Print

Joubarbe wrote:

Maximum is 25*20...


Yes. But where is that limit defined? Galaxy map, core code, something else..?

Joubarbe wrote:

And if you want to have a correct in-game map of your new galaxy, you need to use this.


Links are dead Crying or Very sad

Back to top
View user's profile Send private message
Joubarbe





Joined: 31 Oct 2006
Posts: 2647 on topic
Location: France
Thank you for registering your game
PostPosted: Wed, 13. Sep 17, 11:42    Post subject: Reply with quote Print

To me it's not: https://forum.egosoft.com/viewtopic.php?t=394692&f=94&view=next.

Hardcoded.

EDIT: uh yeah sorry, the download link is dead indeed. Let me upload that somewhere.
EDIT2: Here you go.


_________________
Mayhem - Satellites Monitoring - Internal Distribution Network - ADS (Joubarbian Edition) - Bounty Board - The Joubarbian Toolkit - EasyConnect
Back to top
View user's profile Send private message
lostProfitssssArrgh





Joined: 22 Aug 2017



PostPosted: Wed, 13. Sep 17, 12:07    Post subject: Reply with quote Print

@Joubarbe
Thanks for link.

13. A tough question to formulate : how 'far' in terms of jumps are the NPC ships 'looking'..?
Another way to look at it : are NPC ships only 'aware' of their current sector despite having a destination that can be arbitrarily distant?

Back to top
View user's profile Send private message
lostProfitssssArrgh





Joined: 22 Aug 2017



modified
PostPosted: Wed, 13. Sep 17, 12:21    Post subject: Reply with quote Print

Joubarbe wrote:

...that's because the game acted really strange about my random universe...


14. Is it possibly due to static path-finding markers..?


And, answering my own question:
lpa wrote:

3. What is the maximum file size accessible by the X3AP binary? (likely tied to 32-bit but idk if it's 2gb or 4gb)


Technical limits to file size appear to be determined more by the file system than the bit-flavor of the processor (which can work by only extracting chunks of a large file). After consideration, the issue in the context of this projects' needs is mooted since using a larger number of smaller files has its benefits.



Last edited by lostProfitssssArrgh on Wed, 13. Sep 17, 13:07; edited 2 times in total
Back to top
View user's profile Send private message
Joubarbe





Joined: 31 Oct 2006
Posts: 2647 on topic
Location: France
Thank you for registering your game
PostPosted: Wed, 13. Sep 17, 13:00    Post subject: Reply with quote Print

You should probably wait for an answer from Jack; he's the Epic Warlord of X3 (there should be a special badge for that).


_________________
Mayhem - Satellites Monitoring - Internal Distribution Network - ADS (Joubarbian Edition) - Bounty Board - The Joubarbian Toolkit - EasyConnect
Back to top
View user's profile Send private message
Jack08





Joined: 25 Dec 2005
Posts: 2956 on topic
Location: Australia
Thank you for registering your game
PostPosted: Wed, 13. Sep 17, 14:41    Post subject: Reply with quote Print

2. Its a signed 32bit integer, which means you can actually use negative t'ids too, -2^31 -> 2^31.

3. While i would be inclined to say thats OS dependent, you still face memory issues with large files, especially if your loading the data into the MSCI as you have to load it all at once from a single file, so if your loading 5gb of data, your likely to instantly crash. Not that i can come up with any scenario where you would need to do such a thing?

4. Using tab separation and splitting the input based on \n, and \t would work ok, in LU, you can read this in raw from any file using "$Data = read file: <file>". In vanilla you would have to embed this data in a textdb file instead.

5. No

6. There is a tick, i dont know exactly what it is. It is "hardcoded", and generally, this concept should be entirely ignored when it comes to scripting as its a lower level thing.

7. The MSCI, is a virtual machine, Wrapped in another virtual machine called KC, as such, Each MSCI instruction, would equate to more then 20-30 (perhaps closer to 40) machine code instructions, so a rough estimation would be 50x slower then C on an instruction amount, however with taking the fact that VM's suffering branch-prediction-totally-useless syndrome this value could be higher (depending on implementation).

While this sounds pretty bad, keep in mind that you can always prolong extensive tasks over a period of time, by doing a single (or a couple) of iterations per frame, using waits - it makes tasks take time, rather then bog down the cpu. Improved Races 2.0 does this in the economy booster, it has a very large analysts phase that can take longer then 15 minutes in some cases, but if run without waits, would freeze the game for a few moments.

8. Depends what your asking about, i am going to assume you mean: Argon Prime has a universal grid coordinate of X,Y - then yes the hardcoded limit is 24,20 (iirc) and no, a Z coordinate cannot be added to this to create additional sector dimensions, however...

9. While you cant give sectors a Z coordinate when ti comes to addressing the actual sector, a Z coordinate can be added to the universe "bod" file, which is what visually represents the universe in the universe map, this is an entirely visual representation in the universe map, nothing more. The "universe bod" is what makes the universe map look non-square in the vanilla game, in LU we use a "square bod" which eliminates this; This was our personal preference.

---

13. In vanilla, they dont care at all, there scripting remains unaware of adjacent sectors. In LU they do look ahed to see if the "path is clear", i dont remember the actual range of looking head.

14. The game internally uses an A* pathing method, but these results get cached to prevent extensive lookup, this is why the AI can bug out if you create sectors/gates on the fly. I believe this cache is cleared every time you close the game, but i never did any testing to prove it.


_________________

"One sure mark of a fool is to dismiss anything that falls outside his experience as being impossible."
―Farengar Secret-Fire
Back to top
View user's profile Send private message
lostProfitssssArrgh





Joined: 22 Aug 2017



PostPosted: Wed, 13. Sep 17, 19:13    Post subject: Reply with quote Print

@Jack08
Great stuff! Thanks!

Back to top
View user's profile Send private message
lostProfitssssArrgh





Joined: 22 Aug 2017



PostPosted: Wed, 13. Sep 17, 19:19    Post subject: Reply with quote Print

(My apologies for the bump...)

Jack08 wrote:

4. Using tab separation and splitting the input based on \n, and \t would work ok, in LU, you can read this in raw from any file using "$Data = read file: <file>". In vanilla you would have to embed this data in a textdb file instead.


What's your take on the performance end of those methods..?

Back to top
View user's profile Send private message
Jack08





Joined: 25 Dec 2005
Posts: 2956 on topic
Location: Australia
Thank you for registering your game
PostPosted: Wed, 13. Sep 17, 23:21    Post subject: Reply with quote Print

that depends on the amount of data your trying to push, and if your trying to do it in realtime or not... If its not realtime, then it can be done in a setup script and the performance hit can be entirely ignored.

If your going to try and receive a real-time data stream, then, your going to face significant problems depending on the stream size.


_________________

"One sure mark of a fool is to dismiss anything that falls outside his experience as being impossible."
―Farengar Secret-Fire
Back to top
View user's profile Send private message
lostProfitssssArrgh





Joined: 22 Aug 2017



PostPosted: Fri, 15. Sep 17, 16:23    Post subject: Reply with quote Print

lostProfitssssArrgh wrote:
(My apologies for the bump...)

Jack08 wrote:

4. Using tab separation and splitting the input based on \n, and \t would work ok, in LU, you can read this in raw from any file using "$Data = read file: <file>". In vanilla you would have to embed this data in a textdb file instead.


What's your take on the performance end of those methods..?


After looking into this a bit, my best guess is that xml is faster when it comes to reading 'sorted' data. The script engine being MUCH slower than an api plugin, I really doubt that reading plain text and sorting it in-game can be faster; even if the actual loadCommand-to-dataAvailable time is shorter and the files smaller (untested).

@Jack08
You guessed correctly that the files would be accessed more than once.


-----------


Another question:

15. I remember someone, somewhere in the forum, posting that 'Kingdom End' *HAD* to be at (0, 0). Is this correct? Do we know why? (@Joubarbe, could this be source of your problems with EUG?).

Thanks again,
-lpa

Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic Reply to topic Goto page 1, 2  Next
 
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You cannot download files in this forum
Control Panel
Login Data
The time now is Fri, 24. Nov 17, 15:04

All times are GMT + 2 Hours


Board Security

Copyright © EGOSOFT 1989-2017
Powered by phpBB © 2001, 2005 phpBB Group
Template created by Avatar & BurnIt!
Debug: page generation = 0.77215 seconds, sql queries = 29