Exscriptor - new external script editor

bunkerprivate · Post by **bunkerprivate** » Wed, 3. Sep 08, 02:03

Whimsy wrote:The file must exist somewhere! It's probably in one of the numeric CAT/DAT files, or maybe a mod? I use a library to load all the data files so sometimes it's hard to figure out exactly what's going on...

It might be a product of me installing every script under the sun when I first installed X3... there are conflicts all over the place, though it seems to work all right. XTM especially generates a conflict for every ware they have added. Here is an example of a conflict:

- Conflict in file 445066: command with ID 402 (COMMAND_SDS_STATION_DELIVER_ONE) already exists

But there is definitely no 445066.xml or 445066.pck in t/ or anywhere in the X3 install. I have no idea either what that script is since it's so long ago a put them all in -- I've certainly never used it! ^^

Whimsy wrote:I'm not entirely sure what you mean in your example

If you twice (or more) assign to the same variable, but never use the first assignment; so a proper example would be

Code: Select all

$r = get pid
$r = get task ID
=[THIS]->call script 'something': $r

The first value is ignored, which probably indicates a programming error. The scoping problem is that you could have:

Code: Select all

$r = get pid
gosub something
$r = get task ID
=[THIS]->call script 'something': $r

something:
printf $r
endsub

Which would be valid.

I found one more: when I open !move.jump, "$ship->interrupt with script " is rendered as "$ship interrupt with script ". Also it will only accept this as input -- I'm pretty sure it's not supposed to be like that, anyway!

I look forward to finally finishing off this script with .029!

draffutt · Post by **draffutt** » Wed, 3. Sep 08, 04:02

Whimsy wrote:
draffutt wrote:1st: if for example you make 2 changes to the current open script then you hit undo 3 times; the third time you hit undo the script is blank (deleted?)
I think I've fixed this now, but as I've said in the past, the undo/redo is very klunky. For instance, I noticed the cursor is doing strange things again when you undo; I'm not sure whether that's new or what. At some point I'll redo it all (pun not intended) if I get chance.

Its been doing that for a while and seems to be harmless.

1.) in the compare window can you one more important item there? show the script command for both scripts being compared at the bottom of the compare window?

2.) can you also link the bottom left scroll bar to the right window? I.E. if i use the scroll bar at the bottom left window it moves both the right and left windows.

3.) can you also change the right window scroll bar not to snap back until the user "lets go" of the bar. currently when movement stops it snaps back to the currently line number in the left window even thou i am still clicked on the right window bar. And yes you mentioned about this snap back feature. after you explained it i agree i like it for the most part. but it was annoying when i was trying to compare an old script with the one i had modded 100 lines compared to 130 of the original and i wasn't able to scroll down the rest of the lines to find the one i removed that i shouldn't have. without it snaping back into place as soon as i stopped movement of the scroll bar. so if you could prevent it from snapping back unless the user "lets go" of the scroll bar would be much appreaciated. if not i'll live with it and others will more likely not notice

4.) small isssue with compare: with just a blank script open. click on compare, select a script and compare should open with a blank in the first window and the script you are compairing in the second. now close compare and script screen should turn grey even thou you have a "tab" that is suppose to be active. the only way i could get out of this was to close exscriptor.

5.) when closing Exscriptor using the top right X. can you change the currently closing menu to what you are using now with the close all function?

6.) when compling: if i use

Code: Select all

$thisamount $amounts[$count]

forgetting to put an = between the two $var compling doesn't give an error

7.) using the error above as an example can you include an underline function? I.E. as you are typing in your code it underlines it until it thinks the code is in the correct format and then removes the underline? if you are able/do include this at some point can you make it as an option as i am sure some people will not want this feature.

8.) Version .029 from Blackpanthergroup is corrupt. Version on mediafire is fine (as far as i can tell.)

Question:
when i tried to open this in exscriptor: http://www.4shared.com/file/61531939/c3 ... MMain.html
i received a error message stating: object reference not set to an instance of an object. is this error message trying to tell me there is an NullReferenceException?

wishful thinking on my part.
in the extremely unlikely event (will not know unless asked) what is the chance of getting exscriptor to at least recognize and load the x2 data files? as i said wishful thinking on my part.

Whimsy · Post by **Whimsy** » Thu, 4. Sep 08, 00:46

bunkerprivate wrote:But there is definitely no 445066.xml or 445066.pck in t/ or anywhere in the X3 install. I have no idea either what that script is since it's so long ago a put them all in -- I've certainly never used it! ^^

I tracked that one down - it's in 04.cat/dat, which means it's pretty old. And in fact I dimly remember the SDS (Station Delivery Software, I think) being an X2 script anyway... who knows what it's doing in there!

bunkerprivate wrote:If you twice (or more) assign to the same variable, but never use the first assignment; so a proper example would be
Code: Select all
$r = get pid
$r = get task ID
=[THIS]->call script 'something': $r
The first value is ignored, which probably indicates a programming error.

Hmm. I agree that assigning a variable twice in a row is stupid, possibly even an error, but I think checking for it is a bit over the top - even Visual Studio doesn't pick up on this. Plus there's so many ways it can still be valid, without even involving subroutines, e.g. what if the second assignment was inside an if:

Code: Select all

$newname = 'Ship 1 '
if [THIS]->is of class {M1}
   $newname = 'Carrier 1'
end
[THIS]->set name to $newname

And when you consider the potential havoc that the spaghetti-like gotos and gosubs can wreak, I think any check would be almost impossible to perform accurately anyway. So I think I'll leave this one out...

bunkerprivate wrote:I found one more: when I open !move.jump, "$ship->interrupt with script " is rendered as "$ship interrupt with script ". Also it will only accept this as input -- I'm pretty sure it's not supposed to be like that, anyway!

Ah, it was an omission in the params file. I've fixed it now, together with a couple of other errors I spotted at the same time (see the = bug below).

draffutt wrote:1.) in the compare window can you one more important item there? show the script command for both scripts being compared at the bottom of the compare window?

Assuming you mean script names rather than script commands (which are usually 0), yes, no problem.

draffutt wrote:2.) can you also link the bottom left scroll bar to the right window? I.E. if i use the scroll bar at the bottom left window it moves both the right and left windows.

Okay, I've done this too - the horizontal scrollbar is synchronised too now.

draffutt wrote:3.) can you also change the right window scroll bar not to snap back until the user "lets go" of the bar.

I can see why it might be useful, but I can't do this - the way it synchronises is very clunky, and it's not actually based on a mouse click, so getting it to stay there until you "unclick" is not going to be simple. If you want to be able to view the other script more easily you'll just have to compare them the other way around, I'm afraid.

draffutt wrote:4.) small isssue with compare: with just a blank script open.

I've altered it now so you can only compare if you already have a script open to compare with.

draffutt wrote:5.) when closing Exscriptor using the top right X. can you change the currently closing menu to what you are using now with the close all function?

Done this too now.

draffutt wrote:6.) when compling: if i use
Code: Select all
$thisamount $amounts[$count]
forgetting to put an = between the two $var compling doesn't give an error

This was actually quite tricky! I fixed it easily enough - it will complain now if you try to do this - but when I was testing it on other scripts to make sure it worked, I found that some scripts weren't loading in the = properly. Turns out there were more omissions in the parameters file. Anyway, it's all sorted now.

draffutt wrote:7.) using the error above as an example can you include an underline function?

Kind of like a spellcheck in a word processor, you mean? Alas, though I can see why it would be nice, it wouldn't be a simple task, assuming it's even possible. I may include something like this in my hypothetical rewrite (the Nextscriptor?) but not in the current tool; it would mean too many major changes to how it all works.

draffutt wrote:8.) Version .029 from Blackpanthergroup is corrupt. Version on mediafire is fine (as far as i can tell.)

Odd - it must have got cut off when I uploaded it or something. At least the Mediafire one was okay; the installer version was okay too, apparently, so it was just the zip. Anyway, V1.030 is out now, so it's not a problem - and I've tested them this time.

draffutt wrote:Question:
when i tried to open this in exscriptor: http://www.4shared.com/file/61531939/c3 ... MMain.html
i received a error message stating: object reference not set to an instance of an object. is this error message trying to tell me there is an NullReferenceException?

It was another of those cursed commented commands again. Turns out it wasn't saving commented call commands properly... Anyway, I've fixed it now. It'll warn you when you load up a file so afflicted - any call parameters will be missing, but the rest should be there - and if you just save it again (adding in the missing parameters, if any) then it'll be fine.

draffutt wrote:wishful thinking on my part.
in the extremely unlikely event (will not know unless asked) what is the chance of getting exscriptor to at least recognize and load the x2 data files? as i said wishful thinking on my part.

Is there actually much call for that, do you think? To answer your question, the chances are very good if there's a demand for it. The data is in the same format basically, and with a couple of tweaks (like ignoring X3-specific files), the Exscriptor will happily load in the X2 data files. The problem is that there are new commands and wares etc in X3, so I'd have to have some sort of "X2 Mode" to make sure you only use X2 commands and wares etc. Still, it's very possible. The only reason I didn't do this originally was because I wasn't doing any X2 scripting myself and I didn't think the extra effort was worth it if nobody was going to use the X2 mode.

Anyway, I'll tinker around a bit and you never know - maybe V1.031 or V1.032 will have an X2 compatibility mode.

And as usual V1.030 is here.

draffutt · Post by **draffutt** » Thu, 4. Sep 08, 02:02

Whimsy wrote:
draffutt wrote:1.) in the compare window can you one more important item there? show the script command for both scripts being compared at the bottom of the compare window?
Assuming you mean script names rather than script commands (which are usually 0), yes, no problem.

No i was referring to the script commands. sometimes when i have to change some scripts because of command conflicts i sometimes forget to change the script command because as you said most of the time it is 0. so it would for me at least make it a little bit easier to idenitfy those when doing a compare when they are assigned something other then 0 so i can see if it has been changed or needs to be changed. also if you could the left block is changable for script command. having the script names on the bottom is a good idea when working with a lot of scripts.

Whimsy wrote:Is there actually much call for that, do you think? To answer your question, the chances are very good if there's a demand for it. The data is in the same format basically, and with a couple of tweaks (like ignoring X3-specific files), the Exscriptor will happily load in the X2 data files. The problem is that there are new commands and wares etc in X3, so I'd have to have some sort of "X2 Mode" to make sure you only use X2 commands and wares etc. Still, it's very possible. The only reason I didn't do this originally was because I wasn't doing any X2 scripting myself and I didn't think the extra effort was worth it if nobody was going to use the X2 mode.

Anyway, I'll tinker around a bit and you never know - maybe V1.031 or V1.032 will have an X2 compatibility mode.

When i had made this wishful request i had been thinking along the lines of exscriptor at least loading the x2 data files with a "use at your own risk" type deals. but to answer your question handful maybe 2 at the very most. but as for this feature i wouldn't want you to be to much effort into since there isn't going to be much of a call for it. but i would use whatever you implemented thou.

even if it just to load up the data files anything else would be a bonus above that. so speaking of compatibilty are you planning on supporting exscriptor with X3:TC?

ok have taken a look at .030 only thing i have noticed that doesn't appear to be working is with the find / match whole word selected. selecting/ unselecting this option appears not do make a difference with the search function.

for the next update can you include under options / appearance the ability to change the autocomplete and conflict viewer as well?

s9ilent · Post by **s9ilent** » Sat, 6. Sep 08, 09:23

Just thought I'd point out that, the replace all function doesn't seem to work properly. I find that I have to keep pressing it repeatedly/ manually edit some of the text.

I was trying to get it to replace a word that was part of a string (i.e. it was inside 'word.word.word') And it was suppose to replace one of the words

MJALowe · Post by **MJALowe** » Sat, 6. Sep 08, 10:09

I've had a few problems with labels during compile/save, often the first label in a script (in my case its usually withing 5 lines of the top of the script, and always BEFORE the first goto) and never the second label - will report as not being present in the script generating a compile/save fail - my solution for the moment was to put an unused label before it.

draffutt · Post by **draffutt** » Sat, 6. Sep 08, 21:46

Couple of small loading issues/quirks:

1. When attempting to load a .pck file if there is an issue with it I.E. “object reference not set to an instance of an object” like this one:
http://www.4shared.com/file/61979584/10 ... dmain.html
Exscriptor doesn’t list which script caused the issue; only the error. Makes it hard to track down if it is part of a group of scripts being loaded at the same time. Problem with listing which file(s) when opening .pck’s?

2. FYI I have been able to make Exscriptor crash twice now while trying to open a bunch of .pck’s at a same time. Exscriptor gives the error system out of memory exception despite the fact I have a 64bit OS with 8Gigs of memory. (I realize a 32bit code only uses a max 3.9Gigs due to memory limits of the code. Even then i still have plenty of memory left.)

3. When attempting to load this .xml by itself give the error “an error occurred while parsing Entityname. Line 260, position 48753”:
http://www.4shared.com/file/61980421/64 ... nlist.html

But if loaded as part of a group of scripts causes same error above to be displayed twice implying there are two errors even thou it is the same error.

4. When attempting to load this .xml gives and error “(“<”,hexadecimal value 0x3c, is an invalid attribute character. Line 437, position 95698.”)”)” (and yes there are two of those brackets and quotes at the end even thou the error message only utilized one of each at the beginning) :
http://www.4shared.com/file/61980214/3b ... pTask.html

5. When attempting to load this one:
http://www.4shared.com/file/61979863/18 ... t_use.html
Exscriptor doesn’t explain why it is not able to load the file except to say unable to open script.

6. When attempting to load this one:
http://www.4shared.com/file/61980622/fe ... ement.html

Exscriptor says there is a parsing Entityname. Line 59, position 7287 even thou when opening in MSCI there are only 42 lines of code.

Other stuff:

7. When resizing compare window the script name block doesn’t resize with the window

8. Can you have an option to have a popup for the “close all” that shows all the conflicts that were identified? I.E. when using the “close all” scripts I had ashleys scripts open and just one of them generated at least 100 error/warning messages. That little window in the bottom left corner is hard to navigate when there are a lot of them. Or even better have it open a popup when it finds say more then 20 error/messages.

Whimsy · Post by **Whimsy** » Sun, 7. Sep 08, 04:08

I'm posting empty handed this time. I'm working on a new version but it's taking longer than expected, so I thought I'd give you an update and answer some of your comments in the meantime.

draffutt wrote:No i was referring to the script commands.

Oops! Sorry.

I'll include this in the next version, complete with correct resizing.

draffutt wrote:When i had made this wishful request i had been thinking along the lines of exscriptor at least loading the x2 data files with a "use at your own risk" type deals.

As I said, loading the data files is no problem. I've got a workable X2 mode going quite nicely in the new version but there's still a few issues to iron out before it's ready. This is one reason for the delay.

draffutt wrote:so speaking of compatibilty are you planning on supporting exscriptor with X3:TC?

The short answer is yes but I'll give the long answer at the end.

draffutt wrote:ok have taken a look at .030 only thing i have noticed that doesn't appear to be working is with the find / match whole word selected. selecting/ unselecting this option appears not do make a difference with the search function.

You'll have to give me a bit more to go on! It definitely does make a difference - what was your search and what wasn't it picking up? Note that it only affects words - something like "$searchforme.butnotme" would still pick it up even in match whole word mode, since the . counts as a word break.

draffutt wrote:for the next update can you include under options / appearance the ability to change the autocomplete and conflict viewer as well?

Yes, I've done this now.

s9ilent wrote:Just thought I'd point out that, the replace all function doesn't seem to work properly. I find that I have to keep pressing it repeatedly/ manually edit some of the text.

I was trying to get it to replace a word that was part of a string (i.e. it was inside 'word.word.word') And it was suppose to replace one of the words

The Find/Replace is a constant source of problems. Could you be more specific though? Usually replace all is quite reliable, and I tested it and it does seem to pick out one word in the middle of several others. These sorts of problems are often very obscure so the more information the better.

MJALowe wrote:I've had a few problems with labels during compile/save, often the first label in a script

I know that if the label is on the very first line it doesn't work (I'd already spotted and fixed this one in the new version) but after investigating a bit more it seems that if there's no actual commands before the label, i.e. only blank lines or comments, then the same issue applies. If that's not the case with any of your scripts, let me know since it means there's likely another problem.

draffutt wrote:1. When attempting to load a .pck file if there is an issue with it I.E. “object reference not set to an instance of an object” like this one

I've altered it so it says which file it's loading now, to make it clearer. In this case the file has a really weird commented command in it - an al plugin timer one - that conforms to no known rules or standard, hence the error. I've created a workaround to allow you to load the file (with the command incomplete) but until I figure out the logic behind the format (if there is any!) I can't do much more.

draffutt wrote:2. FYI I have been able to make Exscriptor crash twice now while trying to open a bunch of .pck’s at a same time.

I doubt it's really running out of memory; like you say, you have plenty. It probably just can't take the stress...

draffutt wrote:3. When attempting to load this .xml by itself give the error “an error occurred while parsing Entityname. Line 260, position 48753”:

I can't seem to get it to print out the same error twice, but I've changed a lot so I might have inadvertantly got rid of it. In any case, the file contains invalid characters (ampersands, I think it was).

draffutt wrote:4. When attempting to load this .xml gives and error “(“<”,hexadecimal value 0x3c, is an invalid attribute character. Line 437, position 95698.”)”)”

The brackets match up; the missing one is in the earlier part of the message (it's nested). This file also contains invalid characters, this time < characters. If you can't load it properly in a browser either then it's almost always caused by invalid characters somewhere.

draffutt wrote:5. When attempting to load this one: Exscriptor doesn’t explain why it is not able to load the file except to say unable to open script.

It can't find it!

It gets confused by the weird filename. If you get rid of the " - don't use" bit on the end it's fine. As long as there's no spaces and only one extension, it's okay.

draffutt wrote:6. Exscriptor says there is a parsing Entityname. Line 59, position 7287 even thou when opening in MSCI there are only 42 lines of code.

Invalid characters again; the line 59 refers to the XML file not the script. In this case it's an & on its own in a comment. The codearray is all on one line in the XML files (line 59 in this case), hence the very large character position of 7287. Fortunately they're usually quite easy to find and fix, even if it does take a lot of scrolling!

draffutt wrote:7. When resizing compare window the script name block doesn’t resize with the window

I've done this together with the script command boxes.

draffutt wrote:8. Can you have an option to have a popup for the “close all” that shows all the conflicts that were identified?

I'm not entirely sure what close all has to do with it but I assume you mean when it saves them all and thus produces errors. In which case the thing to do is check (and compile) each file separately; even if it did show all the errors elsewhere, if the files were closed, it wouldn't be much good!

Anyway, this brings me on to my last point. Since X3:TC is getting closer and since it seems like most of the major, crippling bugs are fixed (hopefully), the next one or two versions of Exscriptor will likely be the last ones in its current form, unless there are any major problems in the next version that need fixing. By that I mean there won't be any more features and/or changes, only (preferably major) bugfixes.

The reason for this is that I want to overhaul the code properly rather than just patching it up here and there: I want a more solid foundation to support TC and that means I need to devote more energy to creating V2.0 and less to tweaking V1.0. By all means continue to report bugs and make suggestions, however; I'll still try to fix the former and hopefully incorporate the latter into the new version. But unfortunately my time is limited at present and I need to spend it where it can have the most effect.

The plan is to have a (largely) stable V1.1 that supports X2 and X3 - and that means the next version or so, once the bugs are worked out - and then ultimately a V2.0 that supports X2, X3 and TC later on, which will be a substantially rewritten program.

Anyway, hope that all makes sense - if anyone has any suggestions or comments about the plan, please let me know.

V1.031 is now available here. To use X2 you need to select the "X2 Mode" under the Tools menu after providing paths to X2 in the Options. I've also made quite a few internal alterations which will hopefully fix some bugs, but likely also introduced some too...

bunkerprivate · Post by **bunkerprivate** » Mon, 8. Sep 08, 21:28

Anyway, hope that all makes sense - if anyone has any suggestions or comments about the plan, please let me know.

Well since you ask I can't resist sticking my oar in! ^^

There are two major things that I think would be really handy but are fairly big behavioral changes. First, as with .cpp to .o, I think it would be good to have a .x3s (or .x2s) to .xml. This way you could have more extended macro pre-processing, but still keep all the written code as is (heh, you could pass it through the C preprocessor or m4 easily enough). This would also be handy for having shared information between modules, such as constants for return values like you would access with #include and enum {}, and #ifdefs for debug logging code. It also means you -- or I suppose just me likely -- can use grep, sed and friends to manipulate the source code as plain text.

Second thing is it would nice -- again because I'm a Unix user -- to have a separate command-line compiler which is executed by the exscriptor GUI. Of course this is a bit of a problem since there is the whole X3 file system stuff going on, but it would be very nice if I could write a Makefile to build my entire project. The difficulty again is that the X3 language is not particularly suited to being written externally since the language is entirely tokens, rather than general function names and so on... of course you could just implement a new programming language using the codearray as an assembler language! (That would actually be pretty fun imo, but you might have to grow a beard first ^^)

Of course it's your tool so your choice to design whatever fits you best!

Lastly, a few minor issues that came up recently:

- moving an argument's position does not set the dirty flag.
- script arguments to commands are not checked for presence, eg "send incoming question ... ", "interrupt with ... ", the signal one, some others?
- would be handy if it would say when textids or page ids do not exist. I don't know if it parses this stuff already.
- "$eventQueue = [THIS]->get local variable:" (missing name=whatever) does not result in an error.
- the parameter type "object command" is not available (see !ship.signal.killed).
- the preferences box has no cancel button, and when you click the X, it saves the preferences anyway (I accidentally pressed Ctrl+P then then typed something ^^)
- "scripts directory (where you keep your" is truncated in the preferences box.
- Ctrl+P, tab out from the preferences window, Ctrl+P again. There are two settings windows now.

Whimsy · Post by **Whimsy** » Tue, 9. Sep 08, 22:13

bunkerprivate wrote:First, as with .cpp to .o, I think it would be good to have a .x3s (or .x2s) to .xml.

There is definitely something conceptually ugly about the XML file effectively being both the input and the output in Exscriptor; one of the many unfortunate consequences is that you can only save a valid XML file (which in turn is one of the reasons I added in the txt format). Separating the two would definitely be nice, but the problem is that you still need to be able to edit (or at least translate) the XML files themselves; the .cpp/.o comparison falls down a little here in that there's a separate tool - the game's own editor - that directly produces the XML files (analogous to .o). In which case, you could code a script in .x3s format, tweak it in game, and then not be able to see your changes in the original source. It's still definitely an idea worth investigating further though - separate text-script files would be much easier to work with, if nothing else.

bunkerprivate wrote:This would also be handy for having shared information between modules, such as constants for return values like you would access with #include and enum {}, and #ifdefs for debug logging code.

This sort of thing would potentially be very useful, I agree, but I don't know how easily it could be imposed on the script language; it might be better to move some of these to a higher script language, as you mention later. I've considered this in the past (as have other people - in fact I think nuclear_eclipse even started doing this at one point, though I don't know how far he got) mainly because I think it would be very handy to have a more OO approach, e.g.

Code: Select all

Ship s = new Ship(type, owner, sector, x, y, z)
string name = s.Name
int cargoSpace = s.FreeCargo

It would make it a lot easier to do things like type checking, too. The problem with this idea is that it would mean people have to learn another scripting language, so at best this would have to be an alternative rather than the sole option.

Still, beard or not, it would be fun to try.

bunkerprivate wrote:Second thing is it would nice -- again because I'm a Unix user -- to have a separate command-line compiler which is executed by the exscriptor GUI. Of course this is a bit of a problem since there is the whole X3 file system stuff going on, but it would be very nice if I could write a Makefile to build my entire project.

The file system part is indeed the problem. It would always be pretty slow because it would need to load up the entire game data whenever it wanted to compile. Although I suppose there could be a separate syntax checking stage which would point out simple problems without needing the entire game data.

I think I will still try to have a command line version though, especially if i go down the .x3s/separate language route. It would also make it possible for people to use any editor they like for the changing the code and then use the separate compiler just for the final step, which could be handy.

bunkerprivate wrote:Of course it's your tool so your choice to design whatever fits you best!

I still haven't decided really - I do quite like the idea of a higher script language (though obviously it would mean more work), but how much separation from the XML files is practical (and feasible) I don't know - and there'd still need to be an edit capability for the normal script language anyway.

Thanks for the suggestions though! They're very useful, especially the .x3s for preprocessor #defines etc. When I come up with a more concrete plan, I'll let you know so you can comment on it - assuming you don't mind being a design consultant!

bunkerprivate wrote:- script arguments to commands are not checked for presence, eg "send incoming question ... ", "interrupt with ... ", the signal one, some others?

All others, probably! Since it's only checking for existence, not looking up parameters, this might not be too hard; if that's the case I'll try to put it in V1.032.

bunkerprivate wrote:- would be handy if it would say when textids or page ids do not exist. I don't know if it parses this stuff already.

This is a bit harder; textids is probably possible, but it doesn't remember (or even parse in) the user-defined page ids. This might be one for the Nextscriptor instead.

bunkerprivate wrote:- "$eventQueue = [THIS]->get local variable:" (missing name=whatever) does not result in an error.

I took a look at this and you'll never guess why it doesn't cause an error: it thinks it's a label!

Definitely something to fix in V1.032...

I'll try to fix the other bugs too, but it'll be a few days at the earliest.

draffutt · Post by **draffutt** » Wed, 10. Sep 08, 23:00

Whimsy wrote:
draffutt wrote:3. When attempting to load this .xml by itself give the error “an error occurred while parsing Entityname. Line 260, position 48753”:
I can't seem to get it to print out the same error twice, but I've changed a lot so I might have inadvertantly got rid of it. In any case, the file contains invalid characters (ampersands, I think it was).

What ever you did seem to have fixed this.

Tried out both the x2/x3 potions and aside from what bunker reported i don't have anything to report (as far as findings go)

Keep up the good work and thanks again for this awesome program.

Edit:
ok a couple of small quirks with find:

1st open up a few scripts then open up the find menu and select all open scripts. now search for something that contains a word that is in few of the open scripts. with the find menu open tab to the other scripts. in the past (up to .029 at least) it used to highlight all the words you were searching for; now it only highlights the last one on each of the scripts it finds.

2nd keeping the find open close one of the opened scripts. now click on find next and see what happens. that only way find will work again that i found was to close out find and before opening up find again was to select a good opened script. now when you open the find menu it seems to work again.

would it be possible to do this: say i type in "goto label start" rather then the whole line in white implying the whole line is a command. having the word after "label" a different color as well as the label "start:"?

Complier doesn't complain when you use != improperly I.E.:
if $ware != {BoFu} != {Soja Husk}
or
if $ware != {BoFu} $ware != {Soja Husk}

Whimsy · Post by **Whimsy** » Sat, 13. Sep 08, 22:49

bunkerprivate wrote:- moving an argument's position does not set the dirty flag.

All fixed now.

bunkerprivate wrote:- script arguments to commands are not checked for presence, eg "send incoming question ... ", "interrupt with ... ", the signal one, some others?

It checks for presence of the other scripts now (though it doesn't check parameters or anything).

bunkerprivate wrote:- "$eventQueue = [THIS]->get local variable:" (missing name=whatever) does not result in an error.

Fixed this one too now. It's still possible to fool the program into thinking a command is a label, but it's much harder now.

bunkerprivate wrote:- the parameter type "object command" is not available (see !ship.signal.killed).

Also fixed.

bunkerprivate wrote:- the preferences box has no cancel button, and when you click the X, it saves the preferences anyway (I accidentally pressed Ctrl+P then then typed something ^^)
- "scripts directory (where you keep your" is truncated in the preferences box.
- Ctrl+P, tab out from the preferences window, Ctrl+P again. There are two settings windows now.

And these too.

draffutt wrote:1st open up a few scripts then open up the find menu and select all open scripts. now search for something that contains a word that is in few of the open scripts. with the find menu open tab to the other scripts. in the past (up to .029 at least) it used to highlight all the words you were searching for; now it only highlights the last one on each of the scripts it finds.

I'm not entirely sure what you mean here; as far as I can tell it's working as intended. It shouldn't highlight all search words in all scripts; it should only highlight one at a time. Unless you mean something else?

draffutt wrote:2nd keeping the find open close one of the opened scripts. now click on find next and see what happens. that only way find will work again that i found was to close out find and before opening up find again was to select a good opened script. now when you open the find menu it seems to work again.

Hmm, well spotted! I've hopefully fixed this now, even if you close down all scripts while the Find window is still open.

draffutt wrote:would it be possible to do this: say i type in "goto label start" rather then the whole line in white implying the whole line is a command. having the word after "label" a different color as well as the label "start:"?

Possible, but not in this version.

This should be in V2 though (see below).

draffutt wrote:Complier doesn't complain when you use != improperly I.E.:
if $ware != {BoFu} != {Soja Husk}
or
if $ware != {BoFu} $ware != {Soja Husk}

I've fixed this too now; it was picking up the first part as an expression (upto BoFu) and then ignoring the rest. Now it detects that there's still parts of the line left and tells you about it (saying something like, "Have you missed out an operator?").

Since V1.031 has been out a while and there haven't been too many bugs, I've released V1.032 as V1.1.0. I'll keep fixing bugs if any pop up but there won't be any new features now.

Also, in case anyone's interested, I've put together a short (not terribly coherent) document with some ideas and plans etc for V2; you can get it here. Comments and suggestions welcome!

Moonrat · Post by **Moonrat** » Sat, 13. Sep 08, 23:54

I'm going to drop a note to the Queen (England not Boron) and recommend you for New Year honours mate.....

I think the "Auto completion" feature ala' the SE within X3 itself will be a big step forward.

Keep up the good work, hopefully X:TC won't ruin everything

bunkerprivate · Post by **bunkerprivate** » Fri, 19. Sep 08, 11:12

I had a look through your ideas for exs2, and they seem quite impressive! I think that you are over estimating the amount of work to make a higher-level language, though. Using lex (or flex or whatever) and yacc (or bison, bison++ etc), you can write a fairly simple scripting language which outputs basic assembler code in about 500 lines of C, including the grammar and lexical analyser parts. (Type checking and the like require a bit more hacking since they are context sensitive.) Lex-yacc sort of outputs one of the 'abstract script format' parts itself; the work is in traversing that structure and making assembler (aka codearray) to represent it. Also, as I understand it, the current parser works by using a regexp on each line. With a full almost-context-free language you can have much easier error checking and get rid of some of the esoteric messages that exscriptor gives now (eg "expecting semicolon on line X" rather than "unrecognised symbol 'get'"), and you could easily adapt an existing language syntax fairly easily since these often have other tools like doxygen, existing syntax highlighters or indenters...

A couple of other things occur to me: having the program aware of a 'project' would be useful, so you give it a directory explicitly and it will use t-files from wherever you ask (could be done easily like the -I"path" options with a separate compiler (or do windows compilers have a different syntax..?)). Secondly, in order to get rid of the filesystem issue, one could parse the directories once and use a dbm-style (yeah, unix stuff again, but it tends to be cross-platform anyway) database (or maybe even SQLlite) which is updated only for files which changed. These tasks are easily separated into distinct program tools and, in the instance of dbm reading at least, these tools already exist.

As for language features, the only thing that would additionally be useful is some form of functional programming. For instance, in !move.jump, if I wanted to send a different 'leader jumps' signal to formation followers I would need to re-implement the whole script. If I could say jump(jump_signal_function_or_script), then the entire thing becomes one line of code, or again for the formation tree traversal thing I wrote here a while back it would be good to pass a function as the visitor part. In fact the generic answer to the above example would be traverse_formation(jump_function). I don't know whether it's technically possible with scripts as variables, but it should be for 'internal' functions. Of course I am one of these people who think that a function is just another data type ^^.

The last thing is defining functions in separate files---Java makes you do this and it gets pretty tiresome in big projects. When I mentioned #include/import earlier I was angling more for a direct import of code as this is sometimes necessary anyway to make things atomic rather than a script call. (Come to think of it, I don't know what would happen if you just made all distinct commands/signals into one uberscript per command? Probably X3 is not optimised for this but interesting experiment...)

As for the current version, the only issues I found in 1.1 are that there is a cargo class "{-}", which is never recognised (it thinks it's an assignment), and there still seems to be some problems with find, especially find from start and finding when the cursor is already at the end of the document.

Edit: one more

Code: Select all

$LOGSTART = sprintf: fmt=$LOGSTART, '[%s] util.tansfercargo:', $pid, null, null, null

Doesn't pick up the unassigned variable $LOGSTART.

Whimsy · Post by **Whimsy** » Sat, 20. Sep 08, 16:09

bunkerprivate wrote:I had a look through your ideas for exs2, and they seem quite impressive!

Thanks! Glad you like the ideas.

bunkerprivate wrote:I think that you are over estimating the amount of work to make a higher-level language, though.

Generating a parser for a HLL and adding basic code generation is not necessarily difficult, but it is often more complex than you think it's going to be, at least in my experience. You need to come up with a decent grammar for starters, which is harder than it sounds, and the context sensitive parts (like type checking) inevitably make things more difficult. YACC or similar tools (in fact I was actually looking at ANTLR just the other day) would help in part though, you're right. Even so, there's still the normal X-script language to worry about.

bunkerprivate wrote:Also, as I understand it, the current parser works by using a regexp on each line.

The rubbish error checking in the current version isn't so much due to the language as due to the awful way in which the parser works. The current parser is horrendous. It basically works by stripping out any variables etc using regexes and then compares what's left against a list of "command keys". If it matches, then it's a command, if not, it might be an expression. It's then complicated further by things like the conditional prefixes - if, while etc. ANY other system will probably be superior to the current parser. Automatically e-mailing the code to a team of specially trained chimpanzees who parse it for you would still be cleaner - and probably more robust - than the current parser...

The new parser (this is for the normal X-script, not the HLL version) is a lot better, at least so far. It's probably slower, but it allows partial matches, so it can highlight and point out errors much more accurately - getting rid of the esoteric and usually unhelpful error messages you mentioned. The complexity in V2 comes from its close integration with the editor; often I need to parse only a line at a time, or even just part of a line to figure out what the cursor is on (for autocomplete etc). I haven't decided exactly how the HLL version will work, but I suspect it will be a lot simpler - there'll be a lot fewer commands, for instance, and context-sensitive autocomplete will be made easier.

bunkerprivate wrote:A couple of other things occur to me: having the program aware of a 'project' would be useful, so you give it a directory explicitly and it will use t-files from wherever you ask

I like the idea of having a project, though giving it optionally different data files might get complicated (in fact, why would you want to do that anyway?). It could be handy for sharing #defines and the like, which could be stored in the project rather than any one script. It would also make it much neater in terms of file organisation if you could work in a separate folder. Presumably it would automatically copy the resulting XML files back into /scripts when you compile, though?

bunkerprivate wrote:Secondly, in order to get rid of the filesystem issue, one could parse the directories once and use a dbm-style (yeah, unix stuff again, but it tends to be cross-platform anyway) database (or maybe even SQLlite) which is updated only for files which changed.

I definitely like the idea of "caching" the game data in a persistent database. I'd thought of checking for updated files in the scripts, but not in the game data, and that could work very nicely. That way I only need to update the database when anything changes instead of every time. It would also make the program more portable, so you could develop scripts on a machine without X3 installed as long as the database has been built.

bunkerprivate wrote:As for language features, the only thing that would additionally be useful is some form of functional programming.

The problem with advanced features like this is that ultimately everything has to convert back to the normal X-script format so that it can fit in the codearray. In which case, what you need ideally is the ability to put script names in variables (analogous to function pointers, I suppose), so that you call the value of the variable rather than a fixed script. But unfortunately you can't do that (the call command only takes strings), and I'm not sure how to emulate the capability without basically using a huge switch statement. Unless ES introduce it in TC, which will be nice.

Do you have any ideas? I suppose it could maybe work with subroutines and labels - I think it would still involve an if, but at least it would be feasible to generate it automatically, e.g.:

Code: Select all

function1:
   do something...
   $retvar.function1 = value
end sub
function2:
   do something else...
   $retvar.function2 = value
endsub

...

if $function == 'function1'
   gosub function1
   $retvar = $retvar.function1
else if $function == 'function2'
   gosub function2
   $retvar = $retvar.function2
end

Fairly ugly solution, though.

bunkerprivate wrote:The last thing is defining functions in separate files---Java makes you do this and it gets pretty tiresome in big projects.

With functions in separate files, I meant that when compiling, if you'd written a function in your HLL file, it would get compiled as a separate XML script, so you could call it properly. The alternative is to use subroutines like in the example above, but that's pretty messy. Of course, that's also probably the only way to use functions in variables...

bunkerprivate wrote:When I mentioned #include/import earlier I was angling more for a direct import of code as this is sometimes necessary anyway to make things atomic rather than a script call.

I'm not sure entirely what you mean with the #include directive. Do you mean directly importing the code, i.e. effectively copy & pasting it into the top/bottom of your script? Almost like writing inline functions? If so, I suppose it's possible, but it would be tricky - all the vars would need renaming, for example, and any returns changing to end subs, and all the call parameters and return variables would need setting up. Unless you mean only in the HLL, in which case it wouldn't be so much of a problem as long as the included file contained only functions or something, not just raw code (in which case, where do you insert it so that the imported code makes sense?).

Anyway, if you have any other ideas like these, or any suggestions for the HLL syntax/capabilities, let me know; it's easier to design things into a program than to add them afterwards, after all. And your suggestions have been brilliant so far (especially the database idea!).

bunkerprivate wrote:As for the current version, the only issues I found in 1.1 are that there is a cargo class "{-}", which is never recognised (it thinks it's an assignment), and there still seems to be some problems with find, especially find from start and finding when the cursor is already at the end of the document.

Okay, V1.1.1 should hopefully fix these. I've rewritten the Find/Replace code entirely so hopefully it will work better now.

bunkerprivate · Post by **bunkerprivate** » Thu, 16. Oct 08, 22:38

Well I had a slightly unintentional break from X3 while I RTMed my windows box but I gave exscriptor mkII a fair bit of thought in that time.

Generating a parser for a HLL and adding basic code generation is not necessarily difficult, but it is often more complex than you think it's going to be ...

You're right. I actually tried to resurrect an old grammar I was working on and it seems the theory has totally leaked out of my head in the intervening timespan. I looked at ANTLR and it seems to be very impressive! The debugger, graphing tools, and eBNF notation especially look very useful. Bison (a YACC implementation) will output a vcg graph of the language but it seems none of the viewers can actually deal with a non-trivial language; the file was something like 20k lines!!

The rubbish error checking in the current version isn't so much due to the language ...

Something I noticed in my experiments by the way: ANTLR is significantly better at error reporting than YACC and compatibles.

I like the idea of having a project, though giving it optionally different data files might get complicated (in fact, why would you want to do that anyway?)

It might be that you have multiple X3 installs or are working on different mods. I was mostly referring to having a stable running version and a development branch, though. My preference is always to have fairly little automation, and then have some other program which handles the automated parts -- for instance exscriptor might want to have a default command to run to invoke the compiler on a project, but allow it to be changed to call make (or something) so you can do it yourself.

bunkerprivate wrote: ... some form of functional programming.

The problem with advanced features like this is that ultimately everything has to convert back to the normal X-script format ...

Firstly, I didn't explain myself very well: I'm not necessarily looking for function pointers (although that would be a certain bonus) but some kind of meta-programming would be very useful. A C++ analogy would be:

template <func_ptr Function> calls_something() { Function(); }

There's no denying It would of be very hard to implement, though. [Aside: templates in general would be appropriate since structs have to be implemented with arrays]

... script names in variables ...

Again this is meta-programming. If you can fold the constants, then a call_script(some_variable, ...) is still OK. Again, implementing this -- in a sensible way at least -- would be hard. Maybe 'strict' type which forces the type to be 'const string' instead of a typedef-style type, which is more like an alias...

As for converting back to MSCI format (if that's what you meant), I think you would lose a lot of the benefit of the HLL if it has to happen. For codearray compatibility of course it's implicitly necessary, although maybe petitioning Egosoft for some added features in the assembler language would work, I don't know ^^.

Fairly ugly solution, though. ... any ideas?

Well having all functions export to a separate file is not possible anyway because sometimes the interrupt point is not what you want. I was thinking more of a copy-paste style implementation which the user decided whether a function goes into another xml (so my analogy of .o was flawed really). Exporting functions to their own xml and calling them transparently is a really good idea, though. I thought maybe a statement like "export myfunc(int) as 'something.xml'". Having multiple functions in the something.xml as you write (sort of like a .dll/.soI suppose) would be nice. You could do it with a hash table with a reasonable speed (same method is used for addressing functions in PHP, except that's in C), but again it would require function-pointers in the hash-table, so it's probably no-go anyway.

I'm not sure entirely what you mean with the #include directive.

This strikes me as the least straight-forward bit to implement. It's not possible (I think) to have an include which has no operational statements in it -- you probably need some global variables at some point at least, or a call to an init function for the library. A copy/paste -- or at least the appearance of copy/paste -- was what I was talking about really. My assumption was that variables would either be on an "array alloc"ed stack for recursive functions, or would just be numbered in some way (maybe the problem is avoided by having a scope number and a variable number or something?).

I experimented a bit with this, and more-or-less decided the best way would be to require a compile to some kind of .o and a linker step to .xml. The main problem I could see was that optimisation opportunities may be lost doing that since things are inlined implicitly, rather than being written in a .h instead of .c (the compiler I was writing is an optimising source-to-source compiler, born of a desire to a) not write bin/sh anymore and b) well, to write an optimising compiler

).

Anyway, all this said, the lack of replies to this topic should probably be an indicator not to break your back over writing an HLL (or maybe we scared people off talking about technical matters!?). I would probably use it but there could be a decent middle way if one could just share some declarations between MSCI scripts -- the top of my scripts are always about 100 lines of return code constants.

Whimsy · Post by **Whimsy** » Tue, 21. Oct 08, 00:32

bunkerprivate wrote:Well I had a slightly unintentional break from X3 while I RTMed my windows box but I gave exscriptor mkII a fair bit of thought in that time.

Glad to have you back! So will you be trying X3TC or are you sticking with X3R for the time being?

bunkerprivate wrote:Something I noticed in my experiments by the way: ANTLR is significantly better at error reporting than YACC and compatibles.

Good to know, thanks. ANTLR definitely looks like a good bet.

bunkerprivate wrote:It might be that you have multiple X3 installs or are working on different mods. I was mostly referring to having a stable running version and a development branch, though. My preference is always to have fairly little automation, and then have some other program which handles the automated parts -- for instance exscriptor might want to have a default command to run to invoke the compiler on a project, but allow it to be changed to call make (or something) so you can do it yourself.

Ah, I get it now - that sounds like a very useful idea actually. It should be simple enough; I can bundle some of the options in a project file instead, so it'll load data per project instead. Though it will likely have to be restricted to having only open one project at a time or else it'll potentially need multiple datasets loaded.

bunkerprivate wrote:Firstly, I didn't explain myself very well: I'm not necessarily looking for function pointers (although that would be a certain bonus) but some kind of meta-programming would be very useful. A C++ analogy would be:

template <func_ptr Function> calls_something() { Function(); }

There's no denying It would of be very hard to implement, though. [Aside: templates in general would be appropriate since structs have to be implemented with arrays]

Amazingly enough, Egosoft appear to have granted our wishes! I haven't tested in detail but from my cursory look at the new MSCI V41, there are new commands to call scripts by using their name, i.e. by giving them a string. So effectively you can have function pointers now, since you can store script names in string variables. There's lots of other funky features too - secondary signals (so we can have a proper Observer pattern, unless I'm mistaken), promoting passengers/pilots etc to proper objects (no more fiddling around using local vars all the time), even regular expressions (including for accessing global/local vars), and of course the new custom menus (which look great).

As for templates/generics, that might well be handy in the HLL. I'll put that on the list of things to look at.

bunkerprivate wrote:As for converting back to MSCI format (if that's what you meant), I think you would lose a lot of the benefit of the HLL if it has to happen. For codearray compatibility of course it's implicitly necessary, although maybe petitioning Egosoft for some added features in the assembler language would work, I don't know ^^.

I agree it definitely constrains what's possible, but that's unavoidable, since as you said it has to be ultimately converted to the codearray whatever happens (and the codearray has a 1:1 correspondence with the standard MSCI language). But personally I think it would be important in any case; for example, if you wrote a script in the HLL and in game you tweaked one thing, then that change ought to be reflected in the original source. If not, you either have to use the HLL as the initial source and move onto standard scripting afterwards, or you need to go back to the original source and make the change manually again (which could be a pain - the in game editor is the only way of debugging, for example, and for simple changes it is still faster than an external editor).

bunkerprivate wrote:Well having all functions export to a separate file is not possible anyway because sometimes the interrupt point is not what you want. I was thinking more of a copy-paste style implementation which the user decided whether a function goes into another xml (so my analogy of .o was flawed really).

The interrupt point argument is a good one - in some cases a proper call might not be what you want. In which case, maybe some C/C++-style inlining functionality is more what you mean then?

bunkerprivate wrote:Exporting functions to their own xml and calling them transparently is a really good idea, though. I thought maybe a statement like "export myfunc(int) as 'something.xml'". Having multiple functions in the something.xml as you write (sort of like a .dll/.soI suppose) would be nice. You could do it with a hash table with a reasonable speed (same method is used for addressing functions in PHP, except that's in C), but again it would require function-pointers in the hash-table, so it's probably no-go anyway.

I was going to generate the filenames automatically (e.g. <thisscript>.fn.<functionname>) but allowing the user to export manually is another possibility. The reason I like the idea of exporting each function as a separate file, though, is because it resolves to a single call command in the end; inlining can be done during compilation, but anything at run time - like having a DLL-style multiple function file - would be considerably more complicated since it would presumably involve some other form of invocation.

bunkerprivate wrote:This strikes me as the least straight-forward bit to implement. It's not possible (I think) to have an include which has no operational statements in it -- you probably need some global variables at some point at least, or a call to an init function for the library. A copy/paste -- or at least the appearance of copy/paste -- was what I was talking about really. My assumption was that variables would either be on an "array alloc"ed stack for recursive functions, or would just be numbered in some way (maybe the problem is avoided by having a scope number and a variable number or something?).

With the include mechanism, I was thinking it would be more for defining constants or globals or HLL types etc (which may or may not need resolving into the underlying MCSI, analogous to the #defines versus normal vars in C, say) rather than inlining code, but now I realise what you mean - you want to actually import code from one file into another directly. I can see why that would be handy at times, e.g. in the initialisation of constants etc at the top. But especially if it allows any form of code, is it worth the effort, given how complex it would be to implement? The variable renaming could be tricky even at compile time, but maintining some kind of custom stack at run time would be a nightmare.

There's clearly a case for defining constants, globals, and types etc at a project level - these could then all be tacked on to the tops of all the relevant script files produced, say, or possibly collected together into an init script - but my instinct here would be to limit the include functionality to simple definitions or at most short, inlined functions (with no calls inside them) rather than allowing any possible code.

bunkerprivate wrote:I experimented a bit with this, and more-or-less decided the best way would be to require a compile to some kind of .o and a linker step to .xml. The main problem I could see was that optimisation opportunities may be lost doing that since things are inlined implicitly, rather than being written in a .h instead of .c (the compiler I was writing is an optimising source-to-source compiler, born of a desire to a) not write bin/sh anymore and b) well, to write an optimising compiler).

Believe me, any ideas of optimisation are so far over the "maybe later" horizon that it wouldn't be much of a loss. Even the HLL itself is secondary in priority. But you're right, some kind of intermediate step might be necessary, depending on what functionality is going to be included.

bunkerprivate wrote:Anyway, all this said, the lack of replies to this topic should probably be an indicator not to break your back over writing an HLL (or maybe we scared people off talking about technical matters!?). I would probably use it but there could be a decent middle way if one could just share some declarations between MSCI scripts -- the top of my scripts are always about 100 lines of return code constants.

I tend to think the HLL would certainly be a very "niche" capability, probably limited to you and me

, though some of its capabilities (like project-level definitions) might bleed through to the normal script language. As for scaring people off, I wouldn't be surprised! Some of the technical stuff we've come up with scares me, let alone anyone else...

Anyway, at present the Exscriptor V2 is still a long way off, and I'd probably release an initial version without the HLL first anyway (since I expect that part to take longer). In the mean time, I'm looking at updating V1 to add X3TC compatibility - in a strictly limited use-at-your-own-risk way - mainly for personal reasons since I just cannot use the ingame editor for more than a few minutes at a time without either going crazy or getting distracted by the game itself (I used to script on long flights, but having to break off whenever you fly through a gate is a pain). From what I can tell, there are no major impediments; there are some changes to the various data files - not least the way they're compressed, since now a PCK is just a simple ZIP - but I've already got it to load those in more or less okay, I think. The only thing left to do is to write out the command/param lists, which is just a boring manual task, and then test it to see whether or not it actually works.

So with any luck, and assuming I can tear myself from the game long enough, there might be an Exscriptor V1.2 for X3TC out soon (but no promises!).

bunkerprivate · Post by **bunkerprivate** » Wed, 22. Oct 08, 00:38

Whimsy wrote:Glad to have you back! So will you be trying X3TC or are you sticking with X3R for the time being?

Heh, it sounds really attractive as you explained the features. The menus especially I'm looking forward to, since I think the UI in Reunion really let the game down. Of course I play it like an RTS really so maybe I'm outside the use-case. But I'm not sure my PC can even run it anyway -- It barely runs Reunion, tbh ^^.

Though it will likely have to be restricted to having only open one project at a time or else it'll potentially need multiple datasets loaded.

This normally isn't an issue with dbm databases.

there are new commands to call scripts by using their name, i.e. by giving them a string.

Great! From my point of view, it would actually be quite unlikely that inline template functions would really be necessary, `script-pointers' certainly are for reasonably maintainable code.

It seems I didn't quite communicate exactly what I was after though -- or at least my motivation, so I'll try and remember some code I was working on. It's basically just a simple event handler loop (apologies if I get this a bit wrong):

Code: Select all

// Representation of the 'pseudo-struct'
$EvtStruct.size = 4
$EvtStruct.next = 0
$EvtStruct.prev = 1
$EvtStruct.type = 2
$EvtStruct.data = 3 

// Alias
$queue.front = $EvtStruct.next 
$queue.back = $EvtStruct.prev 

// Base ptrs: when they're equal the queue is empty
$queue = = array alloc: size=2
$queue[$queue.front] = $queue
$queue[$queue.back] = $queue

while [TRUE]
  $front = $queue[$queue.front]
  if $front != $queue
     $RenoveAfter.place = $queue
     $elt = null
     gosub Queue.RenoveAfter:

     $event = $elt[$EvtStruct.type]
     $data = $elt[$EvtStruct.data]
     if $event == $EventType.enemy 
      // call event handler
    else // others
    end

    // $elt is discarded
  end
  // ...
end

// ...

// The parameter is $RemoveAfter.place and the return  is $elt
Queue.RenoveAfter:
$elt = $RenoveAfter.place[$EvtStruct.next]
$RenoveAfter.oldNext = $elt[$EvtStruct.next]
$RenoveAfter.oldNext[$EvtStruct.prev] = $RenoveAfter.place
$RenoveAfter.place[$EvtStruct.next] = $elt[$EvtStruct.next]
   
$elt[$EvtStruct.next] = null
$EvtStruct.prev] = null

(And naturally there is another process somewhere allocating EvtStructs and putting them on this queue.)

So the `function' I want to declare is only 6 lines (less if the language allowed more loose expressions). I can't possibly have it in another file because parts of the code rely on it being atomic (actually this particular example's OK, but there's plenty of instances when it's not - I'd rather explicitly yield). What I'd like to see in an HLL is syntactic sugar for the the funky pseudo struct, and a true function for the gosub which I can import from multiple files. That's why I think it's a valid feature anyway. This is really the same functionality as the template function example -- pure meta-programming. I'm of course completely guessing on how X3 really performs between big files vs. lots of script calls.

Anyway, you see I long for the day when I can #include <x3.h>

and the codearray has a 1:1 correspondence with the standard MSCI language)

It's a shame I was rather hoping there was hidden ability to do this kind of stuff. Especially to set the program counter to a value obtained from a variable. It's something I probably would have put in just for completeness, heh.

for example, if you wrote a script in the HLL and in game you tweaked one thing, then that change ought to be reflected in the original source.

I've found that tweaking things in the game is never really sufficient to fix bugs anyway; I never touch scripts in-game, except ones I have specifically designed to be user editable. I find I need to go in and put in a massive set of logging commands, or I need to tab out and analyse an old log and the X3 environment doesn't really cut it -- I honestly can't make head nor tail of the integrated debugger! Probably this is because I skipped MSCI and went directly to Exscriptor, though.

maybe some C/C++-style inlining functionality is more what you mean then?

Yes, and for bigger functions or compositions of functions I would either export or manually put them in their own file. This is essentially what I do already, except it involves Ctrl+C and Ctrl+V which causes me physical pain every time I do it ^^

I was thinking it would be more for defining constants or globals or HLL types etc

Yes, that's definitely the most important feature. There's no real need to make them preprocessor defines, but that can be added later as an optimisation pretty easily. In fact the only place I would really use a preprocessor is in the case of logging statements, of which my files usually have a massive amount and a `release' build would perform somewhat better if the dead code when $DEBUG=[FALSE] could be removed.

is it worth the effort, given how complex it would be to implement?

I think you hit basically the same problems by just having variables rather than actual code. It's more of an organisational issue -- how do you know which functions/consts got inlined?, how do you stop the parse from taking all day?, how do you get the bloody parser to be *actually* reentrant instead of just saying it is!?! I'm probably over thinking it, really, how it is implemented just seemed have a dis-proportionally large affect on the rest of the design.

The variable renaming could be tricky even at compile time, but maintining some kind of custom stack at run time would be a nightmare.

I was thinking of optimisation again: you need to number variables anyway so you can reuse the ones which are out of scope or aren't being referenced anymore. For a stack, you just reset the numbers to zero when you get to a function and use them as array indices; inlining just adds an offset to the base var number of the function. (I'm sure it wouldn't turn out so simple though ^^)

but my instinct here would be to limit the include functionality to simple definitions

From the previous example, I was really aiming for "import queue" and have that data structure there. Again, I'm not sure how it would turn out but I think that having any kind of declaration in the imports boils down to the same problem, if you handle all script generically of course. I suppose I good hack would be just to specifically parse a bunch of assignments.

(edit:)

There's clearly a case for defining constants, globals, and types etc at a project level

Being a Unix guy, I always tend to think in terms of things which are very simple, generic, and provide a single function, and then have higher level interfaces to them. The `project-level' defining doesn't really appeal to me since it's a very specific implementation which could be done more flexibly by the functionality "include global.h". (This in particular is useful for distributing library code of course.) When I mentioned some things which are common to the project, my assumption was really that it would be a neat user-interface for the general concept of parameters to the compiler. So, say you wanted to have a debug build and a release build you would define a macro RELEASE_FLAGS and DEBUG_FLAGS, then when you pess `build' in the compiler, Exscriptor could run "cmdline-compiler ${${buildtype}_FLAGS} script.exs -o whatever.xml". I think this is really the `right' philosophy, though it's always a matter of opinion -- it's interesting to see the differences in assumptions between windows developers and unix developers. When I implement stuff like this my preference is normally just to dump the problem on the user and give them a box for "command to run", which for me would usually be `/usr/bin/make'.

(end edit)

Believe me, any ideas of optimisation are so far over the "maybe later" horizon that it wouldn't be much of a loss.

But optimisation is so cooool! (And doubly pointless since there's no tool to benchmark the results, but never mind that!)

(I used to script on long flights, but having to break off whenever you fly through a gate is a pain)

It still does that?! Shame on you, Egosoft!

So with any luck, and assuming I can tear myself from the game long enough, there might be an Exscriptor V1.2 for X3TC out soon (but no promises!).

I look forward to it! Seems X3TC is on steam so it's not a big deal to impulse buy it at some point, and I can lend a hand testing.

Whimsy · Post by **Whimsy** » Thu, 23. Oct 08, 01:33

bunkerprivate wrote:Heh, it sounds really attractive as you explained the features. The menus especially I'm looking forward to, since I think the UI in Reunion really let the game down. Of course I play it like an RTS really so maybe I'm outside the use-case. But I'm not sure my PC can even run it anyway -- It barely runs Reunion, tbh ^^.

You might be okay with it, but you probably will need a faster computer then; X3TC is a pretty demanding game, especially memory-wise. But the more I play the game, the more new things I find that I really like (and I don't just mean in the script language). The custom script menus are indeed brilliant - LV posted a tutorial so you can see how they work, and they're very simple. A massive step forwards, I think. Playing these games like an RTS gets a bit risky at times though; one time I gave up on an X3R+XTM game in disgust after I lost a fully-armed Python to a Xenon Q in five seconds of out-of-sector combat. If they made RTS games like that, where 100 million credit units get wiped out in a second if they were offscreen at the time, nobody would play them!

bunkerprivate wrote:This normally isn't an issue with dbm databases.

It's not just the database that's the issue though, it's also the UI and the rest of the innards: multiple selection lists of objects etc would then be needed too, for instance, displaying the appropriate one depending on which script of which project you have open at the time. I think one project at a time is a sensible constraint (and more importantly, saves me effort

), and if need be I suppose the user could always open up multiple instances of the Exscriptor.

bunkerprivate wrote:Great! From my point of view, it would actually be quite unlikely that inline template functions would really be necessary, `script-pointers' certainly are for reasonably maintainable code.

Agreed - the 'script pointers' are definitely useful. You can also see them in action in Cycrow's cool new command queue scripts if you're interested.

Templating... well, we'll see.

bunkerprivate wrote:It seems I didn't quite communicate exactly what I was after though -- or at least my motivation, so I'll try and remember some code I was working on. It's basically just a simple event handler loop (apologies if I get this a bit wrong):

I see what you mean, and I'll admit something like that could well be handy for building up libraries of data types and associated functions that you can include/import for use in different scripts. In that case, I would think something like this:

- a 'type' or 'struct' definition for user-defined data types (like queues etc). May or may not be generic/templated (probably not), depending on how complex that proves.

- a 'define' for constants etc. Probably wouldn't convert to an actual variable assignment.

- some kind of 'global' variable declaration; I was planning this anyway as a way of supporting the existing global variables in the MSCI.

- a 'shared function' or 'inline function' definition, which can be imported and inlined. I think I would restrict these to prevent recursion etc and probably prevent any form of script call too. This would end up as a sort of decorated gosub in the MSCI I suppose.

- an 'exported function' definition, which ends up as its own script file and thus resolves to just an normal script call in the MSCI.

- some kind of 'import'/'include' directive that allows you to load all of the above from a named file. Any other code in them would be ignored and in fact I might make it so that includeable files can only contain the above, to keep things tidy; almost like headers, really.

That way you could have a "Queue" library script file, say, which would include the definition of the queue type plus the associated utility functions (add, remove, insert etc). Your "import Queue" idea, basically.

bunkerprivate wrote:Anyway, you see I long for the day when I can #include <x3.h>

If only...

bunkerprivate wrote:It's a shame I was rather hoping there was hidden ability to do this kind of stuff. Especially to set the program counter to a value obtained from a variable. It's something I probably would have put in just for completeness, heh.

Remember that it's just a game's built-in scripting language - I think it already offers pretty impressive capabilities, especially considering the improvements since the X2 version, but I don't think it was ever envisaged as a true programming language. Assembly with access to program counters and greater access to the underlying functions might be nice, but would probably be irrelevant to 99.99% of the game's users and thus wasted effort on Egosoft's part... I suppose they could have gone down the route of using an existing language to do it, Lua or Python or something, but that would have meant even larger changes to the engine and would have lost the simplicity of the existing MSCI which makes it so popular. Come to think of it, I believe they did have a C-like language they used internally for mission programming etc, but it was a development tool and never meant to be available to players of the game.

Actually I wonder what percentage of X players use the script editor, or even make use of other people's scripts?

bunkerprivate wrote:I've found that tweaking things in the game is never really sufficient to fix bugs anyway; I never touch scripts in-game, except ones I have specifically designed to be user editable. I find I need to go in and put in a massive set of logging commands, or I need to tab out and analyse an old log and the X3 environment doesn't really cut it -- I honestly can't make head nor tail of the integrated debugger! Probably this is because I skipped MSCI and went directly to Exscriptor, though.

To be honest I've never used the in-game debugger much either, though it can be useful at times. The extra information you can view on ships etc to see what scripts they're running is probably the most helpful for debugging, I find. Anyway, I think being convertible to/from MSCI will be useful for the HLL, if only to save time of dropping out and reloading the game each time, but if it proves too complicated then I'll drop it; like you say, if necessary we can make the changes to the original source HLL anyway.

bunkerprivate wrote:In fact the only place I would really use a preprocessor is in the case of logging statements, of which my files usually have a massive amount and a `release' build would perform somewhat better if the dead code when $DEBUG=[FALSE] could be removed.

That sounds like a good idea, I agree. The debug statements can then just be commented out when compiling in "release" mode, I suppose, or even removed altogether (though that would break the reverse compatibility).

bunkerprivate wrote:I'm probably over thinking it, really, how it is implemented just seemed have a dis-proportionally large affect on the rest of the design.

I haven't got far enough yet to think in any detail about the implementation (the HLL is still low priority), but I agree that it will almost certainly be complicated! That's why I'm trying to keep the language itself simple, otherwise Exscriptor 2 may well take two years again to finish...

bunkerprivate wrote:I was thinking of optimisation again: you need to number variables anyway so you can reuse the ones which are out of scope or aren't being referenced anymore. For a stack, you just reset the numbers to zero when you get to a function and use them as array indices; inlining just adds an offset to the base var number of the function. (I'm sure it wouldn't turn out so simple though ^^)

We might have our wires crossed here, but with the variable renaming, I was thinking more along the lines of namespacing, just so that imported vars don't get confused with ones already present. I think what you're talking about is a bit beyond the scope of the scripting language - that level of optimisation would presumably belong in whatever interpreter runs the scripting language in-game. If you tried to do some of this stuff in the MSCI - like recursive gosubs - I think the very fabric of reality would rip open and suck you through into some nameless void beyond...

bunkerprivate wrote:Again, I'm not sure how it would turn out but I think that having any kind of declaration in the imports boils down to the same problem, if you handle all script generically of course. I suppose I good hack would be just to specifically parse a bunch of assignments.

I'm not sure what you mean here; surely simple variable declarations would be no problem, since they'd map directly into either standard or global MSCI variable declarations? In some cases (like constants) the variable could be skipped altogether and the value just substituted. Even types are simple enough if they work along those lines. Inlined functions would be more tricky because they could potentially introduce dependencies on other parts of the script code (whether imported or not), which is why I think it would be neatest to make sure they're self-contained and can't call anything.

bunkerprivate wrote:The `project-level' defining doesn't really appeal to me since it's a very specific implementation which could be done more flexibly by the functionality "include global.h". (This in particular is useful for distributing library code of course.)

When I said that, I was thinking more along the lines of importing files into a project rather than into a specific script, so that you would just select a bunch of libraries to include and then you can use the defs in any script in the project. However, that does mean me checking to see whether things are used in every file, and thus whether or not to actually include them, so maybe shifting the burden onto the scripter isn't such a bad idea after all.

bunkerprivate wrote:When I mentioned some things which are common to the project, my assumption was really that it would be a neat user-interface for the general concept of parameters to the compiler. So, say you wanted to have a debug build and a release build you would define a macro RELEASE_FLAGS and DEBUG_FLAGS, then when you pess `build' in the compiler, Exscriptor could run "cmdline-compiler ${${buildtype}_FLAGS} script.exs -o whatever.xml".

I had this in mind too, but I wasn't thinking of this as part of the language (except possibly an #if DEBUG style directive) and more as just simple project options. Incidentally the plan is to have both a built-in compiler (which is tightly integrated, so it can provide feedback to the UI) and a separate cmd line one, which would still provide error reports but would not need the extra interfacing capabilities.

bunkerprivate wrote:I think this is really the `right' philosophy, though it's always a matter of opinion -- it's interesting to see the differences in assumptions between windows developers and unix developers. When I implement stuff like this my preference is normally just to dump the problem on the user and give them a box for "command to run", which for me would usually be `/usr/bin/make'.

In general, I agree with you - it's neater to have some general underlying functionality and then just pop a more specialised interface to it on top; that way you can tailor your interfaces without needing to duplicate what's underneath. Also personally I too prefer to have the ability to customise things and do things my own way. But some people don't, hence the existence of things like "wizards".

I suppose any differences in assumptions between Windows and UNIX developers might stem from the differences in typical Windows and UNIX users.

The average Windows user probably wouldn't know what to do if presented with a blank "command" box; imagine if there was nothing in the Start menu apart from the Run box - there'd be uproar! Yet UNIX users are probably at least as comfortable with a CLI as they are with a GUI, if not more so. Developers have to target their users, so their thinking will reflect that, I suppose.

In this case, I'm trying to stay close to the simplicity of the original MSCI (and however long-winded it can be at times, it is brilliantly simple to use) and add on any extra, more powerful functionality after that as another option so that you're not forced to use it. Someone who just wants to write a cheat script to create a free ship is not going to be interested in the HLL or the capabilities it offers, for instance. Though having said that, I suppose most of the people using the tool are unlikely to be casual scripters anyway.

bunkerprivate wrote:But optimisation is so cooool! (And doubly pointless since there's no tool to benchmark the results, but never mind that!)

I'm not sure that much optimisation is possible anyway; clearly, some commands will be more costly than others, e.g. "get pilot name" will no doubt be quicker than "fire lasers on target", but it's not as if there are any alternatives if you want to shoot someone. The only real meaningful scope for optimisation is at the algorithm-level, I think, and that's in the scripter's hands.

bunkerprivate wrote:I look forward to it! Seems X3TC is on steam so it's not a big deal to impulse buy it at some point, and I can lend a hand testing.

Well, there's no hurry on the testing Exscriptor part, believe me, but it's well worth playing for its own merits.

bunkerprivate · Post by **bunkerprivate** » Fri, 24. Oct 08, 21:56

Playing these games like an RTS gets a bit risky at times though; one time I gave up on an X3R+XTM game in disgust after I lost a fully-armed Python to a Xenon Q in five seconds of out-of-sector combat.

It's true. I'm kind of in an arms race with myself in X3: I write a script to make RTS easier (that event-loop queue thingy is from a QRF-type script) and then it becomes too easy so I give the enemies some new capabilities!

In that case, I would think something like this: ...

Yes, this is the kind of thing I was thinking of.

Assembly with access to program counters and greater access to the underlying functions might be nice, but would probably be irrelevant to 99.99% of the game's users

Indeed, though I was really talking about MSCI's internal program counter (I only assume it has one). I usually think it's best to make the low level interface have everything exposed, but abstract it to something very simple like MSCI. This way modders essentially do the effort for you --- companies like Valve do this and it gives them a massive developer community. Python/Lua based would of course be my ideal - if MSCI commands were implemented on top of python extension .so's it'd be great, though I have no idea how that's even theoretically achieved!

(something about include files)
I haven't got far enough yet to think in any detail about the implementation (the HLL is still low priority), but I agree that it will almost certainly be complicated!

Well I actually looked into this further and it seems (at least with flex) the `right' way to do it is just keep the parser as it is and set the lexer to load tokens from the included file, then go back down the stack when it's done, so there's no problem of combining parse trees or anything as I was initial thinking.

optimisation ... number variables

We might have our wires crossed here, but with the variable renaming, I was thinking more along the lines of namespacing ... they'd map directly into either standard or global MSCI variable declarations?

The optimisation is fairly simple: in MSCI you often have

$v1 = something
use($v1)
$v2 = something_else
use($v2)

The interpreter can't tell $v1 doesn't get referenced again, but a scoped compiler could, so you get:

{ // this scope is specified by the HLL syntax somehow
$v1 = something
use($v1)
}
// no need to allocate a new variable
$v1 = something_else
use($v1)

It's all hypothetical of course. I'm really thinking `cool things a compiler could do' rather than `things that have practical benefit'. ^^

I'm not sure that much optimisation is possible anyway; clearly, some commands will be more costly than others

I'm thinking more of general optimisations like constant folding, unused code removal, common subexpression relocation, loop invariant checking... Natuarlly the `assembler' commands can't be helped, but if there is a very complete HLL, then these kinds of optimisations would be valid (though I'm not sure they'd ever make a noticeable improvement on gameplay). Just for example, gcc has function attributes so you can write a function: "int factorial(int n) __attribute__(const) { ... }" which tells the compiler that for each n, factorial() returns the same value (I can't remember the proper term for it...). The compiler then causes the code to cache calls to factorial rather than recomputing every call.

egosoft.com

egosoft.com

Exscriptor - new external script editor