embperl/wsh

[Eglin]

Was on the fence about whether to post this here or in the feature request forum, so please send all flames to /dev/null.

I am interested in enhancing eqemu by embedding it with either Perl or WSH (which would indirectly allow for Perl under Windows). I have lots of experience doing this type of thing, and think it could have lots of cool implications.

I am currently investigating the code, trying to decide where to shoehorn the goods in. I suspect that I would shoot for extending the quest code to allow "real" code. I haven't yet decided whether I would try to add some sort of script tag like HTML does or to try to make the embedded language a superset of the existing scripting language or just add hooks to filter the scripts through external apps or what. At any rate, augmenting the script code would make it trivial to implement NPCs that use Web::Babelfish to translate languages or SOAP::Lite to query for real-world weather conditions or IRC gateways or any number of really sexy stuff.

Any suggestions will be most welcomed. If I don't "see" any real interest in this, I'll probably add support for the WSH via COM (which is supa-simple and supports vb/python/perl/ruby/etc, but is Windows only). Otherwise, I'll focus on doing something more multiplatform (probably centered around Perl).

[krich]

This sounds pretty cool. Make it happen, but make it happen on Linux :wink:

Regards,

krich

[kathgar]

I'm not sure, perl/whatever for commands and quests/scripts could be easier for some, harder for others. There is also the question os effciency and maintabililty.
--Adds to the list of things to discuss...but sleep for now.

(Note: I'm not exactly sure the usefulness of things such as the babelfish or whatever, however if we had a fullfledged scripting language we would not need to make additions to our language whenever we needed something.)

[Trumpcard]

We had talked about doing that a long time ago, but Wes opted on redoing the parser.

It sounds like a great idea, as long as it's a configurable through ifdefs or a variable whether to use the old parser code, or the new embedded one.

There are alot of cool functions you could use in perl that would be very difficult to implement elsewhere.. Also, would make programming the scripts alot easier as perl is a widely used and accepted programming language, rather than having to learn a picky 3rd party scripting language, even a slimmed down one like this one.

I guess using the babelfish mod would save a frenchie/spainard the trouble of converting..

[kathgar]

You could easily just run all of the quest sayings through babelfish if you wanted a Spanish server. It would be kind of weird if you hailed an npc and it just hung there trying to connect to something for the response though... people could easily think it's just broken instead of waiting.

[Eglin]

if we had a fullfledged scripting language we would not need to make additions to our language whenever we needed something.)
That isn't actually 100% true. If I were to do this in a Windows-only way, I'd expose eqEmu's guts via COM, so that eqEmu objects could be easily accessed via languges operating under the scripting host. It is really sexy, and super easy. It all depends on COM, though, and as far as I know, Linux doesn't have anything even close to this kind of capability/effort ratio. I am _not_ interested in getting into cross-platform corba or anything. So, embedded Perl is the next best thing. Unfortunately, without exposing eqEmu's objects via COM, we are limited to writing call-back type subs in Perl, synchronously moving variables to or from Perl, or direct manipulation of the Perl stack. This isn't a big deal, since call-backs are well-suited for event driven setups, like the one already written into eqEmu. However, the callbacks only work one-way - it isn't easily possible to have Perl scripts call existing C(++) functions. This means that most of the scripting functions will need to be implemented by maintaining state and variable passing, rather than interactive invocation. This means that the c code will have to be modified to expose some types of new functionality (essentially like the existing code).

I haven't started writing code yet, but this is how I'm envisioning it at this point... If Perl is compiled in, users will be able to replace .qst files with files containing perl code. The code files will contain subs, one for each event that the coder wishes to have handled. Each file will be read into a seperate namespace or possibly be wrapped into a class... not sure about this, yet, but that would probably be nicer than requiring a prefix for each sub or some such thing... Each parser object (read: each instance of the parser class) will have an instance of a perl interpreter. The interpreter would, thusly, be loaded only once and would therefore have minimal overhead (assuming we don't get excessively complex scripts). The memory footprint would vary according to global variable usage. Instead of exposing callback hooks, like "say" or "dbspawn" or whatever, variables would be exposed to indicate to the main parser module what actions the script needs it to take on its behalf. Likewise, variables (like $name) would be exposed to the interpreter in the same way that they currently are. All of this should be possible to add in fewer words than have been used in this post so far :)

About global variables... this part is very cool, and is weighing in on my scope resolution decisions. Global variables would be avaliable to all scripts running under the same parser (i.e. to all npcs in a zone). This could make for some very interesting quest implementations... You could keep track of who's gone where or send messages between multiple npcs or cache global data (like database connections). All kinds of neat stuff.

[Merth]

The quest system is an important part of the emu. I haven't ever worked with them, but I see frequent requests for them.

I don't know anything about Perl, but if it will work in both environments, great.

One thing I have noticed is a lot of professional game companies use Python. Of course, I don't know anything about that - but if you're in this to learn, Python may be a good route to take since it is so widespread in this industry. Personally, I want to learn python and if someone were to implement quests with it, that goal would be that much easier.

On the other hand, I see perl as more of a business language, which doesn't interest me much. I'm sure it could do a fantastic job, but I'd rather get paid to learn something like that.

[krich]

Merth,

Perl is exceedingly good and fast at text parsing and manipulation. That, in itself, makes it ideal for the quest system.

It's also got built-in support for MySQL through the DBI module :D

Regards,

krich

[Trumpcard]

PERL's harder than PYTHON to integrate into a C++ program.. I agree though, PERL is faster...

[Eglin]

Yeah, Python is a _lot_ easier to integrate into apps. I like it a lot, too. In terms of capability, both languages are equivalent. Perl is, IMHO, probably a bit better at one-liners, but python is definately much much better with "real" data structures. Discounting that each language has some methods of running code in each other's language (as well as others), I'd say that Perl's a better choice for embedding just because of the sheer number of avaliable modules, scripts, and informational resources.

At any rate, I refreshed my pristine codebase from cvs and started writing code this morning. It is going surprisingly well, and I don't think it will take long to get a first revision implemented. I will keep you all posted.

p.s. - python is sensative to whitespace... that kind of fortran-like pickyness turns me off quite a bit. I think that I would recommend Ruby over all else, if not for the relative scarcity of enthusiastic supporters. If you are into oo, you will really like Ruby. It is tight without feeling restrictive. Lots of syntactical sugar, too.

p.s.s - the perfect language would have an anonymous global deque... Things like pattern-matches in a loop would automatically (or optionally) toss their results onto the deque. Makes syntax a little less than ideally clear, but such a huge timesaver. NQL is the only language I've seen that is even close to getting this right (sadly, NQL is not a "free" language.)

[Eglin]

OK, I've gotten this to the point where it seems usable. I'd like to submit it directly for consumption into the cvs tree, rather than just post diffs, since I'm sure that there will be many rapid small fixes/changes/suggestions after its release, and having it already in CVS would make application of such changes go much more smoothly. I would appreciate it if a dev would inform me of the preferred way to submit. I have compiled the changes under Linux 2.4 w/ Mandrake RPMs, but have only tested it under Windows w/ VC7 and Activestate Perl. Failing to define EMBPERL or EMBPERL_PLUGIN after merging my changes will result in _almost_ the same code that I started with. I had to make a few changes, either for utility or to get Perl shoehorned in, but they are for the most part minor. Of particular importance to whomever does the merging, though... I had to change the order certain files are #included in, in order to avoid overwriting macros. I wasted like 45 minutes trying to figure out which portions of eqemu's namespace pollution was interfering with Perl's namespace pollution, then gave up. So... take note when making the merge (assuming I haven't scared you off :).

Now, having said all of that, let me tempt you with the benefits. After compiling with the perl options, scripts can/should/must be written in perl. There are currently no facilities for mixing perl and native scripting. Defining EMBPERL_PLUGIN also exposes a simple plugin interface. When zone.exe starts up, all subroutines in ./plugin.pl and ./plugins/*.pl (if they exist) are compiled into package plugin. These subroutines can be called from the EQ client with the #plugin command for all users with high enough access privileges (defaults to >= 200). A #peval command has also been added which allows for direct evaluation of Perl statements.

Writing basic quests w/ embedded Perl (embperl) is almost identical to writing them in the native scripting language. The most significant change is that functions should be prefixed with 'quest::'.

An example:

EVENT_SAY {
if ($1- =~= "Hail") { say("Hello, $name!");}
}

in native script becomes:

sub EVENT_SAY {
if ($text=~/Hail/) { quest::say("Hello, $name!");}
}

in embperl.

Here's an example plugin:

sub calc
{
eval "print PLUGIN $_[0];";
}


Paste this into a file such as .\plugins\calc.pl, and voila! You have a simple calculator. You can use it like: #plugin calc "3*2**2" Note the use of the PLUGIN filehandle? That directs output to the client window. In other respects, it works pretty much like normal perl. Here is a slightly more complex example - it shows that you can use normal modules and do some meaty stuff. You can paste it into a file called .\plugins\iplookup.pl (or whatever). To use it, do #plugin lookup "123.456.678.890" (you can use the #iplookup command to get player ips)

sub lookup
{
use LWP::Simple;
my $url = "http://ws2.serviceobjects.net/gpp/GeoPinPoint.asmx/GetLocationByIP?IPAddress=".$_[0]."&LicenseKey=WS1-ITN1-XMC1";
my $xml = get($url);
if(!$xml or $xml=~/<Error>/)
{
print PLUGIN "ip lookup failed\n";
return;
}
my $city = $1 if $xml=~/<City>(.*)</gm;
my $state = $1 if $xml=~/<Region>(.*)</gm;
my $country = $1 if $xml=~/<Country>(.*)</gm;
my $lon = $1 if $xml=~/<Longitude>(.*)</gm;
my $lat = $1 if $xml=~/<Latitude>(.*)</gm;
my $cert = $1 if $xml=~/<Certainty>(.*)</gm;
print PLUGIN "($cert % certainty) $_[0] => $city, $state, $country ($lat/lat, $lon/lon)";
}

note... this is quick and dirty. If your plugins depend on slow I/O, like this, you should probably fork so that you don't slow down the whole zone process (pretty cool, though, eh? :)

Crafty quest coders will surely find all kinds of fun ways to utilize embperl, enhancing eqemu and distinguishing their servers. The tie-in between the plugin and quest features makes for interesting possibilities, too. Quests can be turned on or off via global flags, set using #peval, or even completely overridden/implemented right there from the console! eg #peval sub qstdefault::EVENT_ATTACK { quest::say("Help! $name is trying to kill me!"); }. That is power, baby!

p.s. a script like this will go a long way towards converting existing quests. Change it to suit your needs.

#!/usr/bin/perl -w
#convert scripts from .qst to .pl
#usage: ./convert.pl [questdir]
use File::Find;
use strict;
sub convert
{
my $infile = $_;
(my $outfile = $infile)=~s/qst$/pl/;
print "Converting file: $infile -> $outfile\n";
if(!open IN, "$infile") {
warn $!;
return;
}
if(!open OUT, ">$outfile") {
warn $!;
return;
}
while(<IN>)
{
#remove stray backslashes
s|\\|\?|;
#change /yada yada yada/ comment lines to #yada yada
s|^/(.*)/\s*$|#($1)|;
#prefix each event block w/ "sub"
s/^(EVENT_)/sub $1/;
#change $1/$1- (etc) notation to instead match against $text
s/\s*\$\d-?\s*=~\s*\"(.*?)\"/\$text=~\/$1\/i/;
s/\$\d-?/\$text/;
#change commands to have a quest:: prefix
s/(say|emote|shout|spawn|echo|summonitem|castspell|d epop|cumflag|flagnpc|flagclient|exp|level|safemove |rain|snow|givecash|pvp|doanim|addskill|me)\(/quest::$1\(/;
print OUT $_;
}
close IN;
close OUT;
}
@ARGV = qw(.) unless @ARGV;
find( sub {/^.*\.qst\z/s && convert("$_")}, @ARGV);

[Mongrel]

I admit, I have no real clue about Perl, but wouldn't this allow quest writers to access other peoples hard drives? Perl seems powerful enough to me to make that possible.
(If it doesn't, just ignore me :oops:)

I can already see them coming ... "HERE!1! Use htis and HAil UR giuldmastre 2 GET KEWL STUFF!!!11!!" ... "Hail, Monk GM" ... bang, hard disks empty :shock:

[kathgar]

The perl is running on the /server/ not the client. It cannot access your harddrive any more than the EQLive servers could.

On merging... There are a couple of ways to go about it.. and i'm not exactly sure which is the best. If we do a CVS push all of the latest changes will be removed by your diff. You're best bet I believe is to do a diff against the current code base, and we'll look at it and merge it in. CVS Is one way at the moment though, so having you make changes and then trying to diff them versus CVS would run into all of these problems again.

[Trumpcard]

The CVS code is current as of this morning, so melding the changes there would be the best bet.

How many files will you need to make changes too? Optionly, we can lock those files for the time being until you are able to merge your changes in, you're best bet is to come onto IRC and PM me, kathgar, LE or merth and we can invite you into the coders channel for a discussion of your changes with any of the dev team present at the time.

[Mongrel]

>> The perl is running on the /server/

Err...I know, that's not the point.

What's the difference between your average Joe setting up a server and your other average Joe clicking "Yes" in that nice popup window with those naked women in the background?

What I meant to say was this:
If EQEmu ever gets popular (it probably is) enough and easy enough to set up (it is) so that anyone can make a server, stuff like that will appear.
Sure, the players aren't harmed by this, but the server ops are.

And don't tell me that people will be clever enough to read through all those quests they're downloading ... I mean, look at the forums!

[Trumpcard]

It would be a pretty bad solution if there wasnt a way to ensure everything was read only. embperl was designed for embedding perl into html pages, im sure it can handle that type of permissions checking.

I doubt anyone is going to be able to drop in 'rm -rf /*' .

[Mongrel]

As I said, I'm a total n00b concerning Perl. Until today I only knew that it existed somewhere, hehe.
Mind you, I'm not trying to stir up shit here, just trying to help. "Perl worms" were the first thing that came into my mind after reading the previous posts, so I just wanted to mention it.

What exactly is the difference between Perl and embedded Perl (embperl)?

I looked around on perl.com and /perl.apache.org/embperl for a bit, but couldn't find an exact explanation of the differences.
Perl has lots of low level IO functions, embperl doesn't seem to have these. Is embperl just an addition to normal Perl featuring all functions of its "father"?

If so, it might be a good idea to block their use in EQEmu.

[Eglin]

I admit, I have no real clue about Perl, but wouldn't this allow quest writers to access other peoples hard drives? Perl seems powerful enough to me to make that possible.
(If it doesn't, just ignore me :oops:)

I can already see them coming ... "HERE!1! Use htis and HAil UR giuldmastre 2 GET KEWL STUFF!!!11!!" ... "Hail, Monk GM" ... bang, hard disks empty :shock:

Yeah, security is always an issue. I imagine that embperl, should it catch on, will become slowly more refined, though. Since perl is a veteran of CGI, it already has powerful security features that could be applied (and currently aren't). If you want to enable Perl, but are highly concerned about security, I'd recommend enabling taint checking or disabling the #plugin interface, chroot'ing the process, and only using quests that you have written/converted yourself. Perl exposes a lot of power... that is precisely why I wanted to embed it.

[Eglin]

As I said, I'm a total n00b concerning Perl. Until today I only knew that it existed somewhere, hehe.
Mind you, I'm not trying to stir up shit here, just trying to help. "Perl worms" were the first thing that came into my mind after reading the previous posts, so I just wanted to mention it.
[quote]
Don't sweat it... it is a valid concern.

[quote]
What exactly is the difference between Perl and embedded Perl (embperl)?

The only real difference is that embedded perl is embedded into another program in some way. You should be able to do everything (and then some) that a standalone perl script could do.


I looked around on perl.com and /perl.apache.org/embperl for a bit, but couldn't find an exact explanation of the differences.

There isn't too much info on it. I am only aware of a handful of popular programs (2 irc clients, Apache, a few others) that have full-fledged native perl interfaces (as opposed to wsh interfaces).


Perl has lots of low level IO functions, embperl doesn't seem to have these. Is embperl just an addition to normal Perl featuring all functions of its "father"?

No, this implementation can do everything that standalone perl can do.

If so, it might be a good idea to block their use in EQEmu.
I doubt it. At present (assuming the #peval interface is not avaliable), there is no way for an end-user to directly inject their code into perl. To be subject to most errors, you would have to have bad quest code. If you are really worried about security, you would have to review each quest. Otherwise, you are implicitly trusting the authors (just like with eqemu itself). The bottom-line is that in most cases a user would have to have already compromised you system with a priviledge escalation attack in order to abuse perl. To the person who can already do this, perl's avaliability is only a slignt convenience.

[kai_shadowbane]

I'll try to not sound like an utter fool here, but here goes:

Wouldn't the parser have to be written in C or C++, since it would have to directly interface with the zone server interactively. (not like just loading something up once, b/c if you load all that in memory initially that would definately be a memory hog) Basically my thought came from that it would have to spawn and despawn mobs and such. And if when written in C or C++ (might be better languages initially, not too sure what all is out there that's cross-platform compat.), then be optimized in ASM.
Techincally, having each seperate quest in a text for is great, since it only has to access a small file for each quest, and then each text file identified with a mob is great as well.

Well, there was was babbling on in random thought, and I probably still came off like a fool, but if you have any comments to ease my curiosity, please feel free.

EDIT:
Side thought, what about writing it in a scripting language?

[Eglin]

I'll try to not sound like an utter fool here, but here goes:

Wouldn't the parser have to be written in C or C++, since it would have to directly interface with the zone server interactively. (not like just loading something up once, b/c if you load all that in memory initially that would definately be a memory hog) Basically my thought came from that it would have to spawn and despawn mobs and such. And if when written in C or C++ (might be better languages initially, not too sure what all is out there that's cross-platform compat.), then be optimized in ASM.
Techincally, having each seperate quest in a text for is great, since it only has to access a small file for each quest, and then each text file identified with a mob is great as well.

Well, there was was babbling on in random thought, and I probably still came off like a fool, but if you have any comments to ease my curiosity, please feel free.

EDIT:
Side thought, what about writing it in a scripting language?
I have no idea what you're going on about. When you read your own post, what kind of responses do you think you might get from it?