Long story short: a few weeks ago I started making a server, but I had a problem: I didn't know any Perl, and the Perl wrapping code and some other things were incomprehensible to me. I had recently gotten a good handle on Lua, though, so I decided to embed my own Lua-based scripting system. At the same time, I couldn't get some of the SQL editors working out of the box, so I decided to just use Lua to database content (items, spells, npcs, spawn points, path grids, etc).
My own server was very custom and I didn't put any care into maintaining compatibility with standard EQEmu, but I've now gone back and started making a clean version, mostly separated by preprocessor definitions, in case anyone else might want a look at it. This post is just to gauge interest to see if I should keep working on this and make it available.
Forewarning: as I said, I don't know much of anything about your Perl system in practice, and I'm not sure whether you're still stuck with static loading for items and spells, so maybe a lot of the things I think are improvements will seem quaint. Also, this is going to be a
long post full of example Lua code. Oh, and one more thing: probably none of this will be of much interest to established servers since the cost of switching would be high. But some tinkerers and start-ups might be interested -- if that's you, read on! Oh, and to make it clear, all the following stuff isn't just pie-in-the-sky theory crafting; my server runs using all these systems in place of the Perl/MySQL versions.
Standard spiel: Lua is a flexible, easy-to-learn language with good
documentation. It's often used in video games and it has the benefit of being designed to embed in C, making it relatively easy to work with from inside C/C++ without the need for complicated third party wrappers. Deep in the manual it's also described (at least historically) as a "data-description language," which we're taking advantage of in using it for databasing.
As far as being a scripting language, I'm assuming it probably wouldn't be any better than your Perl system, and would probably take some time to catch up with Perl in terms of functionality (I'm basically just writing functions as I find a need for them on my own server -- I'll need to start working on documentation soon, too). The syntax is certainly different, and some of the standards will be different too (arguments to functions are explicit rather than implicit, and there's no strict distinction between clients and npcs, instead always being handled as mobs).
In any case, I intend to keep the scripting system proper and the databases under different preprocessor defs, just in case anyone comes out interested in the databasing but less interested in switching from Perl for scripts. Quick rundown of those:
EMBLUA - compile the Lua-handler class. Required to use anything below.
LUA_ITEMDB - use Lua for items.
LUA_SPELLDB - use Lua for spells.
LUA_SPAWNS - use Lua for spawn points and path grids, and effectively for NPCs (as we'll see below).
LUA_SCRIPTS - use Lua in place of Perl for script EVENT calls. The functions will still be loaded if you don't use this, in case you want to use them for LUA_SPAWNS.
To begin looking at Lua databasing, let's start with the most basic difference: rather than sitting in an SQL table, data will start off sitting in Lua scripts. This allows us to make use of the organization of the /quests/ folder; by an large, developers will not need to think about global id numbers for anything except spells and occasionally items. IDs for spawn points, path grids, and items (technically) will always start at 1 within the zone in which they are being loaded.
Also, while we lose SQL's arbitrary searching abilities, we generally won't have to go far to find what we're looking for. Spawn points for gfaydark will always be located in the "spawn_list.lua" file in the /quests/gfaydark/ folder. Items will still have global ID numbers, but they will be standardized so that we can easily work backward from their global ID to see which zone they are defined in.
SPAWNPOINTS
Now, to get into some real, practical examples, let's start with a look at spawnpoints. Spawnpoint scripts, as noted above, are called "spawn_list.lua" and located in the folder for the zone they are spawning things in. When a zone loads, it looks for this file and compiles it into Lua's memory to refer to from then on. A simple spawnpoint script with a single entry might look like this (some basic familiarity with Lua's syntax will help from here on):
Code:
--spawn list for gfaydark
spawn_list = {}
spawn_list[1] = {
{name="a guard",race=4,gender=3,size=5,texture=3,aggro_range=25,loc={5,120,-2,220}}, 60
}
To parse it out: first we create a table called "spawn_list" (it must be called this) and then we put an entry into it. The entry is indexed by an ID number (starting at 1, as here) and consists of two values: a table full of NPC data, and a respawn time in seconds. To help visualize it:
Code:
spawn_list[id] = {
npc_table, respawn_time
}
When it is time to respawn, the code looks for the first value, the NPC data table. If it finds a table there, it ships it off to a spawn function, using any values it finds (for example, if name="X" in the data table, the NPC's name is set to X) and providing sensible defaults for anything that was left out.
Similarly, when the NPC associated with this spawnpoint depops, the code looks for the second value, the respawn time. If it finds an integer there, it uses that value as the respawn time in seconds.
That's all well and good, but where are all the added functionalities Spawn2 gives us? This is where Lua's flexibility comes in. Though we are free to use constants for NPC data and respawn time as we did above, we can also provide functions in their place. If the code finds a function instead of constant data, it will call that function and use its return value as the data it was looking for.
The spawn point and path grid databasing systems coexist with the scripting system, giving us access to its functions -- and meaning that we can perform arbitrary checks inside the functions that will use for our spawnpoints. As a silly example, let's make it so our NPC will only spawn if there is a client with the letter "v" in their name somewhere in the zone at the moment of spawn:
Code:
--spawn list for gfaydark
spawn_list = {}
spawn_list[1] = {
function()
local clients = GetClientList()
for key,cli in pairs(clients) do
if string.match( string.lower( cli:GetName() ), "v") then
return {name="a guard",race=4,gender=3,size=5,texture=3,aggro_range=25,loc={5,120,-2,220}}
end
end
end,
60
}
If no NPC data table is returned, the code will consider the spawnpoint to have depoped again, restarting the respawn timer.
The respawn timer can have similarly arbitrary functions, though in its case it must return a number value or the spawnpoint will be completely disabled. Here's a quick example to make our NPC respawn faster when there are more clients in the zone:
Code:
--spawn list for gfaydark
spawn_list = {}
spawn_list[1] = {
{name="a guard",race=4,gender=3,size=5,texture=3,aggro_range=25,loc={5,120,-2,220}},
function()
local clients = GetClientList()
return 60 - ((#clients < 30) and #clients or 30)
end
}
If there are under 30 clients in the zone, the respawn timer will be 1 second quicker for each client; otherwise it will cap at 30 seconds quicker. (The (x and y or z) is equivalent to C's ternary operator, x ? y : z)
These examples are a bit silly, but I think it shouldn't be too hard to see how these two functions (well, basically just the first) effectively replace spawn groups, spawn conditions, spawn disabling, spawn limits, and so on while offering a huge range of potential uses besides -- the developer is given full control. Also note that spawn points are not explicitly associated with particular locs; when it comes down to it, a spawnpoint is really just a process -- the only time the loc matters is at the very moment the NPC is spawning. As such the loc is instead part of the NPC data, and we are free to change it, to give our spawnpoint variable locations. Or more likely, to reuse NPC data at various spawnpoints, and just change the loc each time. For example:
Code:
--spawn list for gfaydark
spawn_list = {}
spawn_types = {
guard = {name="a guard",race=4,gender=3,size=5,texture=3,aggro_range=25},
}
spawn_list[1] = {
function()
local guard = spawn_types.guard
guard.loc = {5,120,-2,220}
return guard
end,
60
}
spawn_list[2] = {
function()
local guard = spawn_types.guard
guard.loc = {20,130,-2,250}
return guard
end,
60
}
I could go on all day about all the permutations possible here, but for one last example, let's say our spawnpoint can spawn either a named NPC or a trash NPC. We want there to be a 20% chance of spawning the named NPC on any given spawn, but we also want it to be guaranteed to spawn within 5 consecutive kills. For this we can use Lua's closure logic:
Code:
--spawn list for gfaydark
spawn_list = {}
function NamedSpawn()
local spawn_count = 0
return function()
spawn_count = spawn_count + 1
if spawn_count >= 5 or math.random(5) == 1 then
spawn_count = 0
return {name="Named NPC", ...}
end
return {name="a trash NPC", ...}
end
end
named_spawnpoint = NamedSpawn()
spawn_list[1] = {named_spawnpoint, 600}
The outer function (NamedSpawn) acts like a constructor for the inner function. The local variable in the outer function (spawn_count) keeps its state between calls to the inner function, letting us conveniently keep track of data from spawn to spawn. If spawn_count hits 5, the named spawn is guaranteed. Whenever the named spawn does happen, spawn_count is reset to 0. The value of spawn_count will be lost if the zone goes down, but other than that Lua will quietly handle it on its own.
Of course, a constructor that only works for one spawnpoint is not very useful, so let's go ahead and generalize it:
Code:
--spawn list for gfaydark
spawn_list = {}
function NamedSpawn(named,trash,chance,max)
local count = 0
return function()
count = count + 1
if count >= max or math.random(100) <= chance then
count = 0
return named
end
return trash
end
end
named_spawns = {}
named_spawns.example = NamedSpawn({name="Named NPC", ...}, {name="a trash NPC", ...}, 20, 5)
named_spawns.evilington = NamedSpawn({name="Professor Evilington", ...}, {name="a stupid wizard", ...}, 10, 10)
spawn_list[1] = {named_spawns.example, 600}
spawn_list[2] = {named_spawns.evilington, 1200}
Now all of the relevant information is kept in state variables. And again, NamedSpawn() acts as a constructor, so the state variables for "named_spawns.example" are kept separately from those of "named_spawns.evilington".
Even after all of this, our spawnpoints still need to be recorded so that their spawn times will persist between zone boots, so we aren't escaping SQL entirely. At the very least, however, we can replace six (?) interconnected tables (spawn2, respawn_times, spawngroups + spawnentry, spawn_conditions, spawn_condition_values, any more?) with one nice, compact table:
Code:
CREATE TABLE IF NOT EXISTS lua_spawn (
id INT UNSIGNED NOT NULL DEFAULT 0,
zone VARCHAR(32) NOT NULL,
instance INT UNSIGNED NOT NULL DEFAULT 0,
spawntime INT UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY (id, zone, instance)
);
Not too shabby.
It's worth making this explicit: by default, there is no NPC database. NPCTypes are constructed anew every time an NPC is spawned, whether from a spawnpoint or a mob script. In the above examples, NPC data has always been given as constants, but we are free to keep a running list in Lua's memory or to load a list from a secondary file or whatnot. But assuming you aren't doing anything fancy, the spawn_list.lua file also acts as the zone's NPC database.
One last note: in order to update or add spawn points, after putting the updated spawn_list.lua file on our server, we will need to enter the zone in question and use the #reloadspawns command in order to A) inform the zone that the spawn data in Lua's memory is out of date and B) create SQL entries for any new spawn points (this part could probably be automated, but doing it on command seemed safer/easier/more efficient).
PATH GRIDS
Anyway, enough about spawn points! Let's take a quick look at path grids. Path grids are loaded from "quests/<zone shortname>/path_list.lua". A lot of this is going to look familiar.
A pathing grid consists of two values: a table full of locs, and a function. In this case, the function is not optional. Whenever an NPC on a grid reaches one of the nodes, it calls this function and expects two results: the index of the next loc, and the time to wait before proceeding to it. A quick example of a circular path, in maximum-readability form:
Code:
--path list for gfaydark
path_list = {}
path_list[1] = {
{
[1] = {10,-20,-2},
[2] = {20,-20,-2},
[3] = {20,-30,-2},
[4] = {10,-30,-2}
},
function(index,grid_id,npc)
if index == 4 then
return 1, 2000
end
return index+1, 1000
end
}
Fairly simple: our grid has four points. When an NPC on that grid reaches a point, it calls the function, passing the index of the loc we just reached, the id of the grid it's on, and the npc itself. If we are at the final loc in the grid (index 4), we tell it to go back to the first loc after waiting 2000 milliseconds. Otherwise, we queue up the next loc in the list and proceed to it after 1000 milliseconds. (For reference, we would put an NPC on this path by putting "path=1" in its NPC data in spawn_list.lua; it can be varied by spawnpoint just as loc is in one of the above examples.)
The benefits here should be fairly obvious. We no longer need to remember magic numbers for wander type, and we are free to customize pathing behavior without needing to go into specific NPC scripts. For example, let's say we have a 20 point circular grid, but we want points 16-20 to be off limits unless "Named NPC" is dead:
Code:
--path list for gfaydark
path_list = {}
path_list[1] = {
{
[1] = {10,-20,-2},
[2] = {10,-50,-2},
...
[20] = {100,200,-2}
},
function(index,grid_id,npc)
if index == 15 then
local npcs = GetNPCList()
for key,NPC in pairs(npcs) do
if NPC:GetName() == "Named NPC" then
return 1, 1000
end
end
elseif index == 20 then
return 1, 1000
end
return index+1, 1000
end
}
The additional arguments (grid_id and npc) can be used to make generalized functions (we don't want to have to rewrite code for the basic circular path every time, naturally) or to add NPC behaviors upon reaching a point, much like EVENT_WAYPOINT_ARRIVE, but independent of which NPC is on the grid at the time.
Note that the list of grid locs need not be constant. If we assign the table full of locs to a variable, anything that makes calls to Lua -- like spawn points, or mob scripts -- can add, remove, and/or replace locs in the grid in real time. In fact, with an appropriate algorithm we could have the grid's own function alter its nodes, for example creating a geometric pattern without ever having more than 1 node in the list at any one time, like so:
Code:
--path list for gfaydark
path_list = {}
active_geo_path = {[1] = {0,0,0,0}}
path_list[1] = {active_geo_path,
function(index,grid_id,npc)
local cur_loc = path_list[1][1][1] --i.e. path_list[grid][loc_table][first_loc_entry]
local next_loc = {}
--put code to generate next loc here based on heading etc
path_list[grid_id][1][1] = next_loc
return 1, 1000
end
}
As with spawnpoints, there aren't a whole lot of limits to what you could do within the path function, but I don't want to bog this post down with overly complicated examples. In any case, once more we get to drop a couple of SQL tables (grid, grid_entry), so that's a plus.
ITEMS
I don't know if you guys are still held captive by Sharemem/static item loading, but I was on my old server, and it sucked. In this Lua system, items are instead loaded into Lua's dynamic memory. All GetItem() requests are thus simply rerouted to Lua's dynamic item list and const-casted on the way to C++. Cross-zone consistency is maintained in a fairly hacky way: when the #reloaditems command is used, the user sends a special private message to themself through the world server with an impossible channel id, an admin check, and finally the special trigger string (this means an extra check on every /tell, but most will stop at the channel id check, so it's minimal extra processing. Someone tell me if there's a better server packet to co-opt for this!). Every running zone will see this and tell their instance of the Lua interpreter to reload any loaded items. Any zones that aren't currently running will simply load the up-to-date items when they boot. This has the price that clients won't see changes to items they already have until they zone, but that's fairly minor (and could probably be fixed, if you don't mind flooding your players with item packets occasionally). Furthermore, the item database gets its own Lua interpreter, making it impossible to directly access the loaded items from within NPC scripts or what have you, for safety.
Anyway, that's the theory, now for the nitty-gritty:
Most items, like spawn points, are associated with a particular zone, and are loaded from "item_list.lua" in a zone's folder. A file with a single, simple item might look like this:
Code:
require "item"
item_zone = "gfaydark"
item_list[1] = {
name="Ragged Fur Jerkin",hp=10,mana=10,endur=10,ac=5,slot=item.slot.chest,
icon=662,itemtype=item.type.worn,material=1
}
require "item" just loads a module (item.lua) with predefined values, like item.slot.chest and item.type.worn. Otherwise, items data is handled a lot like NPC data. Provided values are sent off to an item-creating function at the appropriate time, and any values not provided are given sensible defaults. Some shortcomings are apparent: you'll need to use a third-party tool to find icon and color values and whatnot.
But unlike spawnpoints and path grids, there is a fair bit of work being done under the covers. Notice that we don't declare item_list (no "item_list = {}") and we have to set a variable with the shortname of the zone. item_list is already defined, with special behaviors, in master_item.lua which will be located in /quests/items/. You won't need to worry about exactly what it does, but it's good to know it's there.
Also, notice that the item ids for the zone start at 1. Under this system, items effectively have two id values: intra-zone ids, and global ids. The system gives every zone 1000 spaces for items by default (which I think is fairly generous) and the global id for an item is simply set to zoneid*1000 + intra-zone_id. This is mostly to simplify things for developers (no need to check if an item range is completely free); if you use the Lua scripting system, functions like AddItem() will take intra-zone ids by default, with an added option to use global ids, as well. And, again, we can easily find out what zone an item is from by doing a quick floor((global_id-1)/1000) = zone_id. Of course, not all items are associated with particular zones; for them, there will be overflow spaces (at least 100,000 of them) as well as spots where there are gaps in zone ids; these items will be defined in "global_items.lua" in the /quests/items/ folder by default, though they can be separated into further files and simply loaded by global_items.lua.
These restrictions might seem off-putting, but they come with another benefit. Since items are kept in relatively small files, and since the file an item comes from can easily be deduced by its global id, we can implement a dynamic loading regiment.
Say your server has 50,000 items. It seems silly to have all those items loaded in every running zone, especially if only, say, 1000 ever appear in a particular zone between boots.
By default, when a zone boots up, it will only load the items in its own item_list.lua. After that, item loading is dynamic. Whenever a GetItem() call happens, Lua will check to see if the item being asked for has already been loaded. If not, it'll check the requested id number and determine if it comes from a particular zone, or from the global item file. After that it will load the correct file into memory (as Lua tables full of item data) and check to see if the intra-zone id appears in that file's item_list. If so, it'll pluck out that one item data table and load it (as an Item_Struct pointer) and then return it to C++.
So basically, items will be loaded as needed, when someone enters a zone with foreign items, or when an item link is sent from another zone, or when someone summons something.
It may add a little loading time, but your RAM will thank you. Furthermore, this takes some of the load off of the worldwide #reloaditems command -- zones will only reload items that they have already loaded up to that point. Anything not loaded will be up-to-date when and if they are dynamically loaded, naturally.
If there is one other downside to all this, it's that a particularly bad typo in an item-defining file can potentially wipe out all the items for a zone. In case of this eventuality, Lua will create a placeholder this-is-clearly-an-error item in place of any item in can't find to load. If the item is in an equip slot, Lua will also set the placeholder item's slot to the value matching the slot the item is in, so hopefully inventory will be kept fully consistent (still need to test that bit, though).
As a last little note, since items are defined in Lua scripts, if you had some
really consistent items you could easily have Lua generate their item data algorithmically, but that is a pretty tiny benefit.
SPELLS
Spells are more or less like items, although I haven't bothered to make a dynamic loading system for them yet (it would have to load on scribe/mem/click to be worth bothering).
I think I'm running out of steam here, there isn't a whole lot to say about spells specifically. They're loaded from "spell_list.lua" in the /quests/spells/ folder. You will still need to generate a spells_us.txt file (I have a script to do this) to give to players whenever you make any changes the client needs to know about, like name, icon, spell gem color, cast time, grey out/recast time, and whether the spell exists in the first place. But other than that, changes can be made dynamically using the #reloadspells function -- things like effects an effect values, reagents, can be changed without needing to get a new spellfile out right away (the reloading is globalized in the same manner as items). And even when you do put out a new spellfile, it'll be enough that the players download it, log out, and then log back in; you won't need to actually bring your server down to add new spells and whatnot.
The spell template currently looks like this:
Code:
spell_list[3] = {
name = "",
teleport_zone = "",
you_cast = "",
other_cast = "",
cast_on_you = "",
cast_on_other = "",
fade_msg = "",
range = 0,
aeradius = 0,
knockback = 0,
knockup = 0,
cast_time = 5000,
recover_time = 1000,
recast_time = 0,
duration = 0,
duration_formula = 0,
aeduration = 0,
mana = 0,
effects = {
{0,0,0,0,100},
{0,0,0,0,100},
{0,0,0,0,100},
{0,0,0,0,100},
{0,0,0,0,100},
{0,0,0,0,100},
{0,0,0,0,100},
{0,0,0,0,100},
{0,0,0,0,100},
{0,0,0,0,100},
{0,0,0,0,100},
{0,0,0,0,100}
},
icon = 0,
memicon = 0,
components = {
{-1,1,-1},
{-1,1,-1},
{-1,1,-1},
{-1,1,-1}
},
light_type = 0,
beneficial = 0,
activated = 0,
resist_type = 0,
target_type = ST.target,
basediff = 0,
skill = skill.evocation,
zone_type = -1,
environment_type = 0,
time_of_day = 0,
level = 1,
cast_anim = 44,
target_anim = 13,
travel_type = 0,
affect_index = -1,
no_sit = 0,
newicon = 161,
spell_anim = 0,
uninterruptable = 0,
resist_diff = 0,
dot_stacking_exempt = 0,
deletable = 0,
recourse = 0,
short_buff_box = -1,
desc = 0,
typedesc = 0,
effectdesc = 0,
bonus_hate = 0,
endur_cost = 0,
endur_timer = 0,
endur_upkeep = 0,
is_disc_buff = 0,
hate_added = 0,
num_hits = 0,
pvp_base = 0,
pvp_calc = 100,
pvp_cap = 0,
category = -99,
can_mgb = 0,
dispel_flag = -1,
min_resist = 0,
max_resist = 0,
viral_targets = 0,
viral_timer = 0,
nimbus_effect = 0,
directional_start = 0,
directional_end = 0,
spell_group = 0,
restriction = 0,
allow_rest = true,
max_targets = 0
}
As with NPCs and items, any values not provided are given sensible defaults. You'll also want to start id values at 3 rather than 1, just because. There is a module to provide certain familiar pre-defined values, like ST.target, skill.evocation, and SE.current_hp (may change the capitalization to be more like spdat.h #defines in the public version)
Annnnd that's it, I guess. I also use Lua in place of SQL for pets, titles, and saylinks (as an EVENT and a standardized function rather than a database file). I don't use factions on my server, but a system similar to items and spells could surely be made for them, and I'll probably work on that when the time comes. I also just use EVENT_SPAWN for loot drops to get away from loot table SQL, but that is surely nothing new.
Anyway, any interest/questions/pointing-and-laughing at how backwards my system is, is welcome!
Lua Scripting+Content Databasing system
Linear Mode
