Rule for Adjusting Logging Debug Levels

[trevius]

In an effort to tweak my server as much as possible, I would like to start work on making a rule setting that allows admins to set what level of logging they want. I am hoping this might reduce disk utilization and possibly even CPU utilization a little, but it also reduces the large amount of logs that eventually get so big that you can't even view them easily.

So, what I propose is a rule system for all logging where there are multiple levels depending on the importance of the log type. In my example here, I give 5 different levels, but it could be expanded to any amount of levels.

Here is the rule that would need to be added:

ruletypes.h

RULE_INT ( World, LogDebugLevel, 5 ) //Changes the level of Debug Logs: 0 = disable Logs, 5 = Log everthing (Default = 5)

And, I am not 100% sure what the best way is to break up logging, but I am pretty sure it would work if I just edited every line in all files of the source that is for logging and changed them into a rule. Here are some examples of logging lines from the source (from random sections of code):

LogFile->write(EQEMuLog::Error, "Error in DeleteLauncher 1 query: %s", errbuf);

_log(WORLD__LS_ERR, "Unknown LSOpCode: 0x%04x size=%d",(int)pack->opcode,pack->size);

_log(WORLD__INIT, "Starting HTTP world service...");

And, the new rule would work like this:

if (RuleI(World, LogDebugLevel) > 0) {
_log(WORLD__INIT, "Starting HTTP world service...");
}

if (RuleI(World, LogDebugLevel) > 1) {
LogFile->write(EQEMuLog::Error, "Error in DeleteLauncher 1 query: %s", errbuf);
}

if (RuleI(World, LogDebugLevel) > 2) {
_log(WORLD__LS_ERR, "Unknown LSOpCode: 0x%04x size=%d",(int)pack->opcode,pack->size);
}

I think that would work, but my coding skill is still in the very early stages. Any suggestions or feedback are greatly appreciated.

Basically, I would define the different log levels to be something like this:

1 - World and Zone init events.
2 - Errors
3 - Looting and Questing
4 - Messages, Version Checks
5 - Unknown and/or Unhandled Opcodes

It should just go from most important (1) to least important (5). Setting the rule to 5 or higher would log everything. Setting it to 0 would disable all logs.

Also, I am not sure if this might help or not, but I did notice that there seems to be categories for logs already if this is still used:

common/logtypes.h

#ifndef LOG_CATEGORY
#define LOG_CATEGORY(name)
#endif
#ifndef LOG_TYPE
#define LOG_TYPE(cat, type, default_value)
#endif
#ifndef ENABLED
#define ENABLED true
#endif
#ifndef DISABLED
#define DISABLED false
#endif




LOG_CATEGORY( CHAT )
LOG_TYPE( CHAT, SAY, DISABLED )
LOG_TYPE( CHAT, EMOTE, DISABLED )
LOG_TYPE( CHAT, OOC, DISABLED )
LOG_TYPE( CHAT, GROUP, DISABLED )
LOG_TYPE( CHAT, GUILD, DISABLED )

LOG_CATEGORY( SPAWNS )
LOG_TYPE( SPAWNS, MAIN, DISABLED )
LOG_TYPE( SPAWNS, CONDITIONS, DISABLED )
LOG_TYPE( SPAWNS, LIMITS, DISABLED )

LOG_CATEGORY( AI )
LOG_TYPE( AI, ERROR, ENABLED )
LOG_TYPE( AI, WAYPOINTS, DISABLED )
LOG_TYPE( AI, BUFFS, DISABLED )
LOG_TYPE( AI, SPELLS, DISABLED )

LOG_CATEGORY( QUESTS )
LOG_TYPE( QUESTS, PATHING, DISABLED )

LOG_CATEGORY( SPELLS )
LOG_TYPE( SPELLS, LOAD, DISABLED )
LOG_TYPE( SPELLS, LOAD_ERR, DISABLED )
LOG_TYPE( SPELLS, CASTING_ERR, DISABLED )
LOG_TYPE( SPELLS, CASTING, DISABLED )
LOG_TYPE( SPELLS, EFFECT_VALUES, DISABLED )
LOG_TYPE( SPELLS, RESISTS, DISABLED )
LOG_TYPE( SPELLS, STACKING, DISABLED )
LOG_TYPE( SPELLS, BARDS, DISABLED )
LOG_TYPE( SPELLS, BUFFS, DISABLED )
LOG_TYPE( SPELLS, PROCS, DISABLED )
LOG_TYPE( SPELLS, MODIFIERS, DISABLED )

LOG_CATEGORY( FACTION )

LOG_CATEGORY( ZONE )
LOG_TYPE( ZONE, GROUND_SPAWNS, DISABLED )
LOG_TYPE( ZONE, INIT, ENABLED )
LOG_TYPE( ZONE, INIT_ERR, ENABLED )
LOG_TYPE( ZONE, WORLD, ENABLED )
LOG_TYPE( ZONE, WORLD_ERR, ENABLED )
LOG_TYPE( ZONE, WORLD_TRACE, DISABLED )

LOG_CATEGORY( TRADING )
LOG_TYPE( TRADING, ERROR, ENABLED )
LOG_TYPE( TRADING, CLIENT, DISABLED )
LOG_TYPE( TRADING, NPC, DISABLED )
LOG_TYPE( TRADING, HOLDER, DISABLED )

LOG_CATEGORY( INVENTORY )
LOG_TYPE( INVENTORY, ERROR, ENABLED )
LOG_TYPE( INVENTORY, SLOTS, ENABLED )

LOG_CATEGORY( TRADESKILLS )
LOG_TYPE( TRADESKILLS, IN, DISABLED )
LOG_TYPE( TRADESKILLS, OUT, DISABLED )
LOG_TYPE( TRADESKILLS, SQL, DISABLED )
LOG_TYPE( TRADESKILLS, TRACE, DISABLED )

LOG_CATEGORY( TRIBUTE )
LOG_TYPE( TRIBUTE, ERROR, DISABLED )
LOG_TYPE( TRIBUTE, IN, DISABLED )
LOG_TYPE( TRIBUTE, OUT, DISABLED )

LOG_CATEGORY( AA )
LOG_TYPE( AA, ERROR, ENABLED )
LOG_TYPE( AA, MESSAGE, DISABLED )
LOG_TYPE( AA, IN, DISABLED )
LOG_TYPE( AA, OUT, DISABLED )


LOG_CATEGORY( DOORS )
LOG_TYPE( DOORS, INFO, DISABLED )

LOG_CATEGORY( PETS )
LOG_TYPE( PETS, AGGRO, DISABLED )

LOG_CATEGORY( COMBAT )
LOG_TYPE( COMBAT, ATTACKS, DISABLED )
LOG_TYPE( COMBAT, TOHIT, DISABLED )
LOG_TYPE( COMBAT, MISSES, DISABLED )
LOG_TYPE( COMBAT, DAMAGE, DISABLED )
LOG_TYPE( COMBAT, HITS, DISABLED )
LOG_TYPE( COMBAT, RANGED, DISABLED )
LOG_TYPE( COMBAT, SPECIAL_ATTACKS, DISABLED )
LOG_TYPE( COMBAT, PROCS, DISABLED )

LOG_CATEGORY( GUILDS )
LOG_TYPE( GUILDS, ERROR, ENABLED )
LOG_TYPE( GUILDS, ACTIONS, ENABLED )
LOG_TYPE( GUILDS, DB, DISABLED )
LOG_TYPE( GUILDS, PERMISSIONS, DISABLED )
LOG_TYPE( GUILDS, REFRESH, DISABLED )	//inter-zone refresh comm
LOG_TYPE( GUILDS, IN_PACKETS, DISABLED )
LOG_TYPE( GUILDS, OUT_PACKETS, DISABLED )
LOG_TYPE( GUILDS, IN_PACKET_TRACE, DISABLED )	//hex dumps
LOG_TYPE( GUILDS, OUT_PACKET_TRACE, DISABLED )	//hex dumps

LOG_CATEGORY( CLIENT )
LOG_TYPE( CLIENT, ERROR, ENABLED )
LOG_TYPE( CLIENT, DUELING, DISABLED )
LOG_TYPE( CLIENT, SPELLS, DISABLED )
LOG_TYPE( CLIENT, NET_ERR, ENABLED )
LOG_TYPE( CLIENT, NET_IN_TRACE, DISABLED )

LOG_CATEGORY( SKILLS )
LOG_TYPE( SKILLS, GAIN,  DISABLED )

LOG_CATEGORY( RULES )
LOG_TYPE( RULES, ERROR,  ENABLED )
LOG_TYPE( RULES, CHANGE, ENABLED )

LOG_CATEGORY( NET )
LOG_TYPE( NET, WORLD, ENABLED )
LOG_TYPE( NET, OPCODES, ENABLED )
LOG_TYPE( NET, IDENTIFY, ENABLED )
LOG_TYPE( NET, IDENT_TRACE, ENABLED )
LOG_TYPE( NET, STRUCTS, ENABLED )
LOG_TYPE( NET, STRUCT_HEX, ENABLED )
LOG_TYPE( NET, ERROR, ENABLED )
LOG_TYPE( NET, DEBUG, DISABLED )
LOG_TYPE( NET, APP_TRACE, DISABLED )
LOG_TYPE( NET, APP_CREATE, DISABLED )
LOG_TYPE( NET, APP_CREATE_HEX, DISABLED )
LOG_TYPE( NET, NET_TRACE, DISABLED )
LOG_TYPE( NET, NET_COMBINE, DISABLED )
LOG_TYPE( NET, FRAGMENT, DISABLED )
LOG_TYPE( NET, FRAGMENT_HEX, DISABLED )
LOG_TYPE( NET, NET_CREATE, DISABLED )
LOG_TYPE( NET, NET_CREATE_HEX, DISABLED )
LOG_TYPE( NET, NET_ACKS, DISABLED )
LOG_TYPE( NET, RATES, DISABLED )

LOG_CATEGORY( DATABASE )

LOG_CATEGORY( COMMON )
LOG_TYPE( COMMON, ERROR, ENABLED )
LOG_TYPE( COMMON, THREADS, ENABLED )

LOG_CATEGORY( LAUNCHER )
LOG_TYPE( LAUNCHER, ERROR, ENABLED )
LOG_TYPE( LAUNCHER, INIT, ENABLED )
LOG_TYPE( LAUNCHER, STATUS, ENABLED )
LOG_TYPE( LAUNCHER, NET, ENABLED )
LOG_TYPE( LAUNCHER, WORLD, ENABLED )

LOG_CATEGORY( WORLD )
LOG_TYPE( WORLD, CONFIG, ENABLED )
LOG_TYPE( WORLD, INIT, ENABLED )
LOG_TYPE( WORLD, INIT_ERR, ENABLED )
LOG_TYPE( WORLD, CLIENT, ENABLED )
LOG_TYPE( WORLD, ZONE, ENABLED )
LOG_TYPE( WORLD, LS, ENABLED )
LOG_TYPE( WORLD, CLIENT_ERR, ENABLED )
LOG_TYPE( WORLD, ZONE_ERR, ENABLED )
LOG_TYPE( WORLD, LS_ERR, ENABLED )
LOG_TYPE( WORLD, SHUTDOWN, ENABLED )
LOG_TYPE( WORLD, CLIENTLIST, DISABLED )
LOG_TYPE( WORLD, CLIENTLIST_ERR, ENABLED )
LOG_TYPE( WORLD, ZONELIST, ENABLED )
LOG_TYPE( WORLD, ZONELIST_ERR, ENABLED )
LOG_TYPE( WORLD, CLIENT_TRACE, DISABLED )
LOG_TYPE( WORLD, ZONE_TRACE, DISABLED )
LOG_TYPE( WORLD, LS_TRACE, DISABLED )
LOG_TYPE( WORLD, CONSOLE, ENABLED )
LOG_TYPE( WORLD, HTTP, ENABLED )
LOG_TYPE( WORLD, HTTP_ERR, ENABLED )
LOG_TYPE( WORLD, PERL, ENABLED )
LOG_TYPE( WORLD, PERL_ERR, ENABLED )
LOG_TYPE( WORLD, EQW, ENABLED )
LOG_TYPE( WORLD, LAUNCH, ENABLED )
LOG_TYPE( WORLD, LAUNCH_ERR, ENABLED )
LOG_TYPE( WORLD, LAUNCH_TRACE, ENABLED )

#undef LOG_TYPE
#undef LOG_CATEGORY

So, maybe the rules could be set in this file and maybe it would avoid having to adjust every single log line in every file manually. Though, I am not sure if this is even used anymore.

I wouldn't mind going through and editing the source to have log levels set for every log line, but it certainly isn't something I would want to do again every time I updated to a new version of the emulator.

[AndMetal]

There's already a logging system in place, but it's somewhat complex. It appears the log system was reworked at some point.
From the Wiki:

Log Systems
eqemu is a mess of various logging mechanisms... here is a quick explanation:

There is an "overarching" log system, which logs into the eqemu_* files which is used for about 90% of the logging, but there is no runtime flexibility in this system

the EQDEBUG define is an older mechanism to tune the log messages coming out, but more or less, you want to leave it at its default of 5 (in makefiles/projects)... lower and you miss interesting stuff, higher and you'll log yourself to death.

the 'loglevel' variable is even older, not really used for much anymore, and should prolly be removed completely.

The logsys stuff linked above is the "new" mechanism of logging info, which allows runtime configuration of logging information and is documented at the link above. This system sits on top of the "overarching" log system, putting all of its output through the "debug" facility, and hence into eqemu_*_debug.log

finally, there is a lot of bullshit logging done with printf/cout in the code, which does not go into any of the eqemu_* files, if you use the launcher, it pipes all output (logger or direct) into the zone_* files.

I do think there might be some opportunities for the logging system which are definitely worth looking into, especially trying to narrow it down to 1 useful system. In addition, I think it would be a lot easier than reworking the entire logging system again

[trevius]

Ya, I agree that it seems like there may still be some old stuff left in there that shouldn't be. I guess that would be part of the clean up.

I am aware that there used to be a logging system where you could adjust it from the variables table (loglevel). But, from all I have heard, the only way to currently adjust what is and is not logged is to manually go into the source and comment out the log lines 1 by 1 and then recompile the source. IMO, that is a bit extreme. I also think that logging as it currently is has so much output that I very rarely even look at it. I would much prefer to just completely disable it and only turn it on if I find a problem.

To simplify the rule, it could even be changed to Boolean and just enable or disable logs completely. If the #rule reload command (or whatever the command is) works now, you could possibly even enable and disable logging on the fly without even restarting the server. IMO, that would be awesome and actually make logs useful.

[cavedude]

log.ini is how the current system is configured and works fine, I use it on TGC. There is no need to change any code or recompile. http://www.eqemulator.net/wiki/wikka...a=ServerLogSys

The categories and types in common/logtypes.h are current and all seem to work, at least I have yet to find any one which has not worked yet.

Edit above: I do remember one of the spell types causing a zone crash because for whatever reason it was unable to write out to the log. Wildcard confirmed this issue at the time, but we just disabled that particular type (can't remember which it was) and moved on.

[rojadruid]

Originally Posted by cavedude

log.ini is how the current system is configured and works fine, I use it on TGC. There is no need to change any code or recompile. http://www.eqemulator.net/wiki/wikka...a=ServerLogSys

The categories and types in common/logtypes.h are current and all seem to work, at least I have yet to find any one which has not worked yet.

Edit above: I do remember one of the spell types causing a zone crash because for whatever reason it was unable to write out to the log. Wildcard confirmed this issue at the time, but we just disabled that particular type (can't remember which it was) and moved on.

log.ini works as intended and does a fine job of showing everything thats needed. I use it as well on my server.

[trevius]

Thanks guys lol. That saves me alot of work then. Last time I was asking about logged (maybe 6 months ago), I was told by multiple people who have been members for a long time that the current logging system could only be adjusted by commenting out lines in the source.

If I had known that the log.ini actually worked properly, I would have been using it long ago lol.

Thanks