Zone.exe crash/loop

[John Adams]

I've noticed on the Windows platform, zone.exe will go into a STOP / START loop endlessly if you modify a faction or loot table entry without restarting your entire server (not sure if world.exe is effected).

Scenario is; You add a new NPC and want to add a new faction and loot for that NPC, or even modify this info for an existing NPC. When you add the faction to the table `npc_factions`, everything looks nice until you zone out (dynamic zones) and zone back in. At that point, the instance crashes and loops as stated above. Looking into the logs, I finally caught this:

[08.30. - 12:05:19] Starting Log: logs/eqemu_debug_zone_0204.log
[08.30. - 12:05:19] [ZONE__INIT] Loading server configuration..
[08.30. - 12:05:19] [ZONE__INIT] Log settings loaded from log.ini
[08.30. - 12:05:19] [ZONE__INIT] Connecting to MySQL...
[08.30. - 12:05:19] [ZONE__INIT] CURRENT_ZONE_VERSION: EQEMu 0.7.0
[08.30. - 12:05:19] [ZONE__INIT] Log settings loaded from ./log.ini
[08.30. - 12:05:19] [ZONE__INIT] Mapping Incoming Opcodes
[08.30. - 12:05:19] [ZONE__INIT] Loading Variables
[08.30. - 12:05:19] [ZONE__INIT] Loading zone names
[08.30. - 12:05:19] [ZONE__INIT] Loading items
[08.30. - 12:05:20] [ZONE__INIT] Loading npc faction lists
[08.30. - 12:05:20] [ZONE__INIT_ERR] Loading npcs faction lists FAILED!
---------------------------------------------

Same thing with Loot. When we were trying to change some loot tables around and create new ones, anytime a new loot table was assigned to an existing NPC, the next time someone entered the zone it would crash:

[08.31. - 13:44:16] Starting Log: logs/eqemu_debug_zone_3760.log
[08.31. - 13:44:16] [ZONE__INIT] Loading server configuration..
[08.31. - 13:44:16] [ZONE__INIT] Log settings loaded from log.ini
[08.31. - 13:44:16] [ZONE__INIT] Connecting to MySQL...
[08.31. - 13:44:16] [ZONE__INIT] CURRENT_ZONE_VERSION: EQEMu 0.7.0
[08.31. - 13:44:16] [ZONE__INIT] Log settings loaded from ./log.ini
[08.31. - 13:44:16] [ZONE__INIT] Mapping Incoming Opcodes
[08.31. - 13:44:16] [ZONE__INIT] Loading Variables
[08.31. - 13:44:16] [ZONE__INIT] Loading zone names
[08.31. - 13:44:16] [ZONE__INIT] Loading items
[08.31. - 13:44:17] [ZONE__INIT] Loading npc faction lists
[08.31. - 13:44:17] [ZONE__INIT] Loading loot tables
[08.31. - 13:44:17] [ZONE__INIT_ERR] Loading loot FAILED!
---------------------------------------------

After a few days of not noticing, you might catch something like this in your world debug log:

[08.31. - 22:34:50] [WORLD__LAUNCH] zones: dynamic_20 reported state STOPPED (552 starts)
[08.31. - 22:34:54] [WORLD__LAUNCH] zones: dynamic_16 reported state STARTED (3070 starts)
[08.31. - 22:34:56] [WORLD__LAUNCH] zones: dynamic_16 reported state STOPPED (3070 starts)
[08.31. - 22:35:02] [WORLD__LAUNCH] zones: dynamic_20 reported state STARTED (553 starts)
[08.31. - 22:35:04] [WORLD__LAUNCH] zones: dynamic_20 reported state STOPPED (553 starts)
[08.31. - 22:35:08] [WORLD__LAUNCH] zones: dynamic_16 reported state STARTED (3071 starts)
[08.31. - 22:35:10] [WORLD__LAUNCH] zones: dynamic_16 reported state STOPPED (3071 starts)
[08.31. - 22:35:16] [WORLD__LAUNCH] zones: dynamic_20 reported state STARTED (554 starts)
[08.31. - 22:35:18] [WORLD__LAUNCH] zones: dynamic_20 reported state STOPPED (554 starts)
[08.31. - 22:35:22] [WORLD__LAUNCH] zones: dynamic_16 reported state STARTED (3072 starts)
[08.31. - 22:35:24] [WORLD__LAUNCH] zones: dynamic_16 reported state STOPPED (3072 starts)
[08.31. - 22:35:30] [WORLD__LAUNCH] zones: dynamic_20 reported state STARTED (555 starts)

Hint: This bug will eventually eat up all Dynamic zones until there are no more to boot.


I do not recall if this happened editing/changing other data. I know this stuff is probably "shared memory" stuff, but I do not remember it ever crashing the zones before - but it could have been my noobness at the time.

Anyway, the only solution we found was to completely shut down the server and restart it. Or, if you were running zones with zone.exe (instead of eqlaunch zones), once the zone.exe reloaded it seemed ok. In this, I am reporting a bug (possibly?) in eqlaunch.exe not reloading the zone.exe properly. Kinda guessing on that last part. Hopefully someone can take a look at this or tell me "works as expected", which would suck, but at least I know how to fix it.

As far as Linux goes, I cannot say I've seen the same behavior, but that's only because Windows logging was easier to watch. My old FC4 server would completely crash (the OS) for what appeared no reason, so maybe there's something similar going on there.


Thanks,
-J

[John Adams]

One more thing, if you are NOT using eqlaunch.exe to launch these zones, and this occurs, the zone.exe simply crashes and gives the usual JIT/WER error boxes and closes down (I think it even brings up a Windows Explorer window), but does not attempt to restart. Hence my leanings towards eqlaunch.exe being suspect - not at fault, but maybe adding to the problem.

[cavedude]

I can say for certain that Linux does not experience this issue, otherwise Grand Creation would be crashing zones left and right. As it stands now we only crash 0-2 a day tops, and 50% of them are caused by a #reloadqst crash bug.

[John Adams]

Cool, I haven't run an emulator on FC5 yet, so it could have been FC4 that was the root of those problems. So does TGC change loot/factions often? So far, that's the only condition I found to recreate this problem. #reloadpl or qst has never been a problem for me.

[cavedude]

Loot, yes. Lately, we've been creating a lot of new loot tables for PoP and back issues. I would think that spells would also be effected, since they also are in shared memory. We've created tons of new spell lists as well with no issue.

The reloadqst bug pops its head up once in a while for us, maybe once a week. So, I doubt many others would see it even that much.