Debug Assertion Failed Zone.exe

[Riklin]

I recompiled yesterday to resolve a bunch of other errors and got a new one. This one is the usual Microsoft Visual C++ Runtime Library message
Debug Assertion Failed!

Program C:\eq\EQEmuServer\zone.exe

This one is specific to list iterator.

Here is the debug info:

[05-14-2015 :: 09:49:44] [Crash] EXCEPTION_BREAKPOINT
[05-14-2015 :: 09:49:44] [Crash] SymInit: Symbol-SearchPath: '.;C:\EQ\EQEmuServer;C:\EQ\EQEmuServer;C:\Windows; C:\Windows\system32;SRV*C:\websymbols*http://msdl.microsoft.com/download/symbols;', symOptions: 530, UserName: 'Rick'
[05-14-2015 :: 09:49:44] [Crash] OS-Version: 6.1.7601 (Service Pack 1) 0x100-0x1
[05-14-2015 :: 09:49:44] [Crash] C:\EQ\EQEmuServer\zone.exe:zone.exe (000000013F210000), size: 38916096 (result: 0), SymType: 'PDB', PDB: 'C:\EQ\EQEmuServer\zone.exe'
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\SYSTEM32\ntdll.dll:ntdll.dll (0000000077740000), size: 1740800 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\SYSTEM32\ntdll.dll', fileVersion: 6.1.7601.18839
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\kernel32.dll:kernel32.dll (0000000077620000), size: 1175552 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\kernel32.dll', fileVersion: 6.1.7601.18839
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\KERNELBASE.dll:KERNELBASE.dll (000007FEFD810000), size: 442368 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\KERNELBASE.dll', fileVersion: 6.1.7601.18839
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\SHELL32.dll:SHELL32.dll (000007FEFE790000), size: 14192640 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\SHELL32.dll', fileVersion: 6.1.7601.18762
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\msvcrt.dll:msvcrt.dll (000007FEFDFF0000), size: 651264 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\msvcrt.dll', fileVersion: 7.0.7601.17744
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\SHLWAPI.dll:SHLWAPI.dll (000007FEFF9D0000), size: 462848 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\SHLWAPI.dll', fileVersion: 6.1.7601.17514
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\GDI32.dll:GDI32.dll (000007FEFDCB0000), size: 421888 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\GDI32.dll', fileVersion: 6.1.7601.18778
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\USER32.dll:USER32.dll (0000000077520000), size: 1024000 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\USER32.dll', fileVersion: 6.1.7601.17514
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\LPK.dll:LPK.dll (000007FEFE130000), size: 57344 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\LPK.dll', fileVersion: 6.1.7601.18768
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\USP10.dll:USP10.dll (000007FEFE340000), size: 823296 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\USP10.dll', fileVersion: 1.626.7601.18454
[05-14-2015 :: 09:49:44] [Crash] C:\Perl\bin\perl514.dll:perl514.dll (0000000028000000), size: 2207744 (result: 0), SymType: '-exported-', PDB: 'C:\Perl\bin\perl514.dll', fileVersion: 5.14.2.1402
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\ADVAPI32.dll:ADVAPI32.dll (000007FEFF8F0000), size: 897024 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\ADVAPI32.dll', fileVersion: 6.1.7601.18839
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\SYSTEM32\sechost.dll:sechost.dll (000007FEFE140000), size: 126976 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\SYSTEM32\sechost.dll', fileVersion: 6.1.7601.18839
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\RPCRT4.dll:RPCRT4.dll (000007FEFD960000), size: 1232896 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\RPCRT4.dll', fileVersion: 6.1.7601.18532
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\WS2_32.dll:WS2_32.dll (000007FEFF8A0000), size: 315392 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\WS2_32.dll', fileVersion: 6.1.7601.17514
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\system32\NSI.dll:NSI.dll (000007FEFDA90000), size: 32768 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\NSI.dll', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:44] [Crash] C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.7601.18201_none_a4d 3b9377117c3df\COMCTL32.dll:COMCTL32.dll (000007FEFA370000), size: 655360 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.7601.18201_none_a4d 3b9377117c3df\COMCTL32.dll', fileVersion: 5.82.7601.18201
[05-14-2015 :: 09:49:44] [Crash] C:\EQ\EQEmuServer\lua51.dll:lua51.dll (000007FEEA8F0000), size: 450560 (result: 0), SymType: '-exported-', PDB: 'C:\EQ\EQEmuServer\lua51.dll'
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\MSVCR110.dll:MSVCR110.dll (000007FEF68F0000), size: 868352 (result: 0), SymType: 'PDB', PDB: 'C:\Windows\system32\MSVCR110.dll', fileVersion: 11.0.51106.1
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\VERSION.dll:VERSION.dll (000007FEFC5D0000), size: 49152 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\VERSION.dll', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\IMM32.DLL:IMM32.DLL (000007FEFF870000), size: 188416 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\IMM32.DLL', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\MSCTF.dll:MSCTF.dll (000007FEFE5A0000), size: 1085440 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\MSCTF.dll', fileVersion: 6.1.7601.18731
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\NLAapi.dll:NLAapi.dll (000007FEFB170000), size: 86016 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\NLAapi.dll', fileVersion: 6.1.7601.17964
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\napinsp.dll:napinsp.dll (000007FEF8C40000), size: 86016 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\napinsp.dll', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\pnrpnsp.dll:pnrpnsp.dll (000007FEF8C20000), size: 102400 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\pnrpnsp.dll', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\System32\mswsock.dll:mswsock.dll (000007FEFCCA0000), size: 348160 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\System32\mswsock.dll', fileVersion: 6.1.7601.18254
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\DNSAPI.dll:DNSAPI.dll (000007FEFCA70000), size: 372736 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\DNSAPI.dll', fileVersion: 6.1.7601.17570
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\System32\winrnr.dll:winrnr.dll (000007FEF8C10000), size: 45056 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\System32\winrnr.dll', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\rasadhlp.dll:rasadhlp.dll (000007FEF7E20000), size: 32768 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\rasadhlp.dll', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\System32\wshtcpip.dll:wshtcpip.dll (000007FEFC6A0000), size: 28672 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\System32\wshtcpip.dll', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\System32\wship6.dll:wship6.dll (000007FEFCC40000), size: 28672 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\System32\wship6.dll', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\IPHLPAPI.DLL:IPHLPAPI.DLL (000007FEFAC70000), size: 159744 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\IPHLPAPI.DLL', fileVersion: 6.1.7601.17514
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\WINNSI.DLL:WINNSI.DLL (000007FEFAC50000), size: 45056 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\WINNSI.DLL', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\System32\fwpuclnt.dll:fwpuclnt.dll (000007FEFAB00000), size: 339968 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\System32\fwpuclnt.dll', fileVersion: 6.1.7601.18283
[05-14-2015 :: 09:49:45] [Crash] C:\Perl\lib\auto\List\Util\Util.dll:Util.dll (0000000010000000), size: 36864 (result: 0), SymType: '-exported-', PDB: 'C:\Perl\lib\auto\List\Util\Util.dll'
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\cryptbase.dll:cryptbase.dll (000007FEFD3A0000), size: 61440 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\cryptbase.dll', fileVersion: 6.1.7600.16385
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\ole32.dll:ole32.dll (000007FEFDAA0000), size: 2109440 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\ole32.dll', fileVersion: 6.1.7601.17514
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\dbghelp.dll:dbghelp.dll (000007FEF9630000), size: 1200128 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\dbghelp.dll', fileVersion: 6.1.7601.17514
[05-14-2015 :: 09:49:45] [Crash] C:\Windows\system32\SspiCli.dll:SspiCli.dll (000007FEFD2D0000), size: 151552 (result: 0), SymType: '-exported-', PDB: 'C:\Windows\system32\SspiCli.dll', fileVersion: 6.1.7601.18839
[05-14-2015 :: 09:49:45] [Crash] f:\dd\vctools\crt\crtw32\stdcpp\stdthrow.cpp (15): std::_Debug_message
[05-14-2015 :: 09:49:45] [Crash] c:\program files (x86)\microsoft visual studio 12.0\vc\include\list (211): std::_List_const_iterator<std::_List_val<std::_List_simple_types<std::pair<unsigned short const ,NPC * __ptr64> > > >::operator*
[05-14-2015 :: 09:49:45] [Crash] c:\program files (x86)\microsoft visual studio 12.0\vc\include\list (364): std::_List_iterator<std::_List_val<std::_List_simple_types<std::pair<unsigned short const ,NPC * __ptr64> > > >::operator*
[05-14-2015 :: 09:49:45] [Crash] c:\program files (x86)\microsoft visual studio 12.0\vc\include\list (368): std::_List_iterator<std::_List_val<std::_List_simple_types<std::pair<unsigned short const ,NPC * __ptr64> > > >::operator->
[05-14-2015 :: 09:49:45] [Crash] c:\eq\source\zone\entity.cpp (739): EntityList::CheckSpawnQueue
[05-14-2015 :: 09:49:45] [Crash] c:\eq\source\zone\entity.cpp (2366): EntityList::Process
[05-14-2015 :: 09:49:45] [Crash] c:\eq\source\zone\net.cpp (424): main
[05-14-2015 :: 09:49:45] [Crash] f:\dd\vctools\crt\crtw32\startup\crt0.c (255): __tmainCRTStartup
[05-14-2015 :: 09:49:45] [Crash] f:\dd\vctools\crt\crtw32\startup\crt0.c (165): mainCRTStartup
[05-14-2015 :: 09:49:45] [Crash] ERROR: SymGetLineFromAddr64, GetLastError: 487 (Address: 00000000776359CD)
[05-14-2015 :: 09:49:45] [Crash] 00000000776359CD (kernel32): (filename not available): BaseThreadInitThunk
[05-14-2015 :: 09:49:45] [Crash] ERROR: SymGetLineFromAddr64, GetLastError: 487 (Address: 000000007776B981)
[05-14-2015 :: 09:49:45] [Crash] 000000007776B981 (ntdll): (filename not available): RtlUserThreadStart

[epilz]

What zone is crashing? Seems mine is always getting this error is ssratemple

[Uleat]

It's definitely an assert failure.

Have you done anything that might increase your in-zone entity count to above 65535?

[Uleat]

You can try this patch to see if it helps identify the problem:

zone/entity.cpp | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/zone/entity.cpp b/zone/entity.cpp
index 5646f08..0f1246a 100644
--- a/zone/entity.cpp
+++ b/zone/entity.cpp
@@ -736,9 +736,14 @@ void EntityList::CheckSpawnQueue()
Mob::CreateSpawnPacket(outapp, ns);
QueueClients(0, outapp);
auto it = npc_list.find(ns->spawn.spawnId);
- NPC *pnpc = it->second;
- pnpc->SendArmorAppearance();
- pnpc->SetAppearance(pnpc->GetGuardPointAnim(),false);
+ if (it == npc_list.end()) {
+ Log.Out(Logs::General, Logs::Error, "Error in EntityList::CheckSpawnQueue: Unable to find NPC for spawnId '%i'", ns->spawn.spawnId);
+ }
+ else {
+ NPC *pnpc = it->second;
+ pnpc->SendArmorAppearance();
+ pnpc->SetAppearance(pnpc->GetGuardPointAnim(), false);
+ }
safe_delete(outapp);
iterator.RemoveCurrent();
}



That assertion looks like it is being triggered by calling a dereference on the list::end() object.


EDIT: The '%i' should probably be '%u' in that log message..but, any value greater than 65535 or less than 0 would indicate an issue.


EDIT2: This is a troubleshooting patch..playing in a zone where the log message is triggered may cause issues with clients being attacked by 'invisible' mobs.

[Humon]

Good to see that i'm not the only one with this problem, mine is a bit different. When it happens i don't get any log crashes but just a window pops up on the server computer with the crash and nothing shows up in the game or server logs, it just sits zoning till i close the game. The 3 zones that this happens to are ones that i have tried and failed to put merchants in. I have deleted the zones form my Maps folder in the server and re-downloaded them but the problem still happens.

The following is the contents of the Micorsoft Visual C++ Runtime Library window that pops up.

Debug Assertion Failed!

Program: C:\EQ\EQEmuServer\zone.exe
File: C:\Program Files\Microsoft Visual Studio 12.0\VC\include\xtree
Line: 237

Expression: map/set iterator not dereferencable

[epilz]

Human, what version of source are you running? Demonstar put in debugging for the crashes and it should always create a crash log. It happens quite frequent in SSRA and used to in Skyfire, but no one is in that zone much anymore

[Uleat]

Those are basically a deference of an empty container failures.

We just need to check the iterator against <container_type>::end() and handle it appropriately.


Humon, if you could post a few more lines of the error log, it would help pin-point the deference catch.

We would just need 3-4 lines of server-originating code (common/world/zone/etc... sub-folder) and then up the the
"f:\dd\vctools\crt\crtw32\stdcpp\stdthrow.cpp (15): std::_Debug_message" line..or whatever equivalent handler is throwing
the message.

[Riklin]

Uleat, I think I understand what you are asking, but want to make sure... This is the code in entity.cpp for void EntityList::CheckSpawnQueue(). I want to confirm, you want me to replace this code with the code you included above?


void EntityList::CheckSpawnQueue()
{
// Send the stuff if the oldest packet on the queue is older than 50ms -Quagmire
if (tsFirstSpawnOnQueue != 0xFFFFFFFF && (Timer::GetCurrentTime() - tsFirstSpawnOnQueue) > 50) {
LinkedListIterator<NewSpawn_Struct *> iterator(SpawnQueue);
EQApplicationPacket *outapp = 0;

iterator.Reset();
NewSpawn_Struct *ns;

while(iterator.MoreElements()) {
outapp = new EQApplicationPacket;
ns = iterator.GetData();
Mob::CreateSpawnPacket(outapp, ns);
QueueClients(0, outapp);
auto it = npc_list.find(ns->spawn.spawnId);
NPC *pnpc = it->second;
pnpc->SendArmorAppearance();
pnpc->SetAppearance(pnpc->GetGuardPointAnim(),false);
safe_delete(outapp);
iterator.RemoveCurrent();
}
tsFirstSpawnOnQueue = 0xFFFFFFFF;
NumSpawnsOnQueue = 0;
}
}

[Uleat]

Yes, if you're not familiar with applying patches, you can manually enter the code.

Just take the lines beginning with '-' and delete them..then add the lines beginning with '+'


Essentially, you will end up with this:
void EntityList::CheckSpawnQueue()
{
// Send the stuff if the oldest packet on the queue is older than 50ms -Quagmire
if (tsFirstSpawnOnQueue != 0xFFFFFFFF && (Timer::GetCurrentTime() - tsFirstSpawnOnQueue) > 50) {
LinkedListIterator<NewSpawn_Struct *> iterator(SpawnQueue);
EQApplicationPacket *outapp = 0;

iterator.Reset();
NewSpawn_Struct *ns;

while(iterator.MoreElements()) {
outapp = new EQApplicationPacket;
ns = iterator.GetData();
Mob::CreateSpawnPacket(outapp, ns);
QueueClients(0, outapp);
auto it = npc_list.find(ns->spawn.spawnId);
if (it == npc_list.end()) {
Log.Out(Logs::General, Logs::Error, "Error in EntityList::CheckSpawnQueue: Unable to find NPC for spawnId '%u'", ns->spawn.spawnId);
}
else {
NPC *pnpc = it->second;
pnpc->SendArmorAppearance();
pnpc->SetAppearance(pnpc->GetGuardPointAnim(),false);
}
safe_delete(outapp);
iterator.RemoveCurrent();
}
tsFirstSpawnOnQueue = 0xFFFFFFFF;
NumSpawnsOnQueue = 0;
}
}
Note: I replaced the '%i' with '%u' in the log message code above.


What is happening is that we're calling a deference on (npc_list::iterator<unsigned short, NPC *>)it when we say 'it->second'

For some reason, npc_list is returning npc_list::end() when we assign 'it = npc_list.find(ns->spawn.spawnId)'

And since npc_list::end() is not dereferenceable, we get an assertion failure and the 'std::_Debug_message' call.


Again, not knowing the cause of this, you may have players being attacked by unseen and/or untargetable mobs..so,
be careful and watchful when your problem zones are active.


I've been looking at this from an downcast issue (uint32 -> uint16 with loss of data.)

One of the other dev's pointed out that it might be an empty npc_list..or some other queue-based issue.

Your results will help point which direction to go with this.

[Uleat]

Btw, have you had issues with mobs that you know should be down being up as soon as a player zones in?

[demonstar55]

https://github.com/ProjectEQ/projecteqquests/blob/master/soltemple/80002.pl That quest is causing soltemple to crash sometimes

[Riklin]

I have problems with Kithicor seemedly always having the high level undead in the zone regardless of it being day or night. I'm now sure the timing stuff works (spawn_events table stuff), because I was able to fix lake rathe undead gnolls at night and live gnolls during day) and qeynos hills (pizgyn). That said, Kithicor still has commanders, etc running around at all hours.

I am not totally sure, but I suspect if someone stayed in the zone for 1-2 game days, this would clear up, but very few people WANT to stay in kith...

Anyway that is the only zone that I know of where things spawn wrong that isn't explainable with the day/nite stuff.

[Uleat]

Ok, that check should be in the code base now as a fix. If that causes any unknown issues, please post back.


I caught wind that the kithicor mobs may be perma-spawned..might check/ask around about that.


EDIT: Riklin, just 'revert' your local repository and then do a 'pull' to sync it with the remote repository.
You shouldn't need to recompile if that code snippet matches and you changed nothing else.

[Riklin]

Thanks! I reverted the repo last night. I'll check for error messages when I get home from work, tonight...

Thanks, BTW!

[Riklin]

Just wanted to chime in about the Kithicor high end mobs being permanent. The main reason to go to kithicor is the ranger ivy armor quests. Server is a progression server now, so there are rangers and I've received a comment about the undead beating on a lower level ranger during the day. It does indeed make things a bit more um, "interesting"...

Need eyes in the back of your head there...