Strange Crash

[KingMort]

Not sure wtf this means but this is pretty much the only crash I get... Was hoping some one could help me figure out how to fix this.


Core was generated by `./zone'.
Program terminated with signal 11, Segmentation fault.
[New process 27175]
[New process 27181]
[New process 27180]
[New process 27179]
[New process 27178]
#0 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x000000000060abf1 in Client::GetReverseFactionCon (this=0x22c02c0,
iOther=0x7f6dfc56eda0) at faction.cpp:180
#2 0x0000000000718dd6 in Mob::CheckWillAggro (this=0x7f6dfc585390,
mob=0x22c02c0) at aggro.cpp:312
#3 0x0000000000719042 in EntityList::CheckClientAggro (this=0xbdcfa0,
around=0x22c02c0) at aggro.cpp:45
#4 0x00000000005c1156 in Client::Process (this=0x22c02c0)
at client_process.cpp:672
#5 0x0000000000585ff0 in EntityList::MobProcess (this=0xbdcfa0)
at entity.cpp:487
#6 0x00000000005d1c6c in main (argc=1, argv=0x7fff116d93a8) at net.cpp:494

[trevius]

I get that crash occasionally as well. I think it has something to do with a certain scenario when it is doing the CheckWillAggro check. I don't really know how to read those very well and didn't see any problems that stood out in the code for it, but I do get it too.

[KingMort]

On my end it's like a plague , hits at least once every couple of hours..

[gaeorn]

this part
#0 0x0000000000000000 in ?? ()

shows it can not identify the function it was in. this often times means memory corruption, if it indeed in memory corruption, it is possible the backtrace is completely incorrect. this kind of problem can be very hard to track down.

[OscarGrouch05]

make sure you have all the zone files in the map dir under eqemu..
it's telling mobs can't argo you because you are missing files
under the map dir or other files in your eq dir to go with the zone
your in. that should solve this problem.

[Secrets]

make sure you have all the zone files in the map dir under eqemu..
it's telling mobs can't argo you because you are missing files
under the map dir or other files in your eq dir to go with the zone
your in. that should solve this problem.

Memory corruption has no relation to .map files. Trust me on this, if it's truely memory corruption, then it's going to be a completely different issue than what the coredump is getting at, most of the time.

[cavedude]

PEQ has seen this exact crash for years, and others very similar to it also calling Mob::CheckWillAggro. So you aren't alone Mort. However, in our case it's a once weekly thing and not as common as you're seeing. Derision's latest commit might actually help this some, the crash he corrected is in the same realm. Though, he worked with quest::depopall, and this crash has been happening to PEQ before that function even went in.

[Derision]

For this particular crash, you can find out what zone it occured in by executing:

gdb ./zone <core file name>
<snip>
(gdb) frame 6
#6 0x0810bc19 in main (argc=142145976, argv=0x9) at net.cpp:473
473 if(net.group_timer.Enabled() && net.group_timer.Check())
(gdb) print zone->short_name
$1 = 0x8431068 "potactics"
(gdb) frame 1
(gdb) print name
<should print client name involved in crash>
(gdb) print iOther->name
<should print mob name in involved in crash>


If the zone and mob name are fairly consistent, it may give you somewhere to concentrate on, e.g. any quest .pl files that mob is involved in.

[KingMort]

Ok this is what i got back for one of the crashes

Core was generated by `./zone kerraridge'.
Program terminated with signal 11, Segmentation fault.
[New process 13604]
[New process 13609]
[New process 13608]
[New process 13607]
[New process 13606]
#0 0x0000000000000000 in ?? ()
(gdb) frame 6
#6 0x00000000005d1c6c in main (argc=2, argv=0x7fff12106dc8) at net.cpp:494
494 entity_list.MobProcess();
(gdb) print zone->short_name
$1 = 0x233df00 "kerraridge"
(gdb) frame 1
#1 0x000000000060abf1 in Client::GetReverseFactionCon (this=0x3088740,
iOther=0x2aafa20) at faction.cpp:180
180 if (iOther->GetPrimaryFaction() < 0)
(gdb) print name
$2 = "Muffinbut", '\0' <repeats 54 times>
(gdb) print iOther->name
$3 = "Harry\000e", '\0' <repeats 56 times>


Alright well Muffinbut is his main char and Harry is his Alt..

[OscarGrouch05]

what it's telling me is that you are not seeing the faction hits message
and when you con the mob it's should tell you if your friend or foe.
like i stated before missing files this could fall on the server side not the
client.

you read books i read machine language!

[KingMort]

No i don't read books

But anyway what I did last night was cleaned up all of our Useless factions and faction_entries there was a ton of them not even in use and I haven't seen this crash since then.. I hope that I fixed it doing this but not posative..

As for the maps thing I will have to check into that but i'm pretty sure we are all caught up on all the maps..

Does some one have a direct SVN addy for me to download all the maps ?

[pfyon]

http://code.google.com/p/eqemumaps/source/checkout

or http://eqemumaps.googlecode.com/svn/trunk/

[KingMort]

Alright well this was the latest drop.. I took out all un-needed factions in the game and found a fairly large reduction in the crashes but still...


Core was generated by `./zone kerraridge'.
Program terminated with signal 11, Segmentation fault.
[New process 5126]
[New process 5131]
[New process 5130]
[New process 5129]
[New process 5128]
#0 0x00000000008df0d6 in typeinfo name for Entity ()
(gdb) bt
#0 0x00000000008df0d6 in typeinfo name for Entity ()
#1 0x000000000060abf1 in Client::GetReverseFactionCon (this=0x7feab85bb4b0,
iOther=0x2921ce0) at faction.cpp:180
#2 0x0000000000718dd6 in Mob::CheckWillAggro (this=0x7feab8613280,
mob=0x7feab85bb4b0) at aggro.cpp:312
#3 0x0000000000719042 in EntityList::CheckClientAggro (this=0xbdcfa0,
around=0x7feab85bb4b0) at aggro.cpp:45
#4 0x00000000005c1156 in Client::Process (this=0x7feab85bb4b0)
at client_process.cpp:672
#5 0x0000000000585ff0 in EntityList::MobProcess (this=0xbdcfa0)
at entity.cpp:487
#6 0x00000000005d1c6c in main (argc=2, argv=0x7fffca4220e8) at net.cpp:494
(gdb) frame 6
#6 0x00000000005d1c6c in main (argc=2, argv=0x7fffca4220e8) at net.cpp:494
494 entity_list.MobProcess();
(gdb) print zone->short_name
$1 = 0x22857e0 "kerraridge"
(gdb) frame 1
#1 0x000000000060abf1 in Client::GetReverseFactionCon (this=0x7feab85bb4b0,
iOther=0x2921ce0) at faction.cpp:180
180 if (iOther->GetPrimaryFaction() < 0)
(gdb) print name
$2 = "Caesar", '\0' <repeats 57 times>
(gdb) print iOther->name
$3 = "Sand\000me", '\0' <repeats 56 times>
(gdb)


Caesar is a player, and so is Sand ...

[Derision]

This makes no sense to me. If we consider frame 3

#3 0x0000000000719042 in EntityList::CheckClientAggro (this=0xbdcfa0,
around=0x7feab85bb4b0) at aggro.cpp:45


This is the code from aggro.cpp around that line:

39 for(iterator.Reset(); iterator.MoreElements(); iterator.Advance()) {
40 _ZP(EntityList_CheckClientAggro_Loop);
41 Mob* mob = iterator.GetData();
42 if(mob->IsClient()) //also ensures that mob != around
43 continue;
44
45 if(mob->CheckWillAggro(around)) {


So in your backtrace, mob->CheckWillAggro is called at line 45, but you are saying that both mobs involved are players, but line 42 checks if mob is a client and will skip line 45 and move on to the next mob if that is the case.

Strange.

I note that both the BTs you posted are for kerraridge. Do all the crashes happen there ?

Have you talked to the players involved in the crashes to see what they were doing at the time the zone crashed ?

[trevius]

Could this be due to players being charmed there by chance?

[KingMort]

No not always in Kerra Ridge it's just wherever and whenever.. Some times there some times in other zones...

I asked them what they were doing before the crash and they said nothing other than clicking on the other player.. And in all cases of this crash it seems it's players clicking on other players that cause the crash and never any NPC's involved..

Is there any more BT commands i could do to help you guys figure this one out ?? I'll do anything at this point lol

[gaeorn]

Mort's code isn't pure emu, nor is it 100% up to date (although it's much more recent than it was a few months ago). So chances are there is some other code around line 45.

[KingMort]

My code is the same as anyones honestly other than a few very minor changes... always tried to do my own thing except recently i have found that my fellow artists and myself need to hopefully gain some type of community effort hehe ..

My database however is probably pretty old school or something..

Gaeorn are you not on IRC anymore I think i have a solution for all of this if you just will contact me please :)

[AndMetal]

What's up with the nulls in the characters' names?


(gdb) print iOther->name
$3 = "Harry\000e", '\0' <repeats 56 times>


(gdb) print iOther->name
$3 = "Sand\000me", '\0' <repeats 56 times>

[KingMort]

Here is a few more that seem a bit different


Core was generated by `./zone'.
Program terminated with signal 11, Segmentation fault.
[New process 5453]
[New process 5459]
[New process 5458]
[New process 5457]
[New process 5456]
#0 0x00000000014fff60 in ?? ()
(gdb) bt
#0 0x00000000014fff60 in ?? ()
#1 0x00000000008b77cf in Bot::AI_Process (this=0x18c3400) at bot.cpp:2147
#2 0x00000000008bd48b in Bot::Process (this=0x18c3400) at bot.cpp:1120
#3 0x0000000000585ff0 in EntityList::MobProcess (this=0xbdcfa0)
at entity.cpp:487
#4 0x00000000005d1c6c in main (argc=1, argv=0x7fffa7da1a68) at net.cpp:494
(gdb) frame 6
#0 0x0000000000000000 in ?? ()
(gdb) print zone->short_name
$1 = 0x7fbd95691c30 "dreadspire"
(gdb) frame 1
#1 0x00000000008b77cf in Bot::AI_Process (this=0x18c3400) at bot.cpp:2147
2147 if(BotOwner->qglobal || (GetAppearance() == eaDead) || BotOwner->IsBot())
(gdb) print name
$2 = "SandsHealerGoon\00000", '\0' <repeats 45 times>
(gdb) print iOther->name
No symbol "iOther" in current context.
(gdb)




Core was generated by `./zone'.
Program terminated with signal 11, Segmentation fault.
[New process 5446]
[New process 5452]
[New process 5451]
[New process 5450]
[New process 5449]
#0 0x00007f32506a2020 in ?? ()
(gdb) bt
#0 0x00007f32506a2020 in ?? ()
#1 0x00000000008b77cf in Bot::AI_Process (this=0x7f3250ddebe0) at bot.cpp:2147
#2 0x00000000008bd48b in Bot::Process (this=0x7f3250ddebe0) at bot.cpp:1120
#3 0x0000000000585ff0 in EntityList::MobProcess (this=0xbdcfa0)
at entity.cpp:487
#4 0x00000000005d1c6c in main (argc=1, argv=0x7fff626e93b8) at net.cpp:494
(gdb) frame 6
#0 0x0000000000000000 in ?? ()
(gdb) print zone->short_name
$1 = 0x7f32496c7330 "kerraridge"
(gdb) frame 1
#1 0x00000000008b77cf in Bot::AI_Process (this=0x7f3250ddebe0) at bot.cpp:2147
2147 if(BotOwner->qglobal || (GetAppearance() == eaDead) || BotOwner->IsBot())
(gdb) print name
$2 = "SandsHealerGoon\00000", '\0' <repeats 45 times>
(gdb)

[KingMort]

Yet another strange Crash


Core was generated by `./zone'.
Program terminated with signal 11, Segmentation fault.
[New process 3775]
[New process 3781]
[New process 3780]
[New process 3779]
[New process 3778]
#0 0x00007fc3f80007e0 in ?? ()
(gdb) bt
#0 0x00007fc3f80007e0 in ?? ()
#1 0x00000000008b30f5 in Bot::ProcessClientZoneChange (
botOwner=0x7fc3fd0c7ac0) at bot.cpp:8276
#2 0x0000000000823e58 in Client::Handle_OP_ZoneChange (this=0x7fc3fd0c7ac0,
app=0x7fc40271b2f0) at zoning.cpp:38
#3 0x00000000006f8266 in Client::HandlePacket (this=0x7fc3fd0c7ac0,
app=0x7fc40271b2f0) at client_packet.cpp:435
#4 0x00000000005c1050 in Client::Process (this=0x7fc3fd0c7ac0)
at client_process.cpp:662
#5 0x0000000000585ff0 in EntityList::MobProcess (this=0xbdcfa0)
at entity.cpp:487
#6 0x00000000005d1c6c in main (argc=1, argv=0x7fff127cf498) at net.cpp:494
(gdb) frame 6
#6 0x00000000005d1c6c in main (argc=1, argv=0x7fff127cf498) at net.cpp:494
494 entity_list.MobProcess();
(gdb) print zone->short_name
$1 = 0x7fc3fd45cbd0 "kerraridge"
(gdb) frame 1
#1 0x00000000008b30f5 in Bot::ProcessClientZoneChange (
botOwner=0x7fc3fd0c7ac0) at bot.cpp:8276
8276 if(tempBot->HasGroup())
(gdb) print name
$2 = 0x9325da "Live"

[gaeorn]

Gaeorn are you not on IRC anymore I think i have a solution for all of this if you just will contact me please :)

I'm on all day during weekdays. Just need to look me up by name if you don't see me in a particular channel.

[OscarGrouch05]

simple and sweet

it's like this the zone # and name is not found by the server.
i looked in the exe and can match up opcodes with zone name
short and long name of zone and zone number. the address are
corrcect for the long name but missing the short name of the zone
was expecting the short name of the zone in the code.

question on the bot AI are you refer i have seen the opcodes for that as well inside exe file.
looks like programer need to rewrite that looks like is calling wrong opcodes and needs
to be fix and recompiled.

what program do you guys use i use this etu program 80)

[KingMort]

So how do I go about fixing this ? Download maps or what

[KingMort]

Just an update, I replaced all my map files with the latest current stuff and it seems some of the crashing has cleared up thanks for that suggestion..

I have some NEW ones I guess (maybe not though could be still related) only difference here is that it seems like it's a mob now instead of a player..




Core was generated by `./zone'.
Program terminated with signal 11, Segmentation fault.
[New process 5084]
[New process 5090]
[New process 5089]
[New process 5088]
[New process 5087]
#0 0x0000000000718752 in EntityList::AIYellForHelp (this=0xbdcfa0,
sender=0x7f39ddec56a0, attacker=0x84d4210) at aggro.cpp:450
450 if (
(gdb) bt
#0 0x0000000000718752 in EntityList::AIYellForHelp (this=0xbdcfa0,
sender=0x7f39ddec56a0, attacker=0x84d4210) at aggro.cpp:450
#1 0x00000000005c7292 in NPC::Process (this=0x7f39ddec56a0) at npc.cpp:620
#2 0x0000000000585ff0 in EntityList::MobProcess (this=0xbdcfa0)
at entity.cpp:487
#3 0x00000000005d1c6c in main (argc=1, argv=0x7fff00976648) at net.cpp:494
(gdb) frame 6
#0 0x0000000000000000 in ?? ()
(gdb) print zone->short_name
$1 = 0x47ce510 "steamfont"
(gdb) frame 1
#1 0x00000000005c7292 in NPC::Process (this=0x7f39ddec56a0) at npc.cpp:620
620 entity_list.AIYellForHelp(this, GetTarget());
(gdb) print name
$2 = "a_large_rat001", '\0' <repeats 49 times>
(gdb)

[gaeorn]

At least that crash has a function name we can check into. I'll see if I can look into this in the next day or so.

[Congdar]

http://code.google.com/p/projecteqemu/issues/detail?id=28

[OscarGrouch05]

that is a ref to what andmetal said above on 1-16-2010
What's up with the nulls in the characters' names?

[KingMort]

Wow still getting these , and Now i know for sure it is BOT related, which on my end they are called GOONS but all i did was change the #bot commands and descriptions to #goon didn't mess with anything else... anyway here is the most recent crash which was just after giving the player his new goon from my RA promo...


Core was generated by `./zone'.
Program terminated with signal 11, Segmentation fault.
[New process 26798]
[New process 26804]
[New process 26803]
[New process 26802]
[New process 26801]
#0 0x00007f74e4000850 in ?? ()
(gdb) bt
#0 0x00007f74e4000850 in ?? ()
#1 0x000000000060abf1 in Client::GetReverseFactionCon (this=0x7f74e443f260,
iOther=0x7f74e43f42b0) at faction.cpp:180
#2 0x0000000000718d5e in Mob::CheckWillAggro (this=0x7f74e41a2c30,
mob=0x7f74e443f260) at aggro.cpp:312
#3 0x0000000000718fca in EntityList::CheckClientAggro (this=0xbdcfa0,
around=0x7f74e443f260) at aggro.cpp:45
#4 0x00000000005c1156 in Client::Process (this=0x7f74e443f260)
at client_process.cpp:672
#5 0x0000000000585ff0 in EntityList::MobProcess (this=0xbdcfa0)
at entity.cpp:487
#6 0x00000000005d1c6c in main (argc=1, argv=0x7ffff75141e8) at net.cpp:494
(gdb) frame 6
#6 0x00000000005d1c6c in main (argc=1, argv=0x7ffff75141e8) at net.cpp:494
494 entity_list.MobProcess();
(gdb) print zone->short_name
$1 = 0x1aff910 "erudsxing"
(gdb) frame 1
#1 0x000000000060abf1 in Client::GetReverseFactionCon (this=0x7f74e443f260,
iOther=0x7f74e43f42b0) at faction.cpp:180
180 if (iOther->GetPrimaryFaction() < 0)
(gdb) print name
$2 = "Xion\000me", '\0' <repeats 56 times>
(gdb) print iOther->name
$3 = "Shao\000me", '\0' <repeats 56 times>
(gdb)

[OscarGrouch05]

did you change this too i think your error is coming from this
they still have the name bot like Botowner an exmple.

#1 0x00000000008b77cf in Bot::AI_Process (this=0x18c3400) at bot.cpp:
2147 if(BotOwner->qglobal || (GetAppearance() == eaDead) || BotOwner->IsBot())

[KingMort]

Would you be able to post a code fix for this ?? I can put stuff into files and compile ok just not very good with C++ in general if you would tell me what code to fix and with what would be a huge help..

:) Thank you guys appreciate it

[OscarGrouch05]

look at the code edit and fine BotOwner->IsBot()) - changes that need to be done or corrected Isbot change to Isgoon recompile program that should help fix the problem. i'm just good at reading the error code i program in Pascal i would have to covert to my lang so i could understand what it's doing i hope someone else can correct this for you. after changing in program you should be able to run the debug program to check the code.

your error are coming from (GetAppearance() == eaDead) || BotOwner->IsBot()) =GDB
look what you have been posting you'll see (gdb) there is your problem.

[OscarGrouch05]

KingMort are you using EQEmu-Rev1180a-Bots? or if your pulling the code from the exe all smiles here 80)

[gaeorn]

The function names containing the word 'bot' does not matter. If you have changed the commands to "goon" it will work fine. If you had changed some of the functions but not all references to the functions, the code would not compile due to unreferenced symbols. No offense, but OscarGrouch05 does not know what he is talking about in this regard.

[OscarGrouch05]

Ref code 2147 if(BotOwner->qglobal || (GetAppearance() == eaDead) || BotOwner->IsBot()) does stand for GetApperance,Dead,Bot = gdb this is where your errors are coming from i would like and update when you solve this problem Thank You!

[KingMort]

Yeah hehe i just want to see this bug fixed it's terrible having players crash every 15 minutes..

Probably one of the major reasons my servers population is so low atm

[OscarGrouch05]

1. make sure you installed InnoDB using your perl client
2. double check your database table's

I)Database changes

-- Table "botinventory" DDL

CREATE TABLE `botinventory` (
`npctypeid` int(11) NOT NULL default '0',
`botslotid` int(11) default NULL,
`itemid` int(11) default NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

-- Table "botleader" DDL

CREATE TABLE `botleader` (
`botid` int(11) NOT NULL default '0',
`leaderid` int(11) default NULL,
PRIMARY KEY (`botid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

-----> Table "botsowners" DDL <-------- double check this one and the one below

CREATE TABLE `botsowners` (
`botnpctypeid` int(11) NOT NULL default '0',
`botleadercharacterid` int(11) NOT NULL default '0',
PRIMARY KEY (`botnpctypeid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

--- The table npc_types must have a new field:
ALTER TABLE npc_types add `isbot` tinyint(4) NOT NULL;

[KingMort]

I thought botsowners wasn't used anymore .. isn't it just "bots" now or ??

If not then somethings out of whack i have 188 records in bots and 161 in botsowners

At any rate i switched them to Innodb

[KingMort]

Just another update it seems like this crash is happening when people camp with goons in group or when multiple people camp at the same time

[KingMort]

It's worse than ever before at this time... 15 crashed zones in less than 24 hours.

King

[Ansley1]

Could this be due to players being charmed there by chance?

Alot of servers I've played on mobs will assist charmed pets. Just throwing that out there....randomly probably :D

[KingMort]

At this point in time, the crashing has gotten so bad that it's averaging about 30 crashes in a 24 hour period..

So at this point in time, i'm offering $$$ to whoever can fix this shit once and for all !!!

Please contact me for more details and how much $$$ i am willing to offer @ KingMortenson@yahoo.com

Thanks

King

[Angelox]

I take you're using Linux server?
Your problem is probably in your hardware, namely your hard disk.
In Windows, the drive gets fragmented. to a point where I had to buy a second party de-fragmenter ("PerfectDisk") and schedule daily de-frags.
Zones will start to crash with little fragmentation, so de-frag, regardless.
Swap drives get fragmented too.
Not sure what happens to Linux, but a Linux swap drive on the same hard disk will slow you down.
100+ static zones on your server will cause zone crashes if your hard disks are not properly tweaked, or are just too slow.
Maybe your hard disk is getting old and slow. Try adding a second hard disk and move your Linux swap partition there there (in Windows, just move the swap file to the new HD).

I've had these problems under both Windows and Linux and have solved them by tweaking the hard disks. Took me all these years to realize this.

[KingMort]

Yeah the nice thing about Renting is that they will replace stuff for you anytime you want.. Guess it's due time for me to do this..

[mmaker]

What's up with the nulls in the characters' names?

That question is a couple of months old now, but in case it helps with future debugging I presume they're defined as char[64] instead of char*, so gdb prints out the full 64 character array. Which could contain any old junk after the first null.