Zones Restarting and deleting corpses!

[cubber]

Trevius Edit: I moved these posts from the SoF Development thread into here. It was getting the other thread off-topic.

I noticed in the last changelog you reverted something because of issues with titanium. Was this issue causing zones to crash and restart? Over the last 2 rev's I have had random zone crashes, and even had one where I crashed after I died and my corpse poofed! I had to revert to backup from the previous day.

I just compiled the rev 346 as of today and have not had a chance to test.

[Andrew80k]

I noticed in the last changelog you reverted something because of issues with titanium. Was this issue causing zones to crash and restart? Over the last 2 rev's I have had random zone crashes, and even had one where I crashed after I died and my corpse poofed! I had to revert to backup from the previous day.

I just compiled the rev 346 as of today and have not had a chance to test.
These crashes are often the result of database errors. Check your logs to see if there is anything in them to cause this.

[cubber]

I did go through the logs and only saw messages that stated that the dynamic zone I was in restarted.

[cubber]

Over the last 3 days of svn revisions around 340 on I have been having issues with zone restarts. For instance I have been hunting in seb with my monk and his bot group consisting of

1 cleric
1 shamy
1 ranger
1 wizard
1 rogue

after about a half hour of play the zone will restart and my client will hang. I can log back in cause the server does not crash just the zone. When I log back in my character is in the same spot he was in when the zone crashed and I can not summon any bots becuase they are already in the zone. So I log out and log back in and I can summon bots. At this point it is just annoying. But if I die it crashes the zone, and the kicker is when I log back in I am in the same spot of the zone where I died and crashed cept my character is naked and the corpse is deleted! I have to revert to backups to get my character back.

Stuff from the logs when it happens:

[Debug] [WORLD__ZONE] [9] [cabeast] ZoneToZone request for Nuggethead current zone 106 req zone 89

[Debug] [WORLD__ZONE] [9] [cabeast] Processing ZTZ for egress from zone for client CHARNAME

[Debug] [WORLD__ZONE] Found a zone already booted for CHARNAME

[Debug] [WORLD__ZONE] [11] [sebilis] ZoneToZone request for CHARNAME current zone 106 req zone 89

[Debug] [WORLD__ZONE] [11] [sebilis] Processing ZTZ for ingress to zone for client CHARNAME

[Debug] [WORLD__CLIENT] New connection from 192.168.42.6:47274
[Debug] [NET__IDENT_TRACE] 192.168.42.6:47274: First opcode 0x4dd0 did not match expected 0x2792
[Debug] [NET__IDENT_TRACE] 192.168.42.6:47274: Tried patch 6.2_world, and it did not match.
[Debug] [NET__IDENT_TRACE] 192.168.42.6:47274: First opcode 0x4dd0 did not match expected 0x2ec9
[Debug] [NET__IDENT_TRACE] 192.168.42.6:47274: Tried patch 6.2_zone, and it did not match.
[Debug] [NET__IDENT_TRACE] 192.168.42.6:47274: First opcode matched 0x4dd0 and length matched 464
[Debug] [NET__IDENTIFY] Identified stream 192.168.42.6:47274 with signature Titanium_world
[Debug] [WORLD__CLIENT] Checking inbound connection 192.168.42.6 against BannedIPs table
[Debug] [WORLD__CLIENT] Connection 192.168.42.6 PASSED banned IPs check. Processing connection.
[Debug] [WORLD__CLIENT] ACCNTNAME: Logged in. Mode=(Zoning)
[Debug] [WORLD__CLIENT] ACCNTNAME: MiniLogin Account #1
[Debug] [WORLD__CLIENT] ACCNTNAME: Telling client to continue session.
Unable to get group id, char not found!
[Debug] [WORLD__CLIENT] ACCTNAME: Zoning to sebilis (89)
[Debug] [WORLD__CLIENT] ACCTNAME: Sending client to zone sebilis (89) at publicipaddress:7010
[Debug] [WORLD__ZONE] [11] [sebilis] Setting to 'sebilis' (89)
[Debug] [WORLD__CLIENT] ACCTNAME: Client disconnected (not active in process)
[Debug] [WORLD__ZONELIST] Removing zoneserver #11 at :7010
[Debug] [WORLD__ZONELIST] Hold Zones mode is ON - rebooting lost zone
[Debug] [WORLD__LAUNCH] zone: dynamic_10 reported state STOPPED (1 starts)
[Debug] [WORLD__LAUNCH] zone: dynamic_10 reported state STARTED (2 starts)
[Debug] [WORLD__ZONE] New TCP connection from 127.0.0.1:43107
[Debug] [WORLD__CONSOLE] New zoneserver #14 from 127.0.0.1:43107
[Debug] [WORLD__ZONE] [14] Zone started with name dynamic_10 by launcher zone
[Debug] [WORLD__ZONE] [14] Auto zone port configuration. Telling zone to use port 7013


this also was inside a crashlog file in my servers root directory.

Wed Feb 18 23:01:02 EST 2009 World crashed.


I have tried 3 revisions so far and all give me the same issue. I have also let a character sit pretty much idle in bazaar for 10 hrs with a cleric and shamy bot up with no crash. Crashes seem to happen always in combat, and on character death. Possibly bot death as well not sure though since I never can tell if a bot died and it caused a crash. I will test this.

[cubber]

Ok the problem seems to be related to dying or bots dying. I just summoned a ranger bot and sicked it on a jugg in seb and as soon as the bot died the zone crashed. Then I logged back in and just ran my monk right at a jugg in seb and attacked it and let it kill me, and it hung up.

Of course when I logged back in I was still in seb dead but with no corpse on the ground.

[cubber]

Please look at this thread I started in the linux forums:

http://www.eqemulator.net/forums/showthread.php?t=27523

it explains more about the issue I am having. It is 100% reproduceable.

The problem has to do with death, if a character or bot dies it crashes the zone and deletes the characters corpse. When the character dies it almost looks like the titanium client is trying to use the newer death window meathod because a text message pops up in the center of my screen telling me what NPC killed me, but I get no confirmation box popup. Then the screen just lags and finally boots me out.

pulling down svn rev 349 now, gonna build and test.

[cubber]

The last svn rev I tried and this was happening on was rev 346, pulling down 349 now and going to test. I restored my database from this mornings backup.

[cubber]

Ok as of rev 349 here is the world log file when I log in a character right when some errors start:

[Error] Starting Log: logs/eqemu_error_world.log
[Error] Inventory::_PutItem: Invalid slot_id specified (2181)
[Error] Warning: Invalid slot_id for item in inventory: name=Character, acctid=1, item_id=6323, slot_id=2181
[Error] Inventory::_PutItem: Invalid slot_id specified (2182)
[Error] Warning: Invalid slot_id for item in inventory: name=Character, acctid=1, item_id=1263, slot_id=2182
[Error] Inventory::_PutItem: Invalid slot_id specified (2189)
[Error] Warning: Invalid slot_id for item in inventory: name=Character, acctid=1, item_id=12738, slot_id=2189
[Error] Inventory::_PutItem: Invalid slot_id specified (2190)
[Error] Warning: Invalid slot_id for item in inventory: name=Character, acctid=1, item_id=28074, slot_id=2190
[Debug] Unable to convert EQ opcode 0x6a5f to an Application opcode.
[Debug] [WORLD__CLIENT_ERR] Account: Received unknown EQApplicationPacket
[Debug] [WORLD__CLIENT_ERR] [OpCode OP_Unknown (0x6a5f) Size=8]
[Debug] [WORLD__CLIENT_ERR] 0000: 49 35 22 2f f0 35 22 2f - | I5"/.5"/
[Debug] Unable to convert EQ opcode 0x6a5f to an Application opcode.


That is when I log in my normal character that I have been testing with this whole time. I created a new test character to test this until it is fixed, as I hate restoring my play character from backup each time:

Here is the output of a new created character:

[Debug] [WORLD__CLIENT_ERR] ACCOUNT: Received unknown EQApplicationPacket
[Debug] [WORLD__CLIENT_ERR] [OpCode OP_FloatListThing (0x6a1b) Size=18]
[Debug] [WORLD__CLIENT_ERR] 0000: e1 1d d1 c3 ea 94 2b 43 - d5 97 0d 42 02 3b 1b 70 | ......+C...B.;.p
[Debug] [WORLD__CLIENT_ERR] 0016: 0e e1 - | ..
[Debug] [WORLD__CLIENT] ACCOUNT: Name approval request. Name=Gonnadiealot, race=Iksar, class=Monk
[Debug] [WORLD__CLIENT] ACCOUNT: Character creation request from LS#0 (192.168.42.6:55346) :
[Debug] [WORLD__CLIENT] ACCOUNT: Name: Gonnadiealot
[Debug] [WORLD__CLIENT] ACCOUNT: Race: 128 Class: 7 Gender: 0 Deity: 203 Start zone: 12
[Debug] [WORLD__CLIENT] ACCOUNT: STR STA AGI DEX WIS INT CHA Total
[Debug] [WORLD__CLIENT] ACCOUNT: 85 85 100 95 80 75 55 575
[Debug] [WORLD__CLIENT] ACCOUNT: Face: 1 Eye colors: 8 8
[Debug] [WORLD__CLIENT] ACCOUNT: Hairstyle: 255 Haircolor: 255
[Debug] [WORLD__CLIENT] ACCOUNT: Beard: 255 Beardcolor: 255
[Debug] [WORLD__CLIENT] Validating char creation info...
[Debug] [WORLD__CLIENT] Found 0 errors in character creation request
[Status] Start zone query: SELECT x,y,z,zone_id,bind_id FROM start_zones WHERE player_choice=12 AND player_class=7 AND player_deity=203 AND player_race=128

[Status] Found starting location in start_zones
[Debug] [WORLD__CLIENT] ACCOUNT: Current location: cabeast -50.00, 421.00, 6.75
[Debug] [WORLD__CLIENT] ACCOUNT: Bind location: cabeast -50.00, 421.00, 6.75
[Debug] [WORLD__CLIENT] ACCOUNT: Character creation successful: Gonnadiealot
[Error] Inventory::_PutItem: Invalid slot_id specified (2181)
[Error] Warning: Invalid slot_id for item in inventory: name=ORIGINAL CHARACTER NOT LOGGED IN, acctid=1, item_id=6323, slot_id=2181
[Error] Inventory::_PutItem: Invalid slot_id specified (2182)
[Error] Warning: Invalid slot_id for item in inventory: name=ORIGINAL CHARACTER NOT LOGGED IN, acctid=1, item_id=1263, slot_id=2182
[Error] Inventory::_PutItem: Invalid slot_id specified (2189)
[Error] Warning: Invalid slot_id for item in inventory: name=ORIGINAL CHARACTER NOT LOGGED IN, acctid=1, item_id=12738, slot_id=2189
[Error] Inventory::_PutItem: Invalid slot_id specified (2190)
[Error] Warning: Invalid slot_id for item in inventory: name=ORIGINAL CHARACTER NOT LOGGED IN, acctid=1, item_id=28074, slot_id=2190


The same thing happend after I made the new guy and got him dead. I recieved a message telling me who killed me in the center of the screen, all my UI windows closed except my chat windows, then after a few I was booted out. When I returned my guy was back where I died with no corpse.

A screenshot for you:

http://img18.imageshack.us/img18/927/screenshot1bn4.th.png (http://img18.imageshack.us/my.php?image=screenshot1bn4.png)

When I get to that screen its either wait for it to boot me out or alt + tab to my terminal wndow and ctrl+C to kill it then restart it.

[cubber]

Same thing happens when I test using the SoF client.

[cubber]

I found out what these are:

[Error] Inventory::_PutItem: Invalid slot_id specified (2181)
[Error] Warning: Invalid slot_id for item in inventory: name=ORIGINAL CHARACTER NOT LOGGED IN, acctid=1, item_id=6323, slot_id=2181
[Error] Inventory::_PutItem: Invalid slot_id specified (2182)
[Error] Warning: Invalid slot_id for item in inventory: name=ORIGINAL CHARACTER NOT LOGGED IN, acctid=1, item_id=1263, slot_id=2182
[Error] Inventory::_PutItem: Invalid slot_id specified (2189)
[Error] Warning: Invalid slot_id for item in inventory: name=ORIGINAL CHARACTER NOT LOGGED IN, acctid=1, item_id=12738, slot_id=2189
[Error] Inventory::_PutItem: Invalid slot_id specified (2190)
[Error] Warning: Invalid slot_id for item in inventory: name=ORIGINAL CHARACTER NOT LOGGED IN, acctid=1, item_id=28074, slot_id=2190

They are errors for items in my main characters bank. The errors seem to be triggered each time my main account is logged in. The items are:

Ice crystal staff
Robe of the foci
Teir' dal Sooth dol
Tambourine of Rituals

Nothing out of the ordinary just sitting in the bank. Not sure why they are spewing errors like that but they are unrealated to this issue.

[cubber]

As of rev 349 it does not crash the whole server just the zone I or a bot dies in. I just lost a bot on my main character which crashed the zone he was in and booted him, but my GM character was on an different pc just sitting in a different zone and was unaffected.

[trevius]

Since this is caused when a bot dies, maybe the problem was started by one of the bot changes recently. The only thing I can think of from SoF that might possibly be related is the addition of the Drakkin race array so that $race can recognize them. The other race issue I was working on for Drakkin to be able to equip items and gain stats was completely backed out, because even though it did correct the issue for Drakkin, all other races could then not gain anything by using items. I am still trying to figure out why it caused that and resolve it so all can play together nicely :P

I don't know exactly what all was added for bots recently. AX always forgets to make a comment when he does commits. Though, at least the changelog is normally updated. But, maybe that box you said you were seeing at the time of the crash was for the new bot corpse summon change? I don't really know, but sounds like a possibility.

[trevius]

I don't normally like to write things off unless I am 100% sure, but since your problem is caused by a bot dying, then I think it is probably a bot related issue. I haven't seen anything like this or heard any reports of anything like this on my server, but I also don't run bots. There have been alot of bot changes too lately, so that is just as likely to be the cause, if not more.

Maybe keep rolling back revisions 1 by 1 until you get to the first one that doesn't cause that crash. Then you know it was the one right after that that started it.

[cubber]

I will do some more troubleshooting over the next few days. This issue does not just happen when a bot dies as I have stated before and in the screenshot. If I die with a normal PC character with no bots up it happens and the corpse vanishes from the world.


Gonna pulling down the latest clean sources first and building without bots. I will report back.

[cubber]

Ok with clean svn sources rev 350 no bot support included the crash still happens when a character dies. However now I don't even see the character's corpse, the zone just drops and the character can still run around the zone until I /q out. But I cannot interact with anything.

here is the output of eqemu_debug_world.log when it happens:

15619 [02.20. - 08:34:16] [WORLD__CLIENT] New connection from 74.41.185.190:33336
15619 [02.20. - 08:34:16] [NET__IDENT_TRACE] 74.41.185.190:33336: First opcode 0x4dd0 did not match expected 0x2792
15619 [02.20. - 08:34:16] [NET__IDENT_TRACE] 74.41.185.190:33336: Tried patch 6.2_world, and it did not match.
15619 [02.20. - 08:34:16] [NET__IDENT_TRACE] 74.41.185.190:33336: First opcode 0x4dd0 did not match expected 0x2ec9
15619 [02.20. - 08:34:16] [NET__IDENT_TRACE] 74.41.185.190:33336: Tried patch 6.2_zone, and it did not match.
15619 [02.20. - 08:34:16] [NET__IDENT_TRACE] 74.41.185.190:33336: First opcode matched 0x4dd0 and length matched 464
15619 [02.20. - 08:34:16] [NET__IDENTIFY] Identified stream 74.41.185.190:33336 with signature Titanium_world
15619 [02.20. - 08:34:16] [WORLD__CLIENT] Checking inbound connection 74.41.185.190 against BannedIPs table
15619 [02.20. - 08:34:16] [WORLD__CLIENT] Connection 74.41.185.190 PASSED banned IPs check. Processing connection.
15619 [02.20. - 08:34:16] [WORLD__CLIENT] myaccount: Logged in. Mode=(CharSel)
15619 [02.20. - 08:34:16] [WORLD__CLIENT] myaccount: MiniLogin Account #1
15619 [02.20. - 08:35:06] [WORLD__CLIENT] myaccount: Attempting autobootup of tutorialb (189)
15619 [02.20. - 08:35:07] [WORLD__ZONE] [11] Setting to 'tutorialb' (189)
15619 [02.20. - 08:35:07] [WORLD__CLIENT] myaccount: Entering zone tutorialb (189)
15619 [02.20. - 08:35:07] [WORLD__ZONE] [11] [tutorialb] Broadcasting a world time update
15619 [02.20. - 08:35:07] [WORLD__ZONE] [11] [tutorialb] Setting to 'tutorialb' (189)
15619 [02.20. - 08:35:07] [WORLD__CLIENT] myaccount: Sending client to zone tutorialb (189) at purpletoad.no-ip.org:7010
15619 [02.20. - 08:35:07] [WORLD__CLIENT] myaccount: Client disconnected (not active in process)
15619 [02.20. - 08:38:27] [WORLD__ZONELIST] Removing zoneserver #11 at :7010
15619 [02.20. - 08:38:27] [WORLD__ZONELIST] Hold Zones mode is ON - rebooting lost zone
15619 [02.20. - 08:38:27] [WORLD__LAUNCH] zone: dynamic_10 reported state STOPPED (1 starts)
15619 [02.20. - 08:38:39] [WORLD__LAUNCH] zone: dynamic_10 reported state STARTED (2 starts)
15619 [02.20. - 08:38:41] [WORLD__ZONE] New TCP connection from 127.0.0.1:50310
15619 [02.20. - 08:38:41] [WORLD__CONSOLE] New zoneserver #14 from 127.0.0.1:50310
15619 [02.20. - 08:38:41] [WORLD__ZONE] [14] Zone started with name dynamic_10 by launcher zone
15619 [02.20. - 08:38:41] [WORLD__ZONE] [14] Auto zone port configuration. Telling zone to use port 7013


Note I am booting 10 dynamic zones and 3 statics (pok, pot, bazaar) the statics are using dynamic ports.

edit: Just tested again and I did actually die the second time and see my corpse hit the floor, got the message saying who killed me and the zone crashed. So I guess it can crash before or after your body hits the floor. Note that if I actually see my corpse when the zone crashes I cannot /q out I need to kill the process or wait for it to close it self out to login.

[cubber]

Ok I think I found my problem. I was able to die properly and actually zone after dying, then I was able to retrieve my corpse!

Seems the emulator does not like to be compiled with the following CFLAGS changes to each makefile:

-fomit-frame-pointer -fforce-addr

I was using these in my make files along with -march=athlon-xp

Once I removed the -fomit-frame-pointer and -fforce-addr and recompiled with out bots enabled all worked properlly. Gonna recompile now with bots enabled and see what happens.

Should I just be changing the march= in the make files or are there other safe optimizations I can use?

I have tried adding -O2 to each one or changing the ones that are -O to -O2 with out issues.

[cubber]

This still happens with bots enabled when a bot dies. But not when a PC dies. The only thing in my make files is -march=athlon-xp where there is a -march=i686 in the makefile. I have been using this since before 0.7.0 with no issues. Will like to this on the bot thread.

[cubber]

More testing more notes:

Seems to be a lot more stable now but here is what I have found so far.

PC death = no crash

Bot death = sometimes crash, sometimes they will crash the zone other times they wont. I have been just killing a bot one at a time, sometimes the first death does it sometimes it takes 2 or 3 bot deaths to crash the zone. I summon them one at a time and just send them at Grand Master Glox in East Cabilis then FD until they are dead. If I can get stand up and still do things like con npc's, use merchants, cast spells etc then I am good. But if I can't do anything then the zone crashed and I gotta kill the game.

[KingMort]

oof sounds like you have a binary issue bud..

[cubber]

This issue is fixed in Rev 355, seems it was an issue related to the bot code.