World crashes
 

For about the past month my server has been experiencing random world crashes. I tend to keep my server pretty recient with SVN and I think it may have been around the time the new bot code was starting to get worked in. I could have a few people playing on the server or just one and it could be stable for a few hours then everyone just goes LD and I see an entry in the crashlog.

I made a new server in a virtualbox on a different PC, using the same OS and all the same settings I normally use with the same DB and still get the issue. So I ruled out hardware.

The logs do not really tell me anything interesting but here is what I have found:

Note these logs are from the VM but appear the same on the live server.

crashlog:

eqemu_debug_5089.log

eqemu_debug_world.log
(only stuff around the time of the crash)

07-06 21-11-14 crash

07-06 21:15:31 crash


the last one was another character creation attempt. All of my users use the SoF client. These crashes also occur randomly when just playing. I don't have any logs of those at this time though.

If anyone would like to see any other logs please let me know. I am anxious to get this issue resolved.

One last bit, my PrivateLogin server console showing the world reconnects:

Thanks!

Do you have any core dumps from the crashes? If so, could you get a backtrace in gdb? Thanks.

Also, what kind of system is this running on: debian, fedora, etc.?

There are no core dumps in the eqemu directory. The os is Gentoo x86 on both systems. I actually have since wiped the live server and am in the process of redoing it so I am just running off the VM.

Also the user that the crashes happend to last night said it was only on character creation. He tried to create an iksar monk 3 x and it failed each time. He had a hydra account up and noticed a "world crashed" notice in yellow letters in his chat.

I just tried to create a new character on my GM account and got the same issue. I could create the character but as soon as I clicked enter world and it tried to put me in glooming deep it crashed the world. This was on a SoF client.

I also noticed that there is an option for start city for each character with the old home town and crescent reach. with crescent reach being the default. Now on a peq clone server crescent reach is not avaialable to non GM accounts.

Ok I just logged into my account I just tried to create the toon on. I had a console window open ssh'd into the server and rain tail -f world in the logs directory. This poped up as soon as I logged into my account:

and the character was not in my character list but it is present in the DB if I look under phpmyadmin. So are all of the characters that the other user tried to create.

Also as soon as I hit create character button the world immediatly crashes in the console.

Your DB will need to be cleaned up to remove the incomplete chars. I have no idea if the incomplete chars could be a cause for the crashes. They might simply be a result of the crashes.

You could run world under gdb and get a backtrace when it crashes. In your server dir, run the following. You type the portions in red.

At that point you can start up your zone servers and then log in and try to create a character. When the process crashes, it will drop you back to the (gdb) prompt.

Post the backtrace output here and I'll see if I can help you further.

I don't have GDB

gdb ./world
-bash: gdb: command not found


I have been deleting the "borked" characters after i attempt to create them. They are sticking in phpmyadmin I just go into the character_ table and remove them one by one.

As for the core dump I do have one just missed it. The file is huge and has just a bunch of characers in it with some words here and there. Want me to post it somewhere for download?

I also just updated my server to SVN Rev751. I tried to create a new toon, and it still has the same issue.

Note I am using PEQ rev703 as a DB with all the updates in the utils/sql/svn folder applied.

You can get a backtrace of the corefile using gdb:

Replace <corefilename> with the name of the core file. Save the output from the backtrace and post it here.

doh. no gdb. hrm, lemme get you information about how to install it.

this look like the right package:

try:

that might get it installed. I'm not a Gentoo expert, though, so perhaps someone who runs Gentoo could chime in here.

Gonna take a bit to compile gdb I will post the backtrace of the core file when it is done.

the file I am looking to use for the backtrace is called "core" and has a bunch of characters in it like @@@@ correct?

Yeah, that looks good. See if you can run gdb after that. If so, there's information in earlier posts to use gdb on the core file. No need to run world under gdb since you have core files.

here is more info on the gdb package I am installing

http://www.gentoo-portage.com/sys-devel/gdb

says gnu debugger so I figured it was correct. I am emerging it now.

File name is dependent on system configuration, but usually includes 'core' in the file name. The file is a binary image of the memory used by the program when it crashed, so it will not appear very human readable. You should be able to type 'file core' and it will tell you what the core file is from.

Ok gdb finished compiling here is the output:

Your world executable appears to be stripped. That removes a lot of debugging information. I would suggest you rebuild it and not strip it. Then get the backtrace from a fresh core dump.

From what I'm hearing, gentoo strips the executable by default. I don't know how to change that behavior. You should do some google searches or ask a gentoo expert to find out how to disable stripping the executable.

here is what I change in my makefiles

DFLAGS=-DDEBUG=5 -DEQDEBUG=2 -DSHAREMEM -DCATCH_CRASH -DNO_PIDLOG -DFIELD_ITEMS -DIRC -Di386 -DAPP_OPCODE_SIZE=2 -DEQBOTS


COPTS=$(WFLAGS) -march=native -O3 -pthread -pipe

I also remove the -g and -pg flags if they are there.

I take it you want me to change the DEQDEBUG=2 to something like DEQDEBUG=5?

You need the -g option in there to include debugging information.

Ok I will try recompiling with -g, then I will reproduce the crash and hit the new core file with gdb and post the results.

Very Interesting...

I wacked my svn source folder and just pulled down new sources. Ended up being rev752.

I then only changed the -march=i686 option in each makefile to -march=native and left everything else at default. I also did NOT enable bots.

I did not change any -O options or remove the -g flags like I normally do nor did I change -DEQDEBUG=5 to -DEQDEBUG=2.

When I went to log in and create a character the character created fine but I got an error saying that the zone was not available yet. It then just stuck the character in my list and I was then able to choose the character and login to Gloomingdeep.

So something in the compile options was screwing me up. Though I have used those settings for quite a while now without an issue. The only change now is that I am running gcc4.3.2 instead of gcc3.2.6 and using -march=native instead of -march=cpufamily_name.

I will have to try recompiling with bots and see if it still works.

If I were to venture a guess, I would suspect the -O3 as the culprit.

yah gonna leave them alone, and just keep everything default with the -march=native and try enabling bots.

does removing -g and changing -DEQDEBUG=2 give that much more of a performance boost over running the server with it enabled?

As far as the DEBUG=2, I would doubt it would have much effect since most of the debugging is configured at runtime now.

As to the -g, I'm not the right person to ask. My guess would be that it doesn't slow it down much.

Compiled with bots and all seems fine. Thanks for the assistance!