INFO: Core Files (core.####)
 

Hello,

I'm an experienced Linux programmer, and one of the major tools for developers are bug reports of any kind. One way to tell that a bug exsists is by having core files in your directory that you run your executables from.

A core file usually looks like this:
(Note the lines starting with ***)

The file format is: core.#### where the number is the process ID of the program that crashed.

If you note, they're fairly large sometimes. The reason why is because a core file is a dump of everything that the program was doing - from memory to function tasks. It's a snapshot of the entire instance of a program before it crashed.

Core files are useful because they help a developer find where a crash originated from. In order to find out this information, you have to use a certain tool commonly found on Linux systems called 'gdb'.

GDB stands for GNU Debugger, and it's a way to examine what a program was doing right when it crashed. Basically, you can usually find the exact part of the code that caused it to crash, however this isn't always the case.

The syntax for GDB is:
Note that you have to tell GDB what program you believe crashed. It won't always be the zone program, but the best way to figure it out is to check your log files. You can do this with the 'tail' command.

The above will show you the last 10 lines of your world log, should it be called 'world.log'. If you have shut down your server, then you should see something that says 'Got signal', followed by a number. If it looks like the log file of a server that's still running, there's a good chance it crashed.

You can also simply gdb --core=<file> and read the first few lines GDB has to say:

When you bring up GDB, it'll load all the symbols needed for it to figure out what went wrong. A tip here: KEEP YOUR SOURCE CODE. Also, make sure you debug a core for the current executable, not a previous compile of an executable. It just won't work.

When you get into GDB, the first command you want to run is 'bt'. This stands for 'backtrace'. It'll give you a list of procedures that were on the stack when the program crashed. This list is in order, the top most one, or Frame #0, was the last function to be called. The one under Frame #0, or Frame #1, is the function that called Frame #0. This goes on all the way down the list. The last function is the one that started it all, and is usually a system call.

Simply posting the backtrace list is helpful enough, but sometimes someone working on the code might need more information. It'd be a good idea to save your core file, along with the executable that generated it. You could do something like this to archive it:
Which will archive the two major files needed, or even better would be to also archive the source:
You can change the filenames to whatever you prefer.

A few other commands you can use while in GDB:

The 'frame' command sets the current frame to whatever number you provide. Lets say you want to investigate the top-most frame: 'frame 0'. This will output something similar to the top line on the backtrace.

You can get information on variables used in the function by using the 'info' command after you select a 'frame':

info args -- This shows the arguments passed to the function
info local -- This shows any local variables used inside the function.

Note: see 'help info' for more information

You can also 'print' a variable:

print tmp
print *tmp

Note that in the case of pointers, if you don't include a *, it'll simply print the memory address. If you want to see what's in the variable, include the *.

Hope this was helpful -- submit corrections please. :)

IMO, *this* is worthy of a sticky.

Thanks a lot for the guide.

Just make sure you compiled all the source with -ggdb switch so that the executable has full symbol tables.

It makes the executables larger, but it keeps enough information for a more useful debugging session.

If you run your zones/world stripped of symbol information, you can go recompile (assuming you've not made changes) with -ggdb and then use gdb on your new zone/world with that core. GDB will probably complain as follows, but should take it just fine:BTW, in general it drops core.pid for multithreaded applications (where pid is the process ID of the thread that cored) and just plain "core" for single threaded apps.

You can also see what application generated the core file, if you don't have gdb installed for some reason (not that the core file will do you much good, at least on that system):

Thanks for clarifying it all, Doodman. I've only ever used GDB in the MUD community, and a few smaller projects. :)

Core files will be truncated to the current limit, set by ulimit -c. You can also change this in /etc/limits on some systems.

NOTE: 0 means 0 bytes, NOT unlimited as on UNIX systems.