Restarting Dynamic Zone After a Crash?

**trevius** · #1 05-01-2008, 07:24 AM

On a Windows server, if a zone crashes, it will send the message box to the server about the crash with the OK button on it. Every zone that crashes will send another box. For every zone that crashes, it won't be allowed to restart until you actually click the OK button on the box from the crash.

Since Linux doesn't popup a similar message, I was wondering if there is a way to restart crashed dynamic zones. I set 50 dynamics and 1 crashed earlier and it didn't restart. The EQEmulator Management panel lists 49 booted zones instead of 50. Does anyone know how to correct this? Maybe there is something I can add in my script to start the world server?

Another thing I have noticed differently about Linux so far is that characters seem to sometimes bounce about 1 time per second if they have been standing idle for a while. I am not sure if this is some kind of zone update or what, but the closest thing I saw on windows was 1 time per minute all NPCs would bounce. Maybe something is set wrong, or is that normal?

Last, I was wondering if anyone had any newer scripts for starting and stopping the world server on Linux other than the ones from PEQ in the "Yet Another Linux Guide" from the wiki. It seems like those are probably old, but maybe they are already pretty tweaked. I won't even pretend to understand them, I was just wondering if there was a more tweaked version out there or if it even matters at all.

Also, if there is possibly a way to have all log files wrap after they reach a certain size, that would be awesome! I am afraid mine will start filling up my hard drive fast. From reading the scripts, I am guessing that they delete all log files after you issue the stop script for the server and start fresh ones when you start it again. But, setting a limit on the size would help a ton.

Here is the start script I am currently using. Also, I am not sure if I need to change the word "test" in there to something else, but it seems to work as is I guess lol:

Code:

#!/bin/sh

ulimit -c 99999999

rm -f .zone_shutdown
rm -f .world_shutdown

set -x

LNAME="zone"     #launcher name
if [ "$1" = "test" ]; then
        LNAME="test"
fi

P=`pwd`
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$P"

#make sure we have a place to log
mkdir -p logs 2>&1 > /dev/null

#clear out old logs, if both are stopped
if [ ! -e .lock-zones -a ! -e .lock-world ] ; then
  for f in logs/eqemu_*.log
  do
        if [ "$f" = "logs/eqemu_commands_zone.log" ]; then
                continue;
        fi

        rm -f $f
  done
fi

#boot up world
if [ ! -e .lock-world ] ; then
        touch .lock-world
#       ./world 2>&1 > logs/world &
        ./persist_world 2>&1 > logs/world &
        # wait for shared memory to load
        sleep 15
fi

#start up the official launcher
if [ ! -e .lock-launcher ]; then
        touch .lock-launcher
        ./eqlaunch $LNAME 2>&1 > logs/launcher &
fi

And here is persist_world:

Code:

#!/bin/sh

ulimit -c 99999999

while true
do
        ./world "$@"
        if [ -r ".world_shutdown" ]; then
                exit 0
        fi

        echo `date` "World crashed." >>crashlog
        sleep 2
done

I am still learning, but I learn fairly quick, so please bear with me :P

Andrew80k · #2 05-01-2008, 11:57 AM

Trev,

I have one that I got out of the message boards from a couple of years ago that I've been working on. I don't have it with me, but I could put it out later. I've only just started moving to Linux in the last couple of days, so I just started working on it last night. I wanna update it some to actually act like a startup script from the distro. But if you want I can post what I'm using.

**AndMetal** · #3 05-01-2008, 02:12 PM

If I'm not mistaken, eqlaunch should automatically reload the zone if it crashes. At least, that has been my experience. If you manually load the zone using zone, then that's another story.

**trevius** · #4 05-01-2008, 05:11 PM

Sure, Andrew80k. I would love to see what you have setup. Only having 1 example makes it hard for me to understand what is happening, and what needs to be there or not.

I would be especially interested in something that wraps log files after they reach maybe 250k or so if that is at all possible. I have no need for a 500MB log file lol. Heck, I wouldn't even mind a script that turns off the log files completely.

**trevius** · #5 05-02-2008, 12:48 AM

What the heck are these core.<number> files that are filling my hard drive? I already have a couple GB worth after less than 1 day of having the server up!

**AndMetal** · #6 05-02-2008, 02:07 AM

Quote:

Originally Posted by trevius

I would be especially interested in something that wraps log files after they reach maybe 250k or so if that is at all possible. I have no need for a 500MB log file lol. Heck, I wouldn't even mind a script that turns off the log files completely.

The easiest way to not output a log file is to redirect to /dev/null. I'm not sure of the best way to do this for the zone files (maybe a symbolic link to /dev/null?), but here is an example for the world server log that is made in your startup script:
Change

Code:

./persist_world 2>&1 > logs/world &

to

Code:

./persist_world 2>&1 > /dev/null &

Quote:

Originally Posted by trevius

What the heck are these core.<number> files that are filling my hard drive? I already have a couple GB worth after less than 1 day of having the server up!

Those are basically debug files. They are enabled with the -pg switch in the makefiles for eqlaunch & zone. I made brief mention of it in the Wiki, but at the time I wrote that part, I didn't know that's what created them. Basically, change the lines in eqlaunch/makefile and zone/makefile that begin with COPTS to remove the -pg.

**trevius** · #7 05-02-2008, 02:47 AM

Weird, I know for sure that I removed them before my compile. I followed your wiki exactly. I've gone through it 2 times now (rebuilt 2 times), and I am going to do it again soon for another PC. I just checked the source I used and -pg is definitely removed. Not sure why it makes these core files, but I don't think it is good for my server load and the amount of data growing that fast is insane. I probably had 20 100MB files in less than 1 day.

**AndMetal** · #8 05-02-2008, 02:04 PM

Apparently I'm retarded and I have no idea what I'm talking about.

However, after searching around teh Interwebs a little bit, I remembered the core.whatever files have to do with the following line in the startup script:

Code:

ulimit -c 99999999

In my script, I just commented it out (with a #), and I no longer get the core files, which definitely eat up a lot of space relatively quickly.

Aramid · #9 05-02-2008, 07:31 PM

I may be wrong, but it seems my setup made core files after a crash. It seems they are like dump files used to debug the crash, but I could be completely wrong here, this is just a guess on my part.
I do have the unlimit -c 999999 in my startup as well, so I will remove it and see what happens....

**trevius** · #10 05-03-2008, 12:59 AM

Makes sense lol. Thanks AndMetal! And ya, they do show up after a crash. That normally wouldn't be a huge issue, but I was still running my zone resetters in all of my custom zones, so essentially it thought I had 20 zone crashes per day and kicked out a 100MB file each time.

I have no use for those files anyway. They are too huge for me to open them with an editor and I have no idea what I would grep for in them. I will definitely be commenting that out!

Thanks guys. This info is really helping me get things setup nicely.

**trevius** · #11 05-03-2008, 07:13 AM

Ok, looks like I am having another issue with hard drive space... For some reason it seems like my hard drive space is getting used by something I can't seem to locate the source of... The webmin tool shows my hard drive space used is up to about 10.5GBs, but when I right click on the File System folder in the File Browser, it reports only 7.2GBs used, which is what things were at before I turned the server on. It seems like every time I refresh the webmin main System Information page, it shows that my hard drive has less and less space left.

I just can't figure out where the extra data is going. I don't see any other files that could be causing it. Is there possibly files that would be outside of the File System folder? This folder includes my home directory, root, and all of the other folders that I am aware of. I have no clue where this hard drive space is going.

My main concern is because I built this Linux build as a temporary place to run my server from. So, I made it a dual boot from a windows box. I didn't have a ton of hard drive space left, so I just gave it 13GBs. I figured that would be enough, and it would if I wasn't running into these weird issues eating up my space lol. I will be moving it back to the server PC with 30GBs soon, but even then I will want to correct these problems before they get out of hand.

System hostname XXXX
Operating system Debian Linux 4.0
Webmin version 1.410
Time on system Sat May 3 06:04:16 2008
System uptime 2 days, 5 hours, 16 minutes
CPU load averages 21.29 (1 min) 17.78 (5 mins) 16.60 (15 mins)
Real memory 2.95 GB total, 1.02 GB used

Virtual memory 651.03 MB total, 64 kB used

Local disk space 13.14 GB total, 10.49 GB used

---------------------------------------------

World is connected to login server eqemulator.net:5998.
World is NOT locked. Lock World.

There are 50 zones booted, 1 launchers connected, and 52 players online.

**trevius** · #12 05-03-2008, 01:20 PM

Almost 1GB in 6hours. Not sure where it is going...

System hostname XXXX
Operating system Debian Linux 4.0
Webmin version 1.410
Time on system Sat May 3 12:12:10 2008
System uptime 2 days, 11 hours, 24 minutes
CPU load averages 23.73 (1 min) 21.77 (5 mins) 19.31 (15 mins)
Real memory 2.95 GB total, 1 GB used

Virtual memory 651.03 MB total, 68 kB used

Local disk space 13.14 GB total, 11.23 GB used

**AndMetal** · #13 05-03-2008, 01:44 PM

After doing some searching, I found this website that says you can run this command to search for large files:

Code:

find / -type f -size +100000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }'

Specifically, that will search for files larger than 100000 KB (~100 MB).

For me, this yielded the following:

Code:

/var/lib/mysql/ibdata1: 114M
/proc/kcore: 897M
/home/eqemu/source/EQEmu-0.7.0-1090/utils/defaults/logs/eqemu_debug_zone.log: 361M
/home/eqemu/source/EQEmu-0.7.0-1090/utils/defaults/logs/zone-dynamic_26.log: 185M
/home/eqemu/common/Maps.tar.gz: 302M
/home/wowemu/source/src/game/libmangosgame.a: 132M

Hope this helps.

Aramid · #14 05-03-2008, 03:20 PM

It's probably your log files, because the start program you are using erases them when you re-boot the server. Every time someone just logs into your server it adds a lot of info. There has to be a way to turn them off....

**trevius** · #15 05-03-2008, 05:04 PM

Well, my eqemu log files aren't very bad at all. They only get to be a couple hundred MB total per day. But, apparently mysql has decided to start some logs of it's own...

Code:

-rw-rw----  1 mysql adm      14523 2008-04-30 03:24 mysql-bin.000001
-rw-rw----  1 mysql adm     510697 2008-04-30 03:24 mysql-bin.000002
-rw-rw----  1 mysql adm        272 2008-04-30 03:24 mysql-bin.000003
-rw-rw----  1 mysql adm        538 2008-04-30 03:24 mysql-bin.000004
-rw-rw----  1 mysql adm        695 2008-04-30 03:24 mysql-bin.000005
-rw-rw----  1 mysql adm        117 2008-04-30 03:24 mysql-bin.000006
-rw-rw----  1 mysql adm  104883277 2008-04-30 04:20 mysql-bin.000007
-rw-rw----  1 mysql adm  104864605 2008-04-30 04:20 mysql-bin.000008
-rw-rw----  1 mysql adm  104868599 2008-04-30 04:20 mysql-bin.000009
-rw-rw----  1 mysql adm  104857658 2008-04-30 04:22 mysql-bin.000010
-rw-rw----  1 mysql adm  104858441 2008-04-30 04:26 mysql-bin.000011
-rw-rw----  1 mysql adm  104857668 2008-04-30 04:29 mysql-bin.000012
-rw-rw----  1 mysql adm   71238529 2008-04-30 06:04 mysql-bin.000013
-rw-rw----  1 mysql adm      58356 2008-04-30 18:41 mysql-bin.000014
-rw-rw----  1 mysql adm      74191 2008-05-01 00:36 mysql-bin.000015
-rw-rw----  1 mysql adm  104880860 2008-05-01 01:01 mysql-bin.000016
-rw-rw----  1 mysql adm  104880772 2008-05-01 01:02 mysql-bin.000017
-rw-rw----  1 mysql adm  104868271 2008-05-01 01:02 mysql-bin.000018
-rw-rw----  1 mysql adm  104870093 2008-05-01 01:02 mysql-bin.000019
-rw-rw----  1 mysql adm  104857810 2008-05-01 01:10 mysql-bin.000020
-rw-rw----  1 mysql adm  104858636 2008-05-01 01:14 mysql-bin.000021
-rw-rw----  1 mysql adm  104857681 2008-05-01 01:21 mysql-bin.000022
-rw-rw----  1 mysql adm  104888312 2008-05-01 01:44 mysql-bin.000023
-rw-rw----  1 mysql adm  104891014 2008-05-01 01:44 mysql-bin.000024
-rw-rw----  1 mysql adm  104886951 2008-05-01 01:45 mysql-bin.000025
-rw-rw----  1 mysql adm  104889375 2008-05-01 01:45 mysql-bin.000026
-rw-rw----  1 mysql adm  104857726 2008-05-01 01:52 mysql-bin.000027
-rw-rw----  1 mysql adm  104858523 2008-05-01 01:53 mysql-bin.000028
-rw-rw----  1 mysql adm  104857765 2008-05-01 01:57 mysql-bin.000029
-rw-rw----  1 mysql adm  104864791 2008-05-01 03:47 mysql-bin.000030
-rw-rw----  1 mysql adm  104872158 2008-05-01 05:17 mysql-bin.000031
-rw-rw----  1 mysql adm  104871022 2008-05-01 07:15 mysql-bin.000032
-rw-rw----  1 mysql adm  104858638 2008-05-01 08:52 mysql-bin.000033
-rw-rw----  1 mysql adm  104895455 2008-05-01 10:24 mysql-bin.000034
-rw-rw----  1 mysql adm  104884507 2008-05-01 12:12 mysql-bin.000035
-rw-rw----  1 mysql adm  104870085 2008-05-01 14:01 mysql-bin.000036
-rw-rw----  1 mysql adm  104881715 2008-05-01 15:43 mysql-bin.000037
-rw-rw----  1 mysql adm  104880952 2008-05-01 17:15 mysql-bin.000038
-rw-rw----  1 mysql adm  104882831 2008-05-01 19:04 mysql-bin.000039
-rw-rw----  1 mysql adm  104859016 2008-05-01 20:37 mysql-bin.000040
-rw-rw----  1 mysql adm  104876683 2008-05-01 22:05 mysql-bin.000041
-rw-rw----  1 mysql adm  104887706 2008-05-01 23:21 mysql-bin.000042
-rw-rw----  1 mysql adm  104882239 2008-05-02 00:57 mysql-bin.000043
-rw-rw----  1 mysql adm  104859634 2008-05-02 03:06 mysql-bin.000044
-rw-rw----  1 mysql adm  104857648 2008-05-02 05:28 mysql-bin.000045
-rw-rw----  1 mysql adm   85713663 2008-05-02 07:35 mysql-bin.000046
-rw-rw----  1 mysql adm  104876061 2008-05-02 09:43 mysql-bin.000047
-rw-rw----  1 mysql adm  104869046 2008-05-02 11:53 mysql-bin.000048
-rw-rw----  1 mysql adm  104860169 2008-05-02 14:21 mysql-bin.000049
-rw-rw----  1 mysql adm  104873747 2008-05-02 16:17 mysql-bin.000050
-rw-rw----  1 mysql adm  104880877 2008-05-02 17:37 mysql-bin.000051
-rw-rw----  1 mysql adm  104892336 2008-05-02 19:03 mysql-bin.000052
-rw-rw----  1 mysql adm  104879856 2008-05-02 20:14 mysql-bin.000053
-rw-rw----  1 mysql adm  104889066 2008-05-02 21:24 mysql-bin.000054
-rw-rw----  1 mysql adm  104878996 2008-05-02 22:32 mysql-bin.000055
-rw-rw----  1 mysql adm  104885167 2008-05-02 23:42 mysql-bin.000056
-rw-rw----  1 mysql adm  104877980 2008-05-03 00:59 mysql-bin.000057
-rw-rw----  1 mysql adm  104895987 2008-05-03 02:39 mysql-bin.000058
-rw-rw----  1 mysql adm  104862508 2008-05-03 05:17 mysql-bin.000059
-rw-rw----  1 mysql adm   89631985 2008-05-03 07:35 mysql-bin.000060
-rw-rw----  1 mysql adm  104895799 2008-05-03 09:38 mysql-bin.000061
-rw-rw----  1 mysql adm  104864798 2008-05-03 11:10 mysql-bin.000062
-rw-rw----  1 mysql adm  104880617 2008-05-03 12:23 mysql-bin.000063
-rw-rw----  1 mysql adm  104863114 2008-05-03 13:57 mysql-bin.000064
-rw-rw----  1 mysql adm  104893903 2008-05-03 15:09 mysql-bin.000065
-rw-rw----  1 mysql adm   59043841 2008-05-03 15:46 mysql-bin.000066

After some research, it looks like these files are all some sort of mysql backup. This is enabled in the my.cnf file. So, you can give temporary edit rights from root so you can open it in a text editor:

server:/etc/mysql# chmod go+rw my.cnf

Then, open the my.cnf file in a text editor and comment these lines by putting a # in front of them:

Code:

log_bin			= /var/log/mysql/mysql-bin.log
expire_logs_days	= 10
max_binlog_size         = 100M

Then save the file and don't forget the remove the rw rights you added before or mysql will have problems loading:

server:/etc/mysql# chmod go-rw my.cnf

I just tried all of this myself, but I haven't restarted mysql yet to see if it works, but I imagine it should. I will restart mysql the next time I reboot the server and I will report back if this doesn't work.