EQEmulator Forums - UCS and Queryserv: Segmentation Fault

EQEmulator Forums (https://www.eqemulator.org/forums/index.php)

- Support::Linux Servers (https://www.eqemulator.org/forums/forumdisplay.php?f=588)

- - UCS and Queryserv: Segmentation Fault (https://www.eqemulator.org/forums/showthread.php?t=42597)

UCS and Queryserv: Segmentation Fault

I recently updated mariadb via

Code:

apt-get dist-upgrade

, and apparently something broken with the mysql connection to my server. When running

Code:

./server_start.sh

, it appears to launch fine:

Code:

[Status] Loading items...

[Status] Loading factions...

[Status] Loading loot...

[Status] Loading skill caps...

[Status] Loading spells...

[Status] Loading base data...

Server started - use server_status.sh to check server status

XXX:/home/eqemu/server# 

Akka's Linux Server Launcher

Zones to launch: 30

But the status is:

Code:

World: UP Zones: (30/30) UCS: DOWN Queryserv: DOWN

There are no "world" logs in the logs/ folder, only zone logs in the "logs/zone" folder.

If I try to run

Code:

sudo ./ucs

I get:

Code:

[UCS Server] Starting EQEmu Universal Chat Server.

[UCS Server] Connecting to MySQL...

Segmentation fault

And the same with

Code:

sudo ./queryserv

Code:

[QS Server] Starting EQEmu QueryServ.

[QS Server] Connecting to MySQL...

Segmentation fault

I tried rebuilding and reinstalling both mysql-server and the EQEmulator Linux server (./install.sh). I updated the eqemu_config.json file like I had previously done:

Code:

{

   "server" : {

      "chatserver" : {

         "host" : "",

         "port" : "7778"

      },

      "database" : {

         "db" : "eqemudb",

         "host" : "localhost",

         "password" : "<PASSWORD>",

         "port" : "3306",

         "username" : "eqemu"

      },

      "mailserver" : {

         "host" : "",

         "port" : "7778"

      },

      "qsdatabase" : {

         "db" : "eqemudb",

         "host" : "localhost",

         "password" : "<PASSWORD>",

         "port" : "3306",

         "username" : "eqemu"

      },

      "webinterface" : {

         "port" : "9081"

      },

      "world" : {

         "http" : {

            "enabled" : "true",

            "mimefile" : "mime.types",

            "port" : "9080"

         },

         "key" : "talDG7ZRhjnVqnqOJCaUF3mneBIGMvm",

         "loginserver1" : {

            "account" : "",

            "host" : "login.eqemulator.net",

            "legacy" : "1",

            "password" : "",

            "port" : "5998"

         },

         "loginserver2" : {

            "account" : "",

            "host" : "192.168.2.50",

            "password" : "",

            "port" : "5998"

         },

         "longname" : "<Server Name>",

         "shortname" : "<Server Name>",

         "tcp" : {

            "ip" : "192.168.2.50",

            "port" : "9001"

         },

         "telnet" : {

            "enabled" : "true",

            "ip" : "0.0.0.0",

            "port" : "9000"

         }

      },

      "zones" : {

         "defaultstatus" : "0",

         "ports" : {

            "high" : "7400",

            "low" : "7000"

         }

      }

   }

}

However, the problem persists...

I'm quite confused what to do next... the server was up and running fine yesterday before the update. Anyone have any ideas how to troubleshoot this?

Some more info using gdb:

Code:

sudo gdb ucs

GNU gdb (Debian 7.12-6) 7.12.0.20161007-git

Copyright (C) 2016 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from ucs...done.

(gdb) run

Starting program: /home/eqemu/server/ucs 

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

[UCS Server] Starting EQEmu Universal Chat Server.

[UCS Server] Connecting to MySQL...



Program received signal SIGSEGV, Segmentation fault.

strlen () at ../sysdeps/x86_64/strlen.S:106

106     ../sysdeps/x86_64/strlen.S: No such file or directory.

(gdb) backtrace

#0  strlen () at ../sysdeps/x86_64/strlen.S:106

#1  0x00007ffff5f31da8 in _IO_vfprintf_internal (s=s@entry=0x7fffffffe180, format=<optimized out>, format@entry=0x555555882580 "Starting File Log 'logs/%s_%i.log'", 

    ap=ap@entry=0x7fffffffe2e8) at vfprintf.c:1637

#2  0x00007ffff5f58e89 in _IO_vsnprintf (string=0x7fffffffe270 "Starting File Log 'logs/\200\037", string@entry=0x0, maxlen=<optimized out>, maxlen@entry=0, 

    format=format@entry=0x555555882580 "Starting File Log 'logs/%s_%i.log'", args=args@entry=0x7fffffffe2e8) at vsnprintf.c:114

#3  0x00005555555db136 in vStringFormat[abi:cxx11](char const*, __va_list_tag*) (format=0x555555882580 "Starting File Log 'logs/%s_%i.log'", args=args@entry=0x7fffffffe348)

    at /home/eqemu/server_source/Server/common/string_util.cpp:47

#4  0x00005555555c9766 in EQEmuLogSys::Out (this=this@entry=0x555555868e20 <LogSys>, debug_level=debug_level@entry=Logs::General, log_category=log_category@entry=25, 

    message="Starting File Log 'logs/%s_%i.log'") at /home/eqemu/server_source/Server/common/eqemu_logsys.cpp:378

#5  0x00005555555c9a04 in EQEmuLogSys::StartFileLogs (this=0x555555868e20 <LogSys>, log_name=...) at /home/eqemu/server_source/Server/common/eqemu_logsys.cpp:493

#6  0x0000555555592d4f in main () at /home/eqemu/server_source/Server/ucs/ucs.cpp:101

Code:

sudo gdb queryserv

GNU gdb (Debian 7.12-6) 7.12.0.20161007-git

Copyright (C) 2016 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".

Type "show configuration" for configuration details.

For bug reporting instructions, please see:

<http://www.gnu.org/software/gdb/bugs/>.

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word"...

Reading symbols from queryserv...done.

(gdb) run

Starting program: /home/eqemu/server/queryserv 

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

[QS Server] Starting EQEmu QueryServ.

[QS Server] Connecting to MySQL...



Program received signal SIGSEGV, Segmentation fault.

strlen () at ../sysdeps/x86_64/strlen.S:106

106     ../sysdeps/x86_64/strlen.S: No such file or directory.

(gdb) backtrace

#0  strlen () at ../sysdeps/x86_64/strlen.S:106

#1  0x00007ffff5f31da8 in _IO_vfprintf_internal (s=s@entry=0x7fffffffe1e0, format=<optimized out>, format@entry=0x555555812c60 "Starting File Log 'logs/%s_%i.log'", 

    ap=ap@entry=0x7fffffffe348) at vfprintf.c:1637

#2  0x00007ffff5f58e89 in _IO_vsnprintf (string=0x7fffffffe2d0 "Starting File Log 'logs/\200\037", string@entry=0x0, maxlen=<optimized out>, maxlen@entry=0, 

    format=format@entry=0x555555812c60 "Starting File Log 'logs/%s_%i.log'", args=args@entry=0x7fffffffe348) at vsnprintf.c:114

#3  0x000055555558d3d6 in vStringFormat[abi:cxx11](char const*, __va_list_tag*) (format=0x555555812c60 "Starting File Log 'logs/%s_%i.log'", args=args@entry=0x7fffffffe3a8)

    at /home/eqemu/server_source/Server/common/string_util.cpp:47

#4  0x000055555558b736 in EQEmuLogSys::Out (this=this@entry=0x5555557fb180 <LogSys>, debug_level=debug_level@entry=Logs::General, log_category=log_category@entry=25, 

    message="Starting File Log 'logs/%s_%i.log'") at /home/eqemu/server_source/Server/common/eqemu_logsys.cpp:378

#5  0x000055555558b9d4 in EQEmuLogSys::StartFileLogs (this=0x5555557fb180 <LogSys>, log_name=...) at /home/eqemu/server_source/Server/common/eqemu_logsys.cpp:493

#6  0x0000555555579f20 in main () at /home/eqemu/server_source/Server/queryserv/queryserv.cpp:79

...not sure if it adds anything new to help troubleshoot though?

So I've now completely wiped and reinstalled mysql-server:

Code:

sudo apt purge mysql-server mysql-client mysql-common

sudo apt autoremove

sudo mv -iv /var/lib/mysql /var/tmp/mysql-backup

sudo rm -rf /var/lib/mysql*

sudo mv /etc/mysql /var/tmp/etc_mysql_backup

sudo rm -rf /etc/mysql



sudo apt update

sudo apt install mysql-server

sudo mysqld --initialize

sudo chown -R mysql:mysql /var/lib/mysql

sudo systemctl start mariadb.service

sudo systemctl enable mariadb.service

sudo /usr/bin/mysql_secure_installation

I also deleted the /home/eqemu folder, then rebuilt the server with the install.sh script. As far as I can tell, the database is fine and the install went well.

I can run shared_memory, world, eqlaunch, and zone without any errors. It's still only ucs and queryserv that give the Segmentation fault errors.

If I launch everything except ucs and queryserv, I am able to get to the character creation screen. Trying to zone in, however, gives me the error:

Code:

[World Server] No zoneserver available to boot up.

[World Server] Attempting autobootup of tutorialb (189:0)

...so I assume ucs at least it critical to run, is that correct?

Some more information from journalctl -xe:

Code:

kernel: traps: ucs[1394] general protection ip:7fab0f107676 sp:7fffed51fd38 error:0

kernel:  in libc-2.24.so[7fab0f087000+195000]

kernel: traps: queryserv[1398] general protection ip:7f8219af8676 sp:7ffef156aec8 error:0

kernel:  in libc-2.24.so[7f8219a78000+195000]

And Valgrind:

Code:

==6770== Command: ./ucs

==6770== 

[UCS Server] Starting EQEmu Universal Chat Server.

[UCS Server] Connecting to MySQL...

==6770== Invalid read of size 1

==6770==    at 0x4C2F1E2: strlen (vg_replace_strmem.c:460)

==6770==    by 0x69D2DA7: vfprintf (vfprintf.c:1637)

==6770==    by 0x69F9E88: vsnprintf (vsnprintf.c:114)

==6770==    by 0x18F135: vStringFormat[abi:cxx11](char const*, __va_list_tag*) (string_util.cpp:47)

==6770==    by 0x17D765: EQEmuLogSys::Out(Logs::DebugLevel, unsigned short, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ...) (eqemu_logsys.cpp:378)

==6770==    by 0x17DA03: EQEmuLogSys::StartFileLogs(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (eqemu_logsys.cpp:493)

==6770==    by 0x146D4E: main (ucs.cpp:101)

==6770==  Address 0x100010001000100 is not stack'd, malloc'd or (recently) free'd

==6770== 

==6770== 

==6770== Process terminating with default action of signal 11 (SIGSEGV)

==6770==  General Protection Fault

==6770==    at 0x4C2F1E2: strlen (vg_replace_strmem.c:460)

==6770==    by 0x69D2DA7: vfprintf (vfprintf.c:1637)

==6770==    by 0x69F9E88: vsnprintf (vsnprintf.c:114)

==6770==    by 0x18F135: vStringFormat[abi:cxx11](char const*, __va_list_tag*) (string_util.cpp:47)

==6770==    by 0x17D765: EQEmuLogSys::Out(Logs::DebugLevel, unsigned short, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ...) (eqemu_logsys.cpp:378)

==6770==    by 0x17DA03: EQEmuLogSys::StartFileLogs(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (eqemu_logsys.cpp:493)

==6770==    by 0x146D4E: main (ucs.cpp:101)

==6770== 

==6770== HEAP SUMMARY:

==6770==     in use at exit: 125,720 bytes in 223 blocks

==6770==   total heap usage: 511 allocs, 288 frees, 275,468 bytes allocated

==6770== 

==6770== LEAK SUMMARY:

==6770==    definitely lost: 0 bytes in 0 blocks

==6770==    indirectly lost: 0 bytes in 0 blocks

==6770==      possibly lost: 0 bytes in 0 blocks

==6770==    still reachable: 125,720 bytes in 223 blocks

==6770==                       of which reachable via heuristic:

==6770==                         length64           : 34,160 bytes in 17 blocks

==6770==         suppressed: 0 bytes in 0 blocks

==6770== Rerun with --leak-check=full to see details of leaked memory

==6770== 

==6770== For counts of detected and suppressed errors, rerun with: -v

==6770== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Segmentation fault

I haven't been ignoring this thread..I just don't have any linux experience...

For 'ucs' and 'queryserv' servers, they do not affect the server's ability to operate.

That logging error, however... That would definitely cause a problem.

I'll take a look at that call and see if I can find anything.

No problem, I am definitely out of my league on this one! UCS and Queryserv are still giving the segmentation fault, but I have been able to get the serving running without them for now. I essentially purged and reinstalled mysql-server and all of the dependencies in install.sh, deleted the old database and eqemu user in mysql, deleted and recreated the eqemu account, restarted my machine, and then reran install.sh to make the binaries. Now the server runs and I can connect to it, and everything on the client side seems to work fine. I'm not sure if there are bigger problems lurking underneath though, with logging for example, like you mention...

What happens if you change the host entries in your config to localhost?

Unfortunately, changing the host entries to "localhost" didn't seem to have an effect... still getting the Segmentation fault :/

In windows, "localhost" will always resolve into tcp/ip (127.0.0.1), but that is not always the case in linux, it can do crazy things, like trying to resolve into an IPv6 address, but either way, not sure why people are so adamant about using that infamous localhost. Using 127.0.0.1 in all the configs, will guarantee to rule out that cause, for anything local.

Debugging it done, the error is actually in eqemu_logsys.cpp in platform_file_name.c_str(). "ucs" is definitely in platform_file_name but somehow c_str is causing a segmentation fault. I am rusty on my C++ at the moment but I will get a resolution.

At some point platform_file_name is getting assigned the value of ucs, but at the time LogSys.StartFileLogs() is ran at ucs.cpp on line 101, that value does not appear to be there, at least on the surface. When we get to eqemu_logsys.cpp line 462 it throws a segmentation fault, my best guess is there is a null there. I am still reading on proper use of c_str but on the surface I guess there is a null there. There is a check for empty but it seems to blow past that.

I'm pretty sure my_string.c_str() should return a nullstring pointer even if my_string.empty() is true.

Variadics on the other hand...

If you pass a nullptr argument, add too many formatizers or too few arguments, it will crash.

Right, I did have a little while to read more about c_str(). Apparently it cannot be null, it can have a null pointer but std::string cannot actually ever be null, there should always be a value even if it is an empty string. Also from reading there can still be quite a few bad things happen to cause std::string to go haywire and I guess that is what is happening. It definitely gets assigned a value at some point because I see the value appear but later on something happens. I have yet to find what is happening. Maybe I can look at it again tomorrow.

I had some time this morning to look at it again. I never would had figured the bug was in this location.

In the directory for ucs and queryserver look at the database.cpp and you will notice my correction here. One thing is that the routine was overwriting the file log settings every iteration, because it always ran through every entry without stopping. So really the routine was not really working properly without this being adding. I am not getting the segmentation fault anymore and my log is being written so I guess it is working? I need someone with a higher pay grade to look over this change though.

Code:

        for (auto row = results.begin(); row != results.end(); ++row) {

                log_category = atoi(row[0]);

                

                if (log_category <= Logs::None || log_category >= Logs::MaxCategoryID) {

                        continue;

                }

                

                log_settings[log_category].log_to_console = atoi(row[2]);

                log_settings[log_category].log_to_file = atoi(row[3]);

Interesting, just seeing this thread for the first time. I'll put it on my list to check this one out tonight

First look this is related to newer GCC versions, investigating closer