Go Back   EQEmulator Home > EQEmulator Forums > Support > Support::Linux Servers

Support::Linux Servers Support forum for Linux EQEMu users.

Reply
 
Thread Tools Display Modes
  #1  
Old 09-09-2019, 06:45 AM
mushko
Fire Beetle
 
Join Date: Aug 2019
Posts: 12
Default UCS and Queryserv: Segmentation Fault

I recently updated mariadb via
Code:
apt-get dist-upgrade
, and apparently something broken with the mysql connection to my server. When running
Code:
./server_start.sh
, it appears to launch fine:

Code:
[Status] Loading items...
[Status] Loading factions...
[Status] Loading loot...
[Status] Loading skill caps...
[Status] Loading spells...
[Status] Loading base data...
Server started - use server_status.sh to check server status
XXX:/home/eqemu/server# 
Akka's Linux Server Launcher
Zones to launch: 30
But the status is:
Code:
World: UP Zones: (30/30) UCS: DOWN Queryserv: DOWN
There are no "world" logs in the logs/ folder, only zone logs in the "logs/zone" folder.

If I try to run
Code:
sudo ./ucs
I get:
Code:
[UCS Server] Starting EQEmu Universal Chat Server.
[UCS Server] Connecting to MySQL...
Segmentation fault
And the same with
Code:
sudo ./queryserv
:
Code:
[QS Server] Starting EQEmu QueryServ.
[QS Server] Connecting to MySQL...
Segmentation fault
I tried rebuilding and reinstalling both mysql-server and the EQEmulator Linux server (./install.sh). I updated the eqemu_config.json file like I had previously done:

Code:
{
   "server" : {
      "chatserver" : {
         "host" : "",
         "port" : "7778"
      },
      "database" : {
         "db" : "eqemudb",
         "host" : "localhost",
         "password" : "<PASSWORD>",
         "port" : "3306",
         "username" : "eqemu"
      },
      "mailserver" : {
         "host" : "",
         "port" : "7778"
      },
      "qsdatabase" : {
         "db" : "eqemudb",
         "host" : "localhost",
         "password" : "<PASSWORD>",
         "port" : "3306",
         "username" : "eqemu"
      },
      "webinterface" : {
         "port" : "9081"
      },
      "world" : {
         "http" : {
            "enabled" : "true",
            "mimefile" : "mime.types",
            "port" : "9080"
         },
         "key" : "talDG7ZRhjnVqnqOJCaUF3mneBIGMvm",
         "loginserver1" : {
            "account" : "",
            "host" : "login.eqemulator.net",
            "legacy" : "1",
            "password" : "",
            "port" : "5998"
         },
         "loginserver2" : {
            "account" : "",
            "host" : "192.168.2.50",
            "password" : "",
            "port" : "5998"
         },
         "longname" : "<Server Name>",
         "shortname" : "<Server Name>",
         "tcp" : {
            "ip" : "192.168.2.50",
            "port" : "9001"
         },
         "telnet" : {
            "enabled" : "true",
            "ip" : "0.0.0.0",
            "port" : "9000"
         }
      },
      "zones" : {
         "defaultstatus" : "0",
         "ports" : {
            "high" : "7400",
            "low" : "7000"
         }
      }
   }
}
However, the problem persists...

I'm quite confused what to do next... the server was up and running fine yesterday before the update. Anyone have any ideas how to troubleshoot this?
Reply With Quote
  #2  
Old 09-09-2019, 10:32 AM
mushko
Fire Beetle
 
Join Date: Aug 2019
Posts: 12
Default

Some more info using gdb:

Code:
sudo gdb ucs
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ucs...done.
(gdb) run
Starting program: /home/eqemu/server/ucs 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[UCS Server] Starting EQEmu Universal Chat Server.
[UCS Server] Connecting to MySQL...

Program received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/x86_64/strlen.S:106
106     ../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) backtrace
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
#1  0x00007ffff5f31da8 in _IO_vfprintf_internal (s=s@entry=0x7fffffffe180, format=<optimized out>, format@entry=0x555555882580 "Starting File Log 'logs/%s_%i.log'", 
    ap=ap@entry=0x7fffffffe2e8) at vfprintf.c:1637
#2  0x00007ffff5f58e89 in _IO_vsnprintf (string=0x7fffffffe270 "Starting File Log 'logs/\200\037", string@entry=0x0, maxlen=<optimized out>, maxlen@entry=0, 
    format=format@entry=0x555555882580 "Starting File Log 'logs/%s_%i.log'", args=args@entry=0x7fffffffe2e8) at vsnprintf.c:114
#3  0x00005555555db136 in vStringFormat[abi:cxx11](char const*, __va_list_tag*) (format=0x555555882580 "Starting File Log 'logs/%s_%i.log'", args=args@entry=0x7fffffffe348)
    at /home/eqemu/server_source/Server/common/string_util.cpp:47
#4  0x00005555555c9766 in EQEmuLogSys::Out (this=this@entry=0x555555868e20 <LogSys>, debug_level=debug_level@entry=Logs::General, log_category=log_category@entry=25, 
    message="Starting File Log 'logs/%s_%i.log'") at /home/eqemu/server_source/Server/common/eqemu_logsys.cpp:378
#5  0x00005555555c9a04 in EQEmuLogSys::StartFileLogs (this=0x555555868e20 <LogSys>, log_name=...) at /home/eqemu/server_source/Server/common/eqemu_logsys.cpp:493
#6  0x0000555555592d4f in main () at /home/eqemu/server_source/Server/ucs/ucs.cpp:101
Code:
sudo gdb queryserv
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from queryserv...done.
(gdb) run
Starting program: /home/eqemu/server/queryserv 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[QS Server] Starting EQEmu QueryServ.
[QS Server] Connecting to MySQL...

Program received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/x86_64/strlen.S:106
106     ../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) backtrace
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
#1  0x00007ffff5f31da8 in _IO_vfprintf_internal (s=s@entry=0x7fffffffe1e0, format=<optimized out>, format@entry=0x555555812c60 "Starting File Log 'logs/%s_%i.log'", 
    ap=ap@entry=0x7fffffffe348) at vfprintf.c:1637
#2  0x00007ffff5f58e89 in _IO_vsnprintf (string=0x7fffffffe2d0 "Starting File Log 'logs/\200\037", string@entry=0x0, maxlen=<optimized out>, maxlen@entry=0, 
    format=format@entry=0x555555812c60 "Starting File Log 'logs/%s_%i.log'", args=args@entry=0x7fffffffe348) at vsnprintf.c:114
#3  0x000055555558d3d6 in vStringFormat[abi:cxx11](char const*, __va_list_tag*) (format=0x555555812c60 "Starting File Log 'logs/%s_%i.log'", args=args@entry=0x7fffffffe3a8)
    at /home/eqemu/server_source/Server/common/string_util.cpp:47
#4  0x000055555558b736 in EQEmuLogSys::Out (this=this@entry=0x5555557fb180 <LogSys>, debug_level=debug_level@entry=Logs::General, log_category=log_category@entry=25, 
    message="Starting File Log 'logs/%s_%i.log'") at /home/eqemu/server_source/Server/common/eqemu_logsys.cpp:378
#5  0x000055555558b9d4 in EQEmuLogSys::StartFileLogs (this=0x5555557fb180 <LogSys>, log_name=...) at /home/eqemu/server_source/Server/common/eqemu_logsys.cpp:493
#6  0x0000555555579f20 in main () at /home/eqemu/server_source/Server/queryserv/queryserv.cpp:79
...not sure if it adds anything new to help troubleshoot though?
Reply With Quote
  #3  
Old 09-09-2019, 07:06 PM
mushko
Fire Beetle
 
Join Date: Aug 2019
Posts: 12
Default

So I've now completely wiped and reinstalled mysql-server:
Code:
sudo apt purge mysql-server mysql-client mysql-common
sudo apt autoremove
sudo mv -iv /var/lib/mysql /var/tmp/mysql-backup
sudo rm -rf /var/lib/mysql*
sudo mv /etc/mysql /var/tmp/etc_mysql_backup
sudo rm -rf /etc/mysql

sudo apt update
sudo apt install mysql-server
sudo mysqld --initialize
sudo chown -R mysql:mysql /var/lib/mysql
sudo systemctl start mariadb.service
sudo systemctl enable mariadb.service
sudo /usr/bin/mysql_secure_installation
I also deleted the /home/eqemu folder, then rebuilt the server with the install.sh script. As far as I can tell, the database is fine and the install went well.

I can run shared_memory, world, eqlaunch, and zone without any errors. It's still only ucs and queryserv that give the Segmentation fault errors.

If I launch everything except ucs and queryserv, I am able to get to the character creation screen. Trying to zone in, however, gives me the error:
Code:
[World Server] No zoneserver available to boot up.
[World Server] Attempting autobootup of tutorialb (189:0)
...so I assume ucs at least it critical to run, is that correct?
Reply With Quote
  #4  
Old 09-10-2019, 03:25 AM
mushko
Fire Beetle
 
Join Date: Aug 2019
Posts: 12
Default

Some more information from journalctl -xe:

Code:
kernel: traps: ucs[1394] general protection ip:7fab0f107676 sp:7fffed51fd38 error:0
kernel:  in libc-2.24.so[7fab0f087000+195000]
kernel: traps: queryserv[1398] general protection ip:7f8219af8676 sp:7ffef156aec8 error:0
kernel:  in libc-2.24.so[7f8219a78000+195000]
And Valgrind:

Code:
==6770== Command: ./ucs
==6770== 
[UCS Server] Starting EQEmu Universal Chat Server.
[UCS Server] Connecting to MySQL...
==6770== Invalid read of size 1
==6770==    at 0x4C2F1E2: strlen (vg_replace_strmem.c:460)
==6770==    by 0x69D2DA7: vfprintf (vfprintf.c:1637)
==6770==    by 0x69F9E88: vsnprintf (vsnprintf.c:114)
==6770==    by 0x18F135: vStringFormat[abi:cxx11](char const*, __va_list_tag*) (string_util.cpp:47)
==6770==    by 0x17D765: EQEmuLogSys::Out(Logs::DebugLevel, unsigned short, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ...) (eqemu_logsys.cpp:378)
==6770==    by 0x17DA03: EQEmuLogSys::StartFileLogs(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (eqemu_logsys.cpp:493)
==6770==    by 0x146D4E: main (ucs.cpp:101)
==6770==  Address 0x100010001000100 is not stack'd, malloc'd or (recently) free'd
==6770== 
==6770== 
==6770== Process terminating with default action of signal 11 (SIGSEGV)
==6770==  General Protection Fault
==6770==    at 0x4C2F1E2: strlen (vg_replace_strmem.c:460)
==6770==    by 0x69D2DA7: vfprintf (vfprintf.c:1637)
==6770==    by 0x69F9E88: vsnprintf (vsnprintf.c:114)
==6770==    by 0x18F135: vStringFormat[abi:cxx11](char const*, __va_list_tag*) (string_util.cpp:47)
==6770==    by 0x17D765: EQEmuLogSys::Out(Logs::DebugLevel, unsigned short, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ...) (eqemu_logsys.cpp:378)
==6770==    by 0x17DA03: EQEmuLogSys::StartFileLogs(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (eqemu_logsys.cpp:493)
==6770==    by 0x146D4E: main (ucs.cpp:101)
==6770== 
==6770== HEAP SUMMARY:
==6770==     in use at exit: 125,720 bytes in 223 blocks
==6770==   total heap usage: 511 allocs, 288 frees, 275,468 bytes allocated
==6770== 
==6770== LEAK SUMMARY:
==6770==    definitely lost: 0 bytes in 0 blocks
==6770==    indirectly lost: 0 bytes in 0 blocks
==6770==      possibly lost: 0 bytes in 0 blocks
==6770==    still reachable: 125,720 bytes in 223 blocks
==6770==                       of which reachable via heuristic:
==6770==                         length64           : 34,160 bytes in 17 blocks
==6770==         suppressed: 0 bytes in 0 blocks
==6770== Rerun with --leak-check=full to see details of leaked memory
==6770== 
==6770== For counts of detected and suppressed errors, rerun with: -v
==6770== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault
Reply With Quote
  #5  
Old 09-10-2019, 07:36 PM
Uleat's Avatar
Uleat
Developer
 
Join Date: Apr 2012
Location: North Carolina
Posts: 2,815
Default

I haven't been ignoring this thread..I just don't have any linux experience...


For 'ucs' and 'queryserv' servers, they do not affect the server's ability to operate.

That logging error, however... That would definitely cause a problem.

I'll take a look at that call and see if I can find anything.
__________________
Uleat of Bertoxxulous

Compilin' Dirty
Reply With Quote
  #6  
Old 09-11-2019, 02:51 AM
mushko
Fire Beetle
 
Join Date: Aug 2019
Posts: 12
Default

No problem, I am definitely out of my league on this one! UCS and Queryserv are still giving the segmentation fault, but I have been able to get the serving running without them for now. I essentially purged and reinstalled mysql-server and all of the dependencies in install.sh, deleted the old database and eqemu user in mysql, deleted and recreated the eqemu account, restarted my machine, and then reran install.sh to make the binaries. Now the server runs and I can connect to it, and everything on the client side seems to work fine. I'm not sure if there are bigger problems lurking underneath though, with logging for example, like you mention...
Reply With Quote
  #7  
Old 09-11-2019, 01:41 PM
joligario's Avatar
joligario
Developer
 
Join Date: Mar 2003
Posts: 1,490
Default

What happens if you change the host entries in your config to localhost?
Reply With Quote
  #8  
Old 09-12-2019, 04:12 AM
mushko
Fire Beetle
 
Join Date: Aug 2019
Posts: 12
Default

Unfortunately, changing the host entries to "localhost" didn't seem to have an effect... still getting the Segmentation fault :/
Reply With Quote
  #9  
Old 09-12-2019, 07:27 AM
Huppy's Avatar
Huppy
Demi-God
 
Join Date: Oct 2010
Posts: 1,333
Default

In windows, "localhost" will always resolve into tcp/ip (127.0.0.1), but that is not always the case in linux, it can do crazy things, like trying to resolve into an IPv6 address, but either way, not sure why people are so adamant about using that infamous localhost. Using 127.0.0.1 in all the configs, will guarantee to rule out that cause, for anything local.
__________________
Hanging out at Antonica.World
Reply With Quote
  #10  
Old 09-15-2019, 06:28 AM
provocating's Avatar
provocating
Demi-God
 
Join Date: Nov 2007
Posts: 2,175
Default

Debugging it done, the error is actually in eqemu_logsys.cpp in platform_file_name.c_str(). "ucs" is definitely in platform_file_name but somehow c_str is causing a segmentation fault. I am rusty on my C++ at the moment but I will get a resolution.

At some point platform_file_name is getting assigned the value of ucs, but at the time LogSys.StartFileLogs() is ran at ucs.cpp on line 101, that value does not appear to be there, at least on the surface. When we get to eqemu_logsys.cpp line 462 it throws a segmentation fault, my best guess is there is a null there. I am still reading on proper use of c_str but on the surface I guess there is a null there. There is a check for empty but it seems to blow past that.
Reply With Quote
  #11  
Old 09-15-2019, 07:20 PM
Uleat's Avatar
Uleat
Developer
 
Join Date: Apr 2012
Location: North Carolina
Posts: 2,815
Default

I'm pretty sure my_string.c_str() should return a nullstring pointer even if my_string.empty() is true.

Variadics on the other hand...

If you pass a nullptr argument, add too many formatizers or too few arguments, it will crash.
__________________
Uleat of Bertoxxulous

Compilin' Dirty
Reply With Quote
  #12  
Old 09-15-2019, 08:52 PM
provocating's Avatar
provocating
Demi-God
 
Join Date: Nov 2007
Posts: 2,175
Default

Right, I did have a little while to read more about c_str(). Apparently it cannot be null, it can have a null pointer but std::string cannot actually ever be null, there should always be a value even if it is an empty string. Also from reading there can still be quite a few bad things happen to cause std::string to go haywire and I guess that is what is happening. It definitely gets assigned a value at some point because I see the value appear but later on something happens. I have yet to find what is happening. Maybe I can look at it again tomorrow.
Reply With Quote
  #13  
Old 09-16-2019, 10:30 AM
provocating's Avatar
provocating
Demi-God
 
Join Date: Nov 2007
Posts: 2,175
Default

I had some time this morning to look at it again. I never would had figured the bug was in this location.

In the directory for ucs and queryserver look at the database.cpp and you will notice my correction here. One thing is that the routine was overwriting the file log settings every iteration, because it always ran through every entry without stopping. So really the routine was not really working properly without this being adding. I am not getting the segmentation fault anymore and my log is being written so I guess it is working? I need someone with a higher pay grade to look over this change though.

Code:
	for (auto row = results.begin(); row != results.end(); ++row) {
		log_category = atoi(row[0]);
		
		if (log_category <= Logs::None || log_category >= Logs::MaxCategoryID) {
			continue;
		}
		
		log_settings[log_category].log_to_console = atoi(row[2]);
		log_settings[log_category].log_to_file = atoi(row[3]);
Reply With Quote
  #14  
Old 09-16-2019, 02:29 PM
Akkadius's Avatar
Akkadius
Administrator
 
Join Date: Feb 2009
Location: MN
Posts: 2,071
Default

Interesting, just seeing this thread for the first time. I'll put it on my list to check this one out tonight
Reply With Quote
  #15  
Old 09-16-2019, 10:46 PM
Akkadius's Avatar
Akkadius
Administrator
 
Join Date: Feb 2009
Location: MN
Posts: 2,071
Default

First look this is related to newer GCC versions, investigating closer
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

   

All times are GMT -4. The time now is 04:24 PM.


 

Everquest is a registered trademark of Daybreak Game Company LLC.
EQEmulator is not associated or affiliated in any way with Daybreak Game Company LLC.
Except where otherwise noted, this site is licensed under a Creative Commons License.
       
Powered by vBulletin®, Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Template by Bluepearl Design and vBulletin Templates - Ver3.3