World Server Disconnects
 

I am not sure what is causing this, but recently I started noticing an much high frequency of World Server Disconnects on my server. I have seen these in the past, but it was pretty rare and only happened when I was using the admin tool that comes with the emulator to search for accounts/character names. Now they are occurring randomly every 1 to 4 hours it seems. I can't seem to find any common thing that could be causing this.

When this happens, it doesn't crash the zones, but it does crash anyone that tries to zone. Then they can log back in again. But, it gets very annoying I am sure to people who box 3 characters and have to relog each one every couple of hours.

Here is what I am seeing in game:

Etc until it lists all dynamic zones in use.

And here are some of the logs from when that has happened:

These are 4 different times that it occured
And here are more detailed logs from when those occur (note that I changed all player names in these logs to "Player"):



If anyone else has been seeing this problem more lately, maybe we can all work together and figure out what is causing it. This is starting to get more than just a little annoying now. The problem with troubleshooting is that it takes at least a day to test each step. I plan to possibly revert back to some older code that I know didn't do this and see if that resolves it. If not, then I plan to remove all level 75 discs which were added recently. After that, I will have to look at quest scripts and see if one of them could possibly be causing it.

I have a build 1129 (cvs db,quest) eqemu peq server running on debian 4.0r1 linux server. Just got this up and running 3 days ago. I had the windows one running before that. No issues at all until today

I had 2 buddies log in and we were doing some hunting in some planes when they disappeared and we then saw a black screen stating "we were disconnected", we could instantly log-back in the server did not report anything in the logs at that time abnormally and it did not crash.

This happened about an hour later, then another 4 times at different times (Hour apart, 2 hours, 45 minutes). I was grouped with them and things stopped responding, they vanish and that black screen comes up with you have been disconnected (we all got the same error). I am extremely new with eqemu. I find it amazing what work was put into this and how skilled everyone is around here doing the work they do.

I will def try and get some logs posted exactly when this occurs again as right now I am not exactly sure the issue is showing in the logs. I reviewed them a few times. I thought it may be network related, due to Saturday heavy network loads with my ISP but I will be testing that out the following days.

After looking at logs and testing in game it seems as though my world server is not disconnecting me. Its when someone turns on PVP and tries to hit someone the zone will crash instantly.

Ya, I think your issue is something completely different than my issue.

Well, I tested every level 70+ disc that I recently added to my server and none of them caused the crash for me. So, I am guessing it is something else. I am still leaning towards it being the source code or one of the custom changes I added in recently since I do run quite a few of them.

I will continue testing and see if I can come up with anything. So far, I tried reverting back to 1128 code version, but I also added in most of the features that have been added since then to it as well. Maybe it is one of those features that is causing the problem. The list of most of the added features I am currently using is:

Enraged, Flurry and Rampage Spamming Fix
http://www.eqemulator.net/forums/showthread.php?t=26258

Technique of Master Wu AA (Fix)
http://www.eqemulator.net/forums/showthread.php?t=26160

quest::traindiscs() and new quest::scribespells()
http://www.eqemulator.net/forums/showthread.php?t=26083

IP Limiting Additional Feature and Correction
http://www.eqemulator.net/forums/showthread.php?t=26122

#ItemLink <Item ID Number> (quest::itemlink())
http://www.eqemulator.net/forums/showthread.php?t=26018

Proc spells don't work after zoning (Proc Buff Fix)
http://www.eqemulator.net/forums/showthread.php?t=25991

New Quest Command quest::clearspawntimers()
http://www.eqemulator.net/forums/showthread.php?t=26261

Damage Bonus Functions (2H Bonus Fix)
http://www.eqemulator.net/forums/showthread.php?t=26262

Resurrection Reworked (with Confirmation box)
http://www.eqemulator.net/forums/showthread.php?t=26049

Buffs Being Blocked That Shouldn't Be (CHA Stacking Fix)
http://www.eqemulator.net/forums/showthread.php?t=25570

Spell Blocking (now with Intra-Zone restrictions!)
http://www.eqemulator.net/forums/showthread.php?t=26051

Spell And Skill Rule for Above MaxLevel Players
http://www.eqemulator.net/forums/showthread.php?t=25937

I've just been looking over the changes to clientlist.cpp in that thread and I think the logic is flawed if that is how you have it coded:

If the code reaches the most deepy nested iterator.RemoveCurrent(), then not only will it it remove the ClientList entry of the connection exceeding the IP limit, but the iterator will be updated to point to the next entry in the ClientList (which probably belongs to someone else), and the flow of execution will then reach the second iterator.RemoveCurrent() and remove that one as well.

My head's hurting from looking at it, so maybe I am misunderstanding the intention of the code.

Ya, I do kinda suspect that code could be the cause of the issue. It does actually seem to work the way I want it to from the testing I have done, but I haven't really checked to see what happens if I try to log in more characters than that.

Basically, I want my normal status 0 accounts to be able to log in 3 characters max at once. I then want my status 2 accounts to be able to login a max of 4 characters (1 more character) as a bonus for purchasing my contributor package. Then, I want anything over status 20 to be exempt from the IP limiting. As it is now, the code I have does seem to work to do exactly what I want, but the flaw you mention probably is a real problem. Maybe it is unable to select another client once the first one is removed, so it causes a world crash.

I would definitely be open to better ways of doing this! I know it hurts my head to think about the code for it, but maybe there is a simpler way of getting it done. As suggested in that thread, maybe just having it so that you can set the max number for all clients below X status. Then, for every point higher than that set status it would allow them to have +1 accounts logged in.

Ultimately, I think there needs to be code added (preferably with a rule option) to limit people to only being able to play 1 character per account. Then have the IP Limiting checked after the 1 character per account check happens. So, if someone already has a character logged in that might be ghosted, it will kick them out of the world instead of completely blocking them with IP limiting.

I will keep looking at this code and see if I can make it work better. However, professional help would be appreciated :)

I am going to try removing this temporarily from my server to see if that resolves my World Disconnect issue. It seems to have started around the same time I added this code in, but I also added alot of other stuff in at that same time, so it is hard to tell. I think this is a likely suspect if no one else is seeing the same exact problem that I currently am.

Do you have gdb installed? You can use that to parse the core dumps to tell you exactly where in the code the crash occurred.

I just had someone test this on my server. They were able to log in 4 accounts just fine, but when they tried to enter world on the 5th account, it caused the World Disconnect! So, this code is obviously the problem.

Now that I am looking at it again, I see that it is really done pretty badly to begin with lol. The only reason I even stickied the post was because it does include a valid fix for the current IP Limiting code, which is just changing a <= into a < check instead. That minor fix wasn't really breaking anything, but it was definitely making the rule settings misleading.

The custom code I added after that is just custom stuff I am trying to get working properly on my server. I thought it was working right until now and after further testing. I don't think it was actually working quite as I thought to begin with...

This is what I think the IP Limiting code should be doing to work how I would like to see it:

***Optional Additional Feature***
Check if a character is already logged in on this account
If so, then kick the logged in character
If no character is already logged in on this account, then continue
***Optional Additional Feature***

1. Check if there are more connections than the IP limit (MaxClientsPerIP) setting
If not, then allow them to log in.
If so, then continue

2. Check if the Account Status is >= the Additional IP Limit (AddMaxClientsStatus) Setting
If not, then IP Limit them to the IP Limit (MaxClientsPerIP) setting
If so, then continue

3. Check if there are more connections than the Additional IP Limit (AddMaxClientsPerIP) setting
If not, then allow them to log in.
If so, then continue

4. Check if the Account Status is >= the Exempt (ExemptMaxClientsStatus) setting
If not, then IP Limit them to the Additional IP Limit (AddMaxClientsPerIP) setting
If so, then allow them to log in.


Or maybe another way to do it would be:

Check Account Status
- If Status >= Exempt (ExemptMaxClientsStatus), then don't IP Limit
- If Status >= Additional IP Limit (AddMaxClientsStatus) && <= Exempt (ExemptMaxClientsStatus), then IP Limit to the Additional IP Limit (AddMaxClientsPerIP) setting
- If Status < Additional IP Limit (AddMaxClientsStatus), then IP Limit to standard IP Limit (MaxClientsPerIP) setting

I will see if I can figure out the flow of how the code should be to make one of these ways work. Really, I wouldn't mind getting another option in instead of just having a single Additional max clients and status setting. I would like for it to have a setting that just adds 1 more client for every status point over the status setting for these additional account connections.

(Account Status - AddMaxClientStatus + 1) = Total Additional Accounts Allowed Per IP. Of course you would have to check to make sure this is >= 0 otherwise any account below (AddMaxClientStatus - 1) will go into the negative. I will post the code if I figure anything out for it.

Sorry not trying to derail the thread but
There may be a way to kick the last logged on character for that account without caring if it is logged in.
The account table keeps the name of the last character logged into with that account.

If I get some free time I will see if trying to kick a character that is not logged in causes any crashes/errors.

I tried reorganising the code to work as you originally intended. This is totally untested.

Note that in the case where a connection is removed, I have put a continue; statement which will go back to the top of the while loop, as I don't believe you want to advance the iterator if you've just deleted an element.

Thanks for the post! I have been working on it some too. This is all I have come up with so far:


And, I think this should work for the most part. This stuff hurts my head to think about lol. The only part I still need to figure out is if there is a way to set it to exempt accounts that are above the AddMaxClientsStatus, but still limit them to the total count that they should be set to. The total limit for accounts over the AddMaxClientsStatus should be:

(countCLEIPs->Admin() - (RuleI(World, AddMaxClientsStatus)) + (RuleI(World, MaxClientsPerIP)) + 1)

So, if the account status is 5 and the AddMaxClientsStatus is set to 2 and the MaxClientsPerIP is set to 3, then the equation would look like:

5 - 2 + 3 + 1 = 7

So, they would be able to log in 7 accounts total which is 4 more than all normal accounts below the AddMaxClientsStatus can log in.

Another example is if the account status is 2 and the AddMaxClientsStatus is set to 2 and the MaxClientsPerIP is set to 1, then the equation would look like:

2 - 2 + 1 + 1 = 2

So, they would be able to log in 2 accounts total which is 1 more than all normal accounts below the AddMaxClientsStatus can log in.

The hard part I can't figure out is what happens if they log in multiple accounts all with different statuses on them.... Does it use the highest given account status to define the total of all accounts that can be connected, or does it go on a case by case basis. So, if you log in an account that should be limited to 4 max connections and then log in 3 accounts that are limited to 3 max, will the 4th account be able to login even though it is only individually set to allot 3 max? I am sure that if you log in 3 accounts that should be limited to 3 max and then log in a 4th that is limited to 4 max, the 4th should be able to connect without a problem.

I will test this out and report back lol. I don't think not knowing how to program is helping me, but I think the worst part is figuring out the logic of the flow process it should be using and how to handle every possible scenario :P

Well, I couldn't get my code working at all. It was limiting them to 1 char max no matter what.

But, I added Derision's code and it at least works the way I was wanting it to work :D

It would be nice to eventually add in the optional check to see of a character is already logged on for an account and boot the character if another character on that same account tries to log on. But, that would be a separate code than the IP limiting. It would also be nice to eventually try to get the new IP Limiting feature to allow the number of allowed accounts to encrement depending on how high the account status is above the AddMaxClientsStatus setting. That is what I was trying to do with the last code I submitted above that isn't working heh.

HUGE thanks, Derision!