Possible Login Server Bug

[Dralanna]

There is a thread going on in the General Forum regarding this issue. (http://www.eqemulator.net/forums/showthread.php?t=22521) It was suggested that we put it here. I'm copying many of the relevant posts here:

Originally Posted by zuperzip

If any of the information below is not solid please let me know what criteria needs to be met to make the information concrete and useful. I would be happy to answer any questions or perform further testing. Contact me by PM or admins may use email.

The eqgame.exe error has been visited many times on this and other forums. For many players of EQEmu there is a certain cause for this error at a certain stage of the login process that can block logins for an hour or more at a time, off and on throughout the day.

I'm going to call this error "SSSE" for Server Select Screen error. There are a number of causes for the eqgame.exe error message and it is necessary to focus on when it occurs on order to define the SSSE.


--- When the SSSE strikes ---
The SSSE occurs when the "LOGIN" or "QUICK CONNECT" button is pressed either with mouse button or tab/enter keys. If you use the "LOGIN" button the SSSE strikes immediately so you never get to press the "PLAY EVERQUEST!" button on the server select screen. Players find themselves trying to get to the "PLAY EVERQUEST!" button as quickly as they can by mouse or tab/enter but fail. A slowed mouse pointer is a symptom that the eqgame.exe error is about to pop up. Often players conclude it is a mouse driver problem until they try using the tab/enter keys but fail. It is also true that if the error does not strike immediately, sitting at the server select screen and doing nothing may eventually result in the SSSE. If the "QUICK CONNECT" button is used the SSSE will strike without displaying the server select screen. Once loading begins for the character select screen the SSSE will not occur.

The above is followed by the generic eqgame.exe error message which reads "eqgame.exe has encountered a problem and needs to close. We are sorry for the inconvenience." In the same window there's also the standard Microsoft message with options to "Send Error Report" or "Don't Send". This is on Windows XP.


--- The SSSE cause ---
Recently I performed a series of packet captures and analyzed them. Searching the forums shows "neville" also did some packet captures. I'm not sure if anything further was done with his results.

http://www.eqemulator.net/forums/sho...5&postcount=21
http://www.eqemulator.net/forums/sho...6&postcount=22

What the results show connects a specific packet to the SSSE. That packet contains the survey information for the client. You may recognize this survey since occasionally it pops up during login, "Is this thing on?......... 1) Yes, 2) No, 3) What thing?, 4) Stop wasting my time!". This will be referred to as the survey packet.

Though the survey rarely displays during login the survey packet is always sent with one exception I'll mention later*. In my captures a survey packet arrives as either the 3rd or 4th packet from the server and never causes a problem. This survey packet arrives before the server list is sent for the server select screen. Anytime a second survey packet is sent to the client an SSSE occurs immediately.

A second survey packet can be sent at the tail end of the login process following the list of servers. This makes it impossible to log into your selected server even using the "Quick Connect" button. Generally repeated tries will eventually get through but may take 20 minutes and 40 login attempts or more.

A second Survey packet may also be sent at some point after the initial login while sitting at the server select screen. This survey packet also arrives at the tail end of the list of servers, but in this case it's the server list update. The delay between login and when this second survey packet is sent may vary from a few seconds to hours. When I was running tests on March 20th a second survey packet arrived with delays between 2.5 minutes and 34 minutes over five captures.


What follows is one of the second survey packets as it appears in the capture. It is always 233 bytes in size:

0000 00 0b 6a 58 fb f5 00 30 ab 07 79 7d 08 00 45 00 ..jX...0..y}..E.
0010 00 db 00 00 40 00 34 11 64 c2 0c a4 14 02 c0 a8 ....@.4.d.......
0020 00 02 17 6e 04 30 00 c7 47 b2 00 09 00 17 29 00 ...n.0..G.....).
0030 00 00 00 00 00 00 00 00 00 00 01 00 55 53 00 45 ............US.E
0040 4e 00 01 00 00 00 49 73 20 74 68 69 73 20 74 68 N.....Is this th
0050 69 6e 67 20 6f 6e 3f 00 04 00 00 00 01 00 00 00 ing on?.........
0060 00 00 00 00 01 00 00 00 00 01 00 00 00 01 00 00 ................
0070 00 00 00 00 00 59 65 73 00 02 00 00 00 00 00 00 .....Yes........
0080 00 01 00 00 00 00 01 00 00 00 02 00 00 00 00 00 ................
0090 00 00 4e 6f 00 03 00 00 00 00 00 00 00 01 00 00 ..No............
00a0 00 00 01 00 00 00 03 00 00 00 00 00 00 00 57 68 ..............Wh
00b0 61 74 20 74 68 69 6e 67 3f 00 04 00 00 00 00 00 at thing?.......
00c0 00 00 01 00 00 00 00 01 00 00 00 04 00 00 00 00 ................
00d0 00 00 00 53 74 6f 70 20 77 61 73 74 69 6e 67 20 ...Stop wasting
00e0 6d 79 20 74 69 6d 65 21 00 my time!.


For comparison a harmless first survey packet shown below is 241 bytes in size.

0000 00 0b 6a 58 fb f5 00 30 ab 07 79 7d 08 00 45 00 ..jX...0..y}..E.
0010 00 e3 00 00 40 00 34 11 64 ba 0c a4 14 02 c0 a8 ....@.4.d.......
0020 00 02 17 6e 04 30 00 cf 2b f6 00 03 04 00 15 00 ...n.0..+.......
0030 01 bf 00 09 00 01 29 00 00 00 00 00 00 00 00 00 ......).........
0040 00 00 02 00 55 53 00 45 4e 00 01 00 00 00 49 73 ....US.EN.....Is
0050 20 74 68 69 73 20 74 68 69 6e 67 20 6f 6e 3f 00 this thing on?.
0060 04 00 00 00 01 00 00 00 00 00 00 00 01 00 00 00 ................
0070 00 01 00 00 00 01 00 00 00 00 00 00 00 59 65 73 .............Yes
0080 00 02 00 00 00 00 00 00 00 01 00 00 00 00 01 00 ................
0090 00 00 02 00 00 00 00 00 00 00 4e 6f 00 03 00 00 ..........No....
00a0 00 00 00 00 00 01 00 00 00 00 01 00 00 00 03 00 ................
00b0 00 00 00 00 00 00 57 68 61 74 20 74 68 69 6e 67 ......What thing
00c0 3f 00 04 00 00 00 00 00 00 00 01 00 00 00 00 01 ?...............
00d0 00 00 00 04 00 00 00 00 00 00 00 53 74 6f 70 20 ...........Stop
00e0 77 61 73 74 69 6e 67 20 6d 79 20 74 69 6d 65 21 wasting my time!
00f0 00 .


* The one exception is some accounts are never sent survey packets. These accounts do not receive the survey packet near the beginning of the login process or at anytime after and so never experience the SSSE. Currently I know of three such accounts, one I have tested myself but do not own. The other two are owned by someone who experiences SSSEs on his other accounts. Both owners would be interested in helping rid the EQEmu community of the SSSE.

I believe firewall settings may somehow help prevent or allow this problem to occur but right now this is nothing more than a hunch. I do not have packet captures showing differences between different firewalls and their settings.


Below is older information that I gathered over the past months and may be extraneous now. I'm including it in case it may be of some value.

--- Gathered information about the SSSE ---
1) Many hours may go by with no accounts/players experiencing SSSEs or at least the frequency of the SSSE is very low.

2) When there is a high frequency of SSSEs all accounts/players that can be affected by the SSSE will have great difficulty logging in. Repeated login attempts will eventually get past the SSSE.

3) While sitting at the Server Select screen an SSSE may occur immediately when there is an update in the server list.

4) Some accounts are completely immune to the SSSE. This is true even when an immune account is used on different computers in different global locations where the SSSE does occur and when it is occurring with a high frequency.

5) Some players never experience the SSSE.

6) For two weeks this past February I visited a different location 140km away using a different DSL modem and ISP. I took one of my EQEmu computer boxes with me but all other equipment used was from the other location. I often loaded up 4 EQEmu accounts and switched accounts to play different characters. For that two weeks I never experienced an SSSE even though other players often mentioned in game how harsh the SSSE was being to them. A couple days before I left home and the day after I returned home I experienced the SSSE on that computer box.


--- What I think ---
SSSEs are not related to drivers or settings on the client system. It is caused by data sent by the login server or server select server to the EQ Titanium client. The the frequency with which the data is sent raises and lowers in an irregular pattern. For some accounts this data is never sent at anytime. Some firewalls filter out this data preventing client crashes.

[Dralanna]

More quotes

Originally Posted by zuperzip

I have found that responding to the survey stops the login server from sending you the survey packet. The survey shows itself very rarely so when it does come up and asks, "Is this thing on?" you need to select "Yes" and click "Submit". I did this and now that one account is no longer sent the survey packet and so no longer gets the SSSE.

Originally Posted by Vastlee

Any follow up on this?

This exact thing you described, as opposed to the random eqgame.exe crash that others seem to get mixed up with this, has been happening to me for over a year now. I have 6 accounts. 2 of them are as you stated, "Immune" to it.

One night I actually counted 36 retries before it let me through, and retries seem to be the only thing that actually gets you through. Not rebooting, not logging other accounts in, nothing. And the # of retries seems to be 100% random.

This is the first post I've seen on the subject that identifies the problem specifically. The unfortunate part is that there are several things that cause eqgame.exe to crash that when someone asks about it on the boards 10 other people pop up with another "Oh yeah mine too" story... only it isn't the same thing.

I really hope that someone at least takes notice of this. Running the game 20 times really really sucks nuts.

P.S. The error I'm talking about is 100% exactly as described above. I implore anyone who does not have the exact same symptoms to please not post. I agree with the above that it is something that is being sent from the login server that is causing the problem, which means if we can't get those who maintain that to take notice then nothing will ever get done. So PLEASE

P.S.S. Is there some way to force the survey thing to come up? If clicking YES to it is all I have to do I'd be happy to. I just never see it come up.

Originally Posted by Erich

I don't think the "I have been bugged about the survey" flag is client side. If it was then it wouldn't be an account that is immune but rather a entire computer. I'm willing to bet that anyone with an account that is immune will tell you that it's immune on any computer that they use while an account that is not immune will not be immune on any computer.

I'm betting that the "I have been bugged about the survey" flag is server side and if it can be turned on for all accounts this problem will go away.

Originally Posted by Vastlee

Agreed, I have 5 machines. The accounts that are "immune" are so on all machines.

Originally Posted by Darkonig

I was getting on just fine for months and then one of my characters had the survey show up. I responded as suggested above and now periodically have the crashes on several characters. If a character is having the problem on one of my machines and I try to log him in on a different machine the account still has problems even tho other accounts on the same computers work fine. It is definitely something the login server is doing. I have never had the problem from the minilogin server.

I grabbed a couple packet dumps from when the one account was crashing excessively (more than 20 times in a row and on multiple computers), but haven't had a chance to review it yet.

Originally Posted by Darkonig

Ok, just went thru the dumps and found like expressed above. On the ones that crashed, there was a survey packet of size 241 was received, then the server list, then 1 or more additional survey packets of size 233.

Even using quick connect, there was a survey packet size 241 sent, then the server list, then additional servey packets of size 233. They are received regardless of whether the client choses to display them.

On the characters that got in fine, there were no survey packets at all.
So, we either need to get the login code changed to quit sending survey packets, or build a filter to remove them from the stream when sent.

I am going to try the latter approach and see if it resolves the problem. I will have it notify me of any filtered survey packets to verify if it makes a difference. I would assume that altering the data stream to remove those packets would not be any more of a legal issue than using a different login server to begin with.

Originally Posted by zuperzip

There is a work around that a player by the name of Blasticus on PEQ recently discovered and told us about. At one point I thought accessing a webpage while simultanously logging in would prevent an SSSE but couldn't get it to work the majority of the time. Blasticus tried using a speed test and found it worked very well and so have I. Here is what works for me, open the EQ client and type in your username and password in windowed mode so it is ready to go. Now find any speed test website with your webbrowser (I've been using performance.toast.com or infospeed.verizon.net) and get ready to start the speed test. Start the speed test then switch to the EQ client and quickly log in.

What's important is you 1) start the EQEmu authentication process and 2) get past the server select screen _while_ the speed test is downloading its sample.

It also apparently works if you do the speed test on another system that uses the same router as your EQEmu system.

Hair brained? It sure is... but it works so well for me during a meteor shower of SSSEs that it's more than just dumb luck. I suppose if I want to figure out why it works I'll need to put my nose back into packet sniffing.

Originally Posted by Dralanna

I'm guessing that people are having luck with the speed test trick because it uses a lot of bandwidth and is probably delaying the survey packet long enough for you to login.

Did anyone trying this also do a packet capture?

Originally Posted by Dralanna

Not sure if anything has changed recently, but yesterday 3 of my accounts got the "is this thing on?" survey... to which I (obviously) said yes. Now those accounts no longer crash. The remaining accounts continue to crash at the server select screen.

Originally Posted by Darkonig

4 of my accounts yesterday got a successful survey. The survey pack contains the question Is this thing on? and 4 possible answers. With responses of submit and skip. So, when I stated before that you always got a survey packet of one size, then list of servers, then 1 or more additional survey packets I was incorrect. The actual transmission is a survey packet, then the list of servers, then zero or more additional survey packets less an 8 byte custom udp header. If no bogus survey packets follow then you get the survey displayed, if 1 or more bogus ones arrive then you crash.

I answered all 4 the same way, as above. One of them crashed again several hours later when I went to log it back in. So, answering the survey does not guarantee the login server will not ask again.

As to the other poster about cannot packet sniff the login server, all of the survey packets are identified as coming from the login server IP and Port with appropriate sequence numbers to suggest they really did.

Originally Posted by Vastlee

Of all my accounts (9 active now) only 1 got prompted with the survey. (Is this thing on) and I answered "Yes" too. The account still drops though.

I hope that the surveys were sent out as an attempted response by the devs to this thread. I'd be happy just to know someone is aware and looking into it, even if the first attempt didn't work. If the survey pops up again I might try answering something else and see what happens.

Good work to all the non-dev people for providing useful information.

Originally Posted by Vastlee

I had the same thought as you Dralanna, but the 2 accounts I have that never crash have never seen the Survey pop up, and the 1 I got the other day still crashes. I'm still fairly sure it has something to do with the survey but I'm not sure that answering it solves the problem (Or has any effect). I also considered the thought that maybe it is the answer that makes or breaks you. I answered "Yes" to it this time. If it ever comes up again I'll try skipping it and see if that changes anything.

Although, it seems the easiest answer would be for the login server to simply stop sending the packet that causes the crash.

Originally Posted by Dralanna

With all due respect, Angelox.... I am convinced this is a bug with the login server. I have several accounts, when I captured packets every single one got the packet listed in the first post at one time or another. When I got that packet, I crashed. When I didn't get that packet I didn't crash. When the survey finally popped up on 3 of my accounts and I answered correctly those 3 accounts never got that packet again, and subsequently have not once crashed at the server select screen. Now, to rule out the possibility of a client or computer issue, I've logged my accounts (both the bugged and the unbugged) on using multiple machines, OSs, Connection types.... anything I could change and consistently the bugged accounts stayed bugged and the unbugged accounts stayed unbugged.

[Vast]

Thanks Dralanna, hopefully this will catch some attention now.

[Vast]

I'd like to note that every account that the survey has come up on and I selected "Yes" to the question "Is this thing on" is now fixed and immune to the bug.

This is definitely survey related. If they could just do a round of survey's for people to answer or stop them all the way it would solve this problem.

[Vast]

Anyone seen this? I am still very much affected by this as I have been for a year now, and OOC is full nightly cursing 3SE bug, so I'm not the only one.

[Vastlee]

I'd like to note something here. Last night I was one of the rare few who was on when the login servers came back up. When they did I started logging in and was extremely surprised when all of my characters logged in on the first go. This is something that hasn't happened.

I was thrilled, I assumed that during this down time someone had addressed this issue, but to make sure it wasn't just an amazing coinky dink I logged in and out a total of 13 times and every single time I got in fine.

So I go to bed and get up this morning to find that it's back. I tried to remember what was different. The one thing I noticed is that not a whole lot of the servers had reattached to the login server. So there is a possibility that one server name or number of servers in the list is directly tied with the survey.

Anywho, there appears to be a missing X variable along with the survey one. Hopefully someone will catch this soon.