Player Character Ghosting (after player leaves game)

[trevius]

The problem happens most commonly if people /quit or /exit, or if they go LD, but I am pretty sure it even has a rare chance to happen if they /camp. On a public server when I was running Windows, I would commonly see up to 15 or more ghosted characters in nexus at any given time of the day if I didn't boot them out manually every few hours or so.

The easiest way to tell 100% if a character is a player ghost is to turn on inspect so you can inspect people "/toggleinspect on" (you can test by inspecting yourself or another player you know isn't a ghost) then try right clicking a suspected ghost player and if the inspect window doesn't pop up, they are sure to be a ghost.

Even after they log off, you will continue to get net error messages from them in the log files. I believe the error that shows up is something like this:

Tried to write a packet beyond the end of the queue!

The error starts with showing incremented packet numbers, but eventually I think it just shows the same packet number over and over and over. And that error will continue to flood server logs even after the player has logged off and I think even after you #kick the ghosted character. So, it is like a connection isn't being brought down for some reason even if there is no connection. I think I have even had a player reboot their PC and it didn't stop the errors or disconnect the ghosted character. So, to me it seems like the issue may be that the server is just hung in some sort of loop.

More info on that log error can be seen in this post:

http://www.eqemulator.net/forums/sho...light=flooding

[KLS]

I don't get the packet flood with ghosting players. It's something with the netcode isn't timing out connections for windows in the zone server.

[trevius]

After reading through that post I linked above again, I notice that the error that seems to be generated when this problem occurs seems to be coded here:

EQStream.cpp

if(CompareSequence(NextOutSeq, seq_send) == SeqFuture) {
_log(NET__ERROR, _L "Tried to write a packet beyond the end of the queue! (%d is past next out %d)" __L, seq_send, NextOutSeq);
sitr=SequencedQueue.end();
continue;
}

And, looking into what that piece of code does, I looked at CompareSequence and SeqFuture so far. Here is the code for that command and I noted a part in RED that I think might be a mistake:

//returns SeqFuture if `seq` is later than `expected_seq`
EQStream::SeqOrder EQStream::CompareSequence(uint16 expected_seq , uint16 seq)
{
	if (expected_seq==seq) {
		// Curent
		return SeqInOrder;
	}  else if ((seq > expected_seq && (uint32)seq < ((uint32)expected_seq + EQStream::MaxWindowSize)) || seq < (expected_seq - EQStream::MaxWindowSize)) {
		// Future
		return SeqFuture;
	} else {
		// Past
		return SeqPast;
	}
}

I think that line might need to be removed and replaced with this:

	}  else if (seq > expected_seq && (uint32)seq < ((uint32)expected_seq + EQStream::MaxWindowSize))  {

That should label all packets that are lower than the expected sequence as SeqPast. It will also label any any packets that are above the MaxWindowSize as SeqPast. And, if an ack is set as SeqPast, it should always SendOutOfOrderAck, which means it should correct most out of order issues I think.

Here is the MaxWindowSize setting code:

uint16 EQStream::MaxWindowSize=2048;

I will try to get the minor change noted above in and tested on my server and see if anything funky starts happening or if it has any noticeable effect. But, since I only run a Linux server, I won't see player ghosts anyway, so it might be pointless for me to test it. But, as long as it doesn't break anything, I could always update it in the new SVN and when Cavedude builds the Win32 binaries, windows servers could easily start testing and see what happens if anything.

Just from reading through the EQStream.cpp file, to me it looks like they are doing some sloppy stuff and maybe some extra work that isn't really needed. But, I am definitely no expert or anywhere near as skilled as the people who originally wrote that file. It seems like we could just check if an ack is in order and if not then always send an OutOfOrder ack. Maybe if we play with this file for a while, we can get most or all network issues worked out.

[KLS]

A lot of what goes on isn't because we want to do it that way. The client controls the netcode and we don't control the client. =p

[trevius]

Ya, but if we are getting out of order acks, doesn't that mean something is messed up and the server needs to compensate for it to recover properly? Obviously it is working for the most part or there would be more issues. But, I think a couple little tweaks could help resolve some of the issues like hung sessions, out of order acks, and whatever is needed to fix the Windows player ghosting issue.

[RhinoDude]

Trevius, did you end up making this change to your Linux server, and if so, did you note any adverse side effects?

If not, perhaps we could submit this as a fix, or find somebody who can test it under Windows? I will be setting up a Windows server eventually.

Also, is this TCP or UDP traffic we are talking about? If UDP the state of the client (rebooted or not, etc) would have zero impact normally. On the other hand, if we are talking TCP, then a difference in the TCP stack implementations could be the core underlying issue, and explain why it's visible only under Windows.

Edit: I forgot to add, I can't remember if EQ is UDP only or UDP + TCP. I've been away for 5+ years.

- Rhino

[trevius]

LOL, ya I tried making the change, but it didn't work right. I have tried a few other things and still no-go. My understanding of the network packets are less than what they should be lol. Maybe at some point this will get resolved by someone who catches the reason for causing ghosts only in Windows and not in Linux :P