Player Character Ghosting (after player leaves game)

[trevius]

I have seen this issue with almost all emu servers I have played on over the past couple of years. Players will use /exit or /quit instead of /logging out, or they will crash to desktop and leave a ghost in game. Unless that player logs back in, their character will continue to stand there until the zone or server are reset or a GM does a #kick on the player. This is most apparent in more popular zones.

I have nexus setup as my server base and after a day of running the server, Nexus will have an average of 10-15 ghost characters when I get home and check it. My server peaks around 60+ players, and these ghosts definitely seem to take a toll on server performance. I know my server isn't the only one affected by them at this time.

I have searched these forums multiple times and have never even seen a dev mention or confirm the issue. I couldn't find any bugs listed for it, or anything good in server support. The few posts I seen that mentioned ghosting at all were unanswered posts.

I've tried setting the rulesets related to linkdeath and timeouts, but that makes no difference.

For some reason, these ghosts seem to Blink off and on. First you can see them, then they will disappear for a couple of seconds, and then reappear. The best way to tell if a player is a ghost is to try to inspect them by right clicking. If you get the con message, but no inspect window, they are definitely a ghost.

I remember playing on Zebuxoruk server in late 2006 and the ghost issue was initially a problem there. But once that server was updated to a certain version of code, the ghosting seemed to stop completely and the server seemed more stable. So, it seems that the ghosting issue was resolved at some point of the code releases, but it is definitely back.

My theory to this is that for some reason, the connections aren't being torn down after characters leave the world. This may only be an issue on Windows, but I haven't heard anyone confirm that Linux never ghosts players. The reason I think that connections aren't being torn down is because while I was troubleshooting another issue, I found that I was getting traffic from IP addresses that weren't even logged into my server anymore. I had an issue with high latency players causing my debug logs to be flooded, so I downloaded an IP blocker and setup a block for IPs that were causing the error. I found that even after the player had logged off, or after I #kicked them off, they would continue sending me traffic indefinitely until I rebooted my server. Even if I blocked the IP for a while and then unblocked it, the connection would still continue! I am beginning to suspect that these connections staying active might be the reason Windows servers need to be restarted so often to keep lag from reaching insane levels. If each connection did stay active even after people logged off, that would well explain why it would impact server performance over time. Having a couple hundred connections is enough to bog down almost any server.

Does anyone out there know a fix for this for Windows? Maybe there is a setting I am missing somewhere. If not, maybe this is something the Devs can look into if they get a chance.

Thanks

[moydock]

I see the same thing on my server. You can send tells to them, they show up on who all, and you can hit them, but you can't inspect them.

[trevius]

If your server has a decent amount of players on it, the ghosting can cause real performance issues in a day or so. If you have any issues with it, I suggest trying my zone reset quest to clear out the ghosts regularly. You can add a zone resetter to every custom zone you have for maximum benefit. Here is the link to the post:

http://www.eqemulator.net/forums/showthread.php?t=24485

[trevius]

I know this report is fairly old, but player ghosting is still an issue on Windows servers according to some of the admins I speak with regularly. I don't have any solution, but I just wanted to post some code here which is probably the cause of the problem. I don't understand what is being done at all in this, but this is where the disconnect is supposed to happen that I assume isn't happening when players are ghosted. Maybe someone with more experience can check this out sometime and see if anything stands out as a possible issue.


TCPConnection.cpp
void TCPConnection::FinishDisconnect() {
MState.lock();
if (connection_socket != INVALID_SOCKET && connection_socket != 0) {
if (pState == TCPS_Connected || pState == TCPS_Disconnecting || pState == TCPS_Disconnected) {
bool sent_something = false;
SendData(sent_something);
}
pState = TCPS_Closing;
shutdown(connection_socket, 0x01);
shutdown(connection_socket, 0x00);
#ifdef WIN32
closesocket(connection_socket);
#else
close(connection_socket);
#endif
connection_socket = 0;
rIP = 0;
rPort = 0;
ClearBuffers();
}
pState = TCPS_Disconnected;
MState.unlock();
}

void TCPConnection::Disconnect() {
MState.lock();
if(pState == TCPS_Connected || pState == TCPS_Connecting) {
pState = TCPS_Disconnecting;
}
MState.unlock();
}

[Angelox]

Here's some thoughts I have on this, I have a windows server, but not many players to test with.
Could it be related to when they '/q' the game or LD? a lot of players don't '/camp', they '/q'.
Also, could it be related to the client playing on the same server machine? < maybe this creates some sort of lag/missed packets that creates this problem.
One thing would be to temporarily disable the '/q' command in the code, and instead place a message like 'please camp out normally, as we are testing for a PC ghosting bug'.
You could then see if at least this problems diminishes.

[ChaosSlayer]

Could it be related to when they '/q' the game or LD? a lot of players don't '/camp', they '/q'.

these guys are strange. I would never use /q comand on a legal char - what if mob spawns right on top of me? Thats a garanteed death

[So_1337]

I do it constantly. Granted, I don't know of any mob spawns in the Nexus =P

[Angelox]

I'm thinking if you are at the same IP of your server, you won't see any ghosting - you'll need people that log in from distant IPs and suddenly drop out.
Maybe some how this remains cached in the system and the server thinks there's someone still there, or maybe the server missed some packets with the sudden drop out.
could be they get LD on zoning, then the character arrives in the next zone, but the PC is gone.
Problem is, it's seems related to other players on your server. At one time, I remember having this problem when I ran a windows server. it was when I first started out. At the time I had a Pentium 3 with little memory and was playing on the server with the client on the same machine.
I remember seeing a lot of the ghosting effect. I figured it was related to lag , and lag created by multiple players zoning and logging in (LD).
I could play alone fine, but this would start when others logged in.
Also, I don't think this happens with multiple players on Minilogin - I think its related to the Public Login server.
Anyway, those were thoughts an ideas I had at the time, Went Linux and the problem for me ended with that. I still use a windows server for testing, but it's Minilogin, and I don't see any ghosting there.
You might want to try getting some friends to try you out on a Minilogin server.
Thing is, you can't have a solution until you can find what makes the problem, these are some things I'd try out, to see if anything were different.

Make observations when you see a ghosted character;
was it static or dynamic zone? was he zoning ? did he log? (you can find the account by the ghosted character name). See if you can find a pattern somewhere.

[paaco]

Seems like on my server I find myself ghosting a lot. Playing from the same machine the server is located on. It is on a static zone and happens if I right click EQ on the taskbar and close it out like that.

All it shows in the logs is:

[09.19. - 23:52:28] Starting Log: logs/eqemu_error_zone_1300.log
[09.19. - 23:52:28] Ghosting client: Account ID:1 Name:paaco Character:Paaca IP:192.168.1.3

[trevius]

The problem happens most commonly if people /quit or /exit, or if they go LD, but I am pretty sure it even has a rare chance to happen if they /camp. On a public server when I was running Windows, I would commonly see up to 15 or more ghosted characters in nexus at any given time of the day if I didn't boot them out manually every few hours or so.

The easiest way to tell 100% if a character is a player ghost is to turn on inspect so you can inspect people "/toggleinspect on" (you can test by inspecting yourself or another player you know isn't a ghost) then try right clicking a suspected ghost player and if the inspect window doesn't pop up, they are sure to be a ghost.

Even after they log off, you will continue to get net error messages from them in the log files. I believe the error that shows up is something like this:

Tried to write a packet beyond the end of the queue!

The error starts with showing incremented packet numbers, but eventually I think it just shows the same packet number over and over and over. And that error will continue to flood server logs even after the player has logged off and I think even after you #kick the ghosted character. So, it is like a connection isn't being brought down for some reason even if there is no connection. I think I have even had a player reboot their PC and it didn't stop the errors or disconnect the ghosted character. So, to me it seems like the issue may be that the server is just hung in some sort of loop.

More info on that log error can be seen in this post:

http://www.eqemulator.net/forums/showthread.php?t=24339&highlight=flooding

[KLS]

I don't get the packet flood with ghosting players. It's something with the netcode isn't timing out connections for windows in the zone server.

[trevius]

After reading through that post I linked above again, I notice that the error that seems to be generated when this problem occurs seems to be coded here:

EQStream.cpp
if(CompareSequence(NextOutSeq, seq_send) == SeqFuture) {
_log(NET__ERROR, _L "Tried to write a packet beyond the end of the queue! (%d is past next out %d)" __L, seq_send, NextOutSeq);
sitr=SequencedQueue.end();
continue;
}


And, looking into what that piece of code does, I looked at CompareSequence and SeqFuture so far. Here is the code for that command and I noted a part in RED that I think might be a mistake:

//returns SeqFuture if `seq` is later than `expected_seq`
EQStream::SeqOrder EQStream::CompareSequence(uint16 expected_seq , uint16 seq)
{
if (expected_seq==seq) {
// Curent
return SeqInOrder;
} else if ((seq > expected_seq && (uint32)seq < ((uint32)expected_seq + EQStream::MaxWindowSize)) || seq < (expected_seq - EQStream::MaxWindowSize)) {
// Future
return SeqFuture;
} else {
// Past
return SeqPast;
}
}

I think that line might need to be removed and replaced with this:

} else if (seq > expected_seq && (uint32)seq < ((uint32)expected_seq + EQStream::MaxWindowSize)) {

That should label all packets that are lower than the expected sequence as SeqPast. It will also label any any packets that are above the MaxWindowSize as SeqPast. And, if an ack is set as SeqPast, it should always SendOutOfOrderAck, which means it should correct most out of order issues I think.

Here is the MaxWindowSize setting code:

uint16 EQStream::MaxWindowSize=2048;

I will try to get the minor change noted above in and tested on my server and see if anything funky starts happening or if it has any noticeable effect. But, since I only run a Linux server, I won't see player ghosts anyway, so it might be pointless for me to test it. But, as long as it doesn't break anything, I could always update it in the new SVN and when Cavedude builds the Win32 binaries, windows servers could easily start testing and see what happens if anything.

Just from reading through the EQStream.cpp file, to me it looks like they are doing some sloppy stuff and maybe some extra work that isn't really needed. But, I am definitely no expert or anywhere near as skilled as the people who originally wrote that file. It seems like we could just check if an ack is in order and if not then always send an OutOfOrder ack. Maybe if we play with this file for a while, we can get most or all network issues worked out.

[KLS]

A lot of what goes on isn't because we want to do it that way. The client controls the netcode and we don't control the client. =p

[trevius]

Ya, but if we are getting out of order acks, doesn't that mean something is messed up and the server needs to compensate for it to recover properly? Obviously it is working for the most part or there would be more issues. But, I think a couple little tweaks could help resolve some of the issues like hung sessions, out of order acks, and whatever is needed to fix the Windows player ghosting issue.

[RhinoDude]

Trevius, did you end up making this change to your Linux server, and if so, did you note any adverse side effects?

If not, perhaps we could submit this as a fix, or find somebody who can test it under Windows? I will be setting up a Windows server eventually.

Also, is this TCP or UDP traffic we are talking about? If UDP the state of the client (rebooted or not, etc) would have zero impact normally. On the other hand, if we are talking TCP, then a difference in the TCP stack implementations could be the core underlying issue, and explain why it's visible only under Windows.

Edit: I forgot to add, I can't remember if EQ is UDP only or UDP + TCP. I've been away for 5+ years. ;-)

- Rhino

[trevius]

LOL, ya I tried making the change, but it didn't work right. I have tried a few other things and still no-go. My understanding of the network packets are less than what they should be lol. Maybe at some point this will get resolved by someone who catches the reason for causing ghosts only in Windows and not in Linux :P