Ridiculous lag on Varlyndria with an 8 gig ram machine.

[Drakiyth]

My server is experiencing some horrid lag in certain zones with players running around with their 5 bots. The lag is so bad 2-3 players with bots (18 total characters) can crash it completely. There absolutely has to be something wrong here. The original EQ servers back in 1999-2000 probably didn't even have near 8 gigs of ram and T+ connection speed.. The machine and server host is solid as solid can be for this type of set up, yet it still crashes when people try to raid. (this goes for any zone). Is there anything I can personally do in the Server Rules Table or something that can fix this?


CURRENT ZONE SETTINGS

1 GM:MinStatusToZoneAnywhere 250 This setting overrides the minstatus setting in the zones table if set
1 World:ZoneAutobootTimeoutMS 120000 notation
1 Zone:GraveyardTimeMS 1200000
1 Zone:EnableShadowrest 1
1 Zone:ClientLinkdeadMS 180000
1 Zone:MQWarpExemptStatus -1
1 Zone:MQZoneExemptStatus -1
1 Zone:MQGateExemptStatus -1
1 Zone:MQGhostExemptStatus -1
1 Zone:EnableMQWarpDetector true
1 Zone:EnableMQZoneDetector true
1 Zone:EnableMQGateDetector true
1 Zone:EnableMQGhostDetector true
1 Zone:AutoShutdownDelay 5000
1 World:TutorialZoneID 272 notation
1 World:SoFStartZoneID 272 Sets the Starting Zone for SoF Clients separate from Titanium Clients (-1 is disabled)
1 Zone:UsePEQZoneDebuffs true Will determine if #peqzone will debuff players or not when used.
1 Zone:PEQZoneReuseTime 900 Amount of time, in seconds, before you can reuse the #peqzone command
1 Zone:PEQZoneDebuff1 4454 First debuff casted by #peqzone Default is Cursed Keepers Blight.
1 Zone:PEQZoneDebuff2 2209 Second debuff casted by #peqzone Default is Tendrils of Apathy.
1 Zone:MQWarpDetectionDistanceFactor 9.0 clients move at 4.4 about if in a straight line but with movement and to acct for lag we raise it a bit
1 Zone:NPCPositonUpdateTicCount 32 ms between intervals of sending a position update to the entire zone.
1 Zone:MarkMQWarpLT false
1 Zone:HotZoneBonus 0.2 hot zone XP bonus
1 World:GuildBankZoneID 345
1 Character:MaintainIntoxicationAcrossZones true If true, alcohol effects are maintained across zoning and logging out/in.
1 Zone:UsePlayerCorpseBackups true Keeps backups of player corpses.

[Irreverent]

Too many tweets and eqemulator spams? Haha, had to do it. ServerOps have to retain a sense of humor or stuff like this will make burn you out, especially as a n00b.

Are you using a repurposed machine? With those specs you shouldn't have any issues. If so, servers are like home machines. Every 6 months or so its best to whipe and start fresh every time with a new build with only what you need.

[chrsschb]

My original eqemu server (2008-2009) ran with only 8mb down/768kb up and was able to support 30 in one zone. The server itself was a repurposed laptop with a MASSIVE 1 gig of ram on a shit-tastic intel p4. The connection itself handled around 40-50 people as long as they were spread out.

My newest server is a AMD Phenom X4 overclocked, 8gig of RAM, etc on a 100mb down/10mb up connection and not even a single hiccup has been had and I've done some pretty crazy stress testing. I've AOE'd down entire zones. I've done raid testing with 30+ bots while I was also botting 6-7 real characters plus 2-3 other players 3-boxing.

Make sure you setup your database correctly to handle a ton of concurrent connections. Make sure your router/pc can handle so many concurrent connections. What connection speed are you running?

[Drakiyth]

Originally Posted by chrsschb

My original eqemu server (2008-2009) ran with only 8mb down/768kb up and was able to support 30 in one zone. The server itself was a repurposed laptop with a MASSIVE 1 gig of ram on a shit-tastic intel p4. The connection itself handled around 40-50 people as long as they were spread out.

My newest server is a AMD Phenom X4 overclocked, 8gig of RAM, etc on a 100mb down/10mb up connection and not even a single hiccup has been had and I've done some pretty crazy stress testing. I've AOE'd down entire zones. I've done raid testing with 30+ bots while I was also botting 6-7 real characters plus 2-3 other players 3-boxing.

Make sure you setup your database correctly to handle a ton of concurrent connections. Make sure your router/pc can handle so many concurrent connections. What connection speed are you running?

The problem right now is the server is set up on another rig that I have limited remote control over. Disconnects are killing the server right now and I can't even restart that PC. I could use mine but had the same problems yet I have the fastest comcast (non fiber optic) you can get which I believe is 2-3 up 1-2 mb down. I should be able to handle this server a lot better than this. None of it makes sense.

Chrssb, can you explain some things I may have missed using this guide
here to set up my server?

http://www.eqemulator.org/forums/showthread.php?t=32980

I followed that guide completely. Am I missing something somewhere where I don't have some concurrent connections set up properly or what?

[Secrets]

Originally Posted by Drakiyth

The problem right now is the server is set up on another rig that I have limited remote control over. Disconnects are killing the server right now and I can't even restart that PC. I could use mine but had the same problems yet I have the fastest comcast (non fiber optic) you can get which I believe is 2-3 up 1-2 mb down. I should be able to handle this server a lot better than this. None of it makes sense.

Chrssb, can you explain some things I may have missed using this guide
here to set up my server?

http://www.eqemulator.org/forums/showthread.php?t=32980

I followed that guide completely. Am I missing something somewhere where I don't have some concurrent connections set up properly or what?

Fastest comcast around here is 30down/5up. I was able to support 50-110 players without getting any latency issues.

You may need to look into dedicated hosting, although, if the machine isn't responsive, that may be a result of too much bandwidth. You may wish to install a bandwidth monitor to see what is actually causing the issue. It may be something else entirely.

Another question: What are the specs of the machine running the server? Do you see any "Zone has stopped working" on any zoneservers, or are you 100% sure it's even bandwidth related?

[Drakiyth]

Originally Posted by Secrets

Fastest comcast around here is 30down/5up. I was able to support 50-110 players without getting any latency issues.

You may need to look into dedicated hosting, although, if the machine isn't responsive, that may be a result of too much bandwidth. You may wish to install a bandwidth monitor to see what is actually causing the issue. It may be something else entirely.

Another question: What are the specs of the machine running the server? Do you see any "Zone has stopped working" on any zoneservers, or are you 100% sure it's even bandwidth related?

It's a Debian build, 3x3.0GHz Processor, 8GB DDR3 Ram, 10 down / 2 up internet connection.

According to Lord Adakos it has dedicated hosting already. Bandwidth monitor? Hmm okay. What else could it be if it's not bandwidth?

[bad_captain]

I have been running into this this past weekend as I was doing some testing. I have reverted back a few revisions and was still getting the issue. I was getting some extreme lag until I camped out. I reentered the zone without rebooting it (it didn't crash, just lagged), and was able to resummon my 54 bots with no problems.

One thing I noticed, when I got the lag, my server network utilization jumped up to 3-4% for a while, then jumped to 6-7% (100Mb) just before completely falling off (My client network utilization jumped up as well.). I have a 10 /2 connection, so 6-7% would just about be maxing my connection, since I am running my client within the same network.

After camping and entering zone again, I never went above 0.5% network utilization, even with the 54 bots running.

Something appears to be causing some massive network spikes that eventually caused either the zone to give up or something. I will try to test this out further as I get a chance.

This is on an old 2.0 Ghz, 1 GB desktop I threw together.

[image]

EverQuest traffic is based on UDP. Your issue is more likely network related than hardware at this point. Keep in mind that if you have some cheap home router you bought from Best Buy that would likely be your first problem.

After that it is more of the hops between you and whatever client is on your server. UDP traffic has no guarantee so we will retransmit until it is received - packets get dropped because there is some point from start to end that is not doing a good job handling the packets.

My guess is if you are running this at home, whether it be on cable or DSL you probably do not have a good ISP for 'hosting'. Infact I would venture to guess you have frequent latency issues where you live.

Also winblows network utilization is worthless -- it is giving you details over the LAN network utilization not your connection between the internet which is completely different and a much more limited connection speed.

[Cilraaz]

Originally Posted by Drakiyth

It's a Debian build, 3x3.0GHz Processor, 8GB DDR3 Ram, 10 down / 2 up internet connection.

According to Lord Adakos it has dedicated hosting already. Bandwidth monitor? Hmm okay. What else could it be if it's not bandwidth?

I had my test server up on a Debian server, dual core 3.16GHz CPU, 3.5GB DDR3 (32-bit OS), and 25 down / 25 up fiber business connection. The box's primary use is as a web/mail server. I was static loading 121 zones and dynamic loading 10 fallback zones.

With a similar amount of load (3 players and 5 bots each) in a single zone, I had no lag issues on a fresh build as of one month ago. Your server is more powerful than mine in pretty much every area other than bandwidth. I can't see 3 players using your full 2Mbit connection, though, so if it's latency as opposed to the server having load issues, it's probably somewhere in the network configuration.

Are you loading the zones statically (I'd assume so with 8GB RAM available)? Are you having any errors popping up in the logs? Have you nailed down where the "lag" is coming from, whether latency or server/software hiccups?

In any case, I'd probably note that 10/2 isn't going to be sufficient for a server with any type of community. When you're serving data like that, your upstream bandwidth becomes very important. 2Mbps probably isn't going to be enough bandwidth, and a home 10/2 connection usually isn't optimized for large amounts of upstream data.

[Drakiyth]

This was posted on my Varlyndria Forums.


UPDATE 05/23/12 - Stress test

Tonight I summoned everybody to the guildlobby and even force summoned people there as well. The entire server fought Sharphin in the Guild Lobby's East Wing and while they were fighting I summoned over 220+ NPC humans guards around them. This showed the server could handle A LOT of animations at one time with absolutely no problems. This test was done without bots.


CONCLUSION: This leads me to believe that bots have been causing a lot of the disconnection problems and since we know that now we're going to give it our best shot at eliminating this issue if possible.






This was a very successful test and it shows the server can handle a lot of animations at once. Not one person dysynced tonight. So, what could be causing the bots to do it?

[chrsschb]

Originally Posted by Drakiyth

This was posted on my Varlyndria Forums.


UPDATE 05/23/12 - Stress test

Tonight I summoned everybody to the guildlobby and even force summoned people there as well. The entire server fought Sharphin in the Guild Lobby's East Wing and while they were fighting I summoned over 220+ NPC humans guards around them. This showed the server could handle A LOT of animations at one time with absolutely no problems. This test was done without bots.


CONCLUSION: This leads me to believe that bots have been causing a lot of the disconnection problems and since we know that now we're going to give it our best shot at eliminating this issue if possible.






This was a very successful test and it shows the server can handle a lot of animations at once. Not one person dysynced tonight. So, what could be causing the bots to do it?

Bots pull from the up/down bandwidth a lot more than npcs do, but not as much as players do. I would (and I think this was mentioned somewhere else) put bots on a 3-to-1 scale with players. Bots will be a little higher on the system drag than network drag due to their AI.

[Secrets]

Originally Posted by Drakiyth

This was posted on my Varlyndria Forums.


UPDATE 05/23/12 - Stress test

Tonight I summoned everybody to the guildlobby and even force summoned people there as well. The entire server fought Sharphin in the Guild Lobby's East Wing and while they were fighting I summoned over 220+ NPC humans guards around them. This showed the server could handle A LOT of animations at one time with absolutely no problems. This test was done without bots.


CONCLUSION: This leads me to believe that bots have been causing a lot of the disconnection problems and since we know that now we're going to give it our best shot at eliminating this issue if possible.






This was a very successful test and it shows the server can handle a lot of animations at once. Not one person dysynced tonight. So, what could be causing the bots to do it?

I'm gonna be honest and say I haven't touched the bot code in ages. So if it is bots, I know nothing of that.

It's possible there's an infinite loop in bots, but without technical knowledge and remote access to the machine, it's kind of hard to tell.

A 3-core processor should still allow the system to function... so I really can't say at this point. Not enough information. I'd need to see the CPU of the zone with bots in it. If it's stuck at 33% cpu, then it's probably an infinite loop that was never cleaned up. None of the major servers run bots aside from THF, but they run so much custom code that it's hard to say.

[provocating]

More than likely it is the Bots. When I allowed more Bots per player a few years ago I noticed this issue. Basically the bandwidth will saturate the upload speed. This was back when I had DSL and sucky upstream. It only happened when a player would spawn over 4 bot groups.

We now have 50/5mb and zero issues but I also only allow 5 bots per character to make sure it is never an issue again.

[bad_captain]

I did some testing last night. I zoned into Yxtta, summoned my 9 groups of bots (54). After attacking a few mobs, I got the lag. Logged out and back in, summoned bots, got lag. Next time, I logged back in, didn't summon bots and still had the lag build up.

I did a packet capture and noticed a few things. The packets causing the issue weren't packets recognized by eqextractor, it was one packet back and forth : 12 bytes to server, 72 (if I remember correctly) back. If they didnt occur only when running eqemu, and with bots, i would think it wasnt eqemu related. I know I was debugging most of the time, but I dont think I was all the time.

Are you able to do packet collects? That might help to determine if it's the same issue I'm having.

When I did not have the lag isses, My network utilization high was around 0.25% , 6-7% with lag. I know exact numbers are mostly meaningless, bit relative differences are not. I was later able to fight to and take out the Primal Lizard with no lag issues, and everything seemed normal. I'll try to do some more testing this weekend, but will be out of town for most of it.

Anyone mess with packets enough to help me decifer these packets? I can figure out ports and what the actual data is, but cant tell if it's an eq packet, the opcode if it is, etc.

[bad_captain]

It may not matter, but I'm using SoD. I will test out UF, and maybe even check out VoA to see if the problem persists. Maybe something new for VoA is confusing SoD? Probably not, but you never know.