New Concept: Proximity Spawning (to reduce bandwidth usage)

[AndMetal]

I wonder if we might be able to look at this a different way, especially since this seems to be more a bandwidth issue than a processing issue (which could also explain the issues with Warrior Rampage in AoE situations).

While roaming around in a zone, what kind of information is transmitted to the client on a regular basis? Spawn information, etc? Also, are there any kind of limits? So, is everything being sent to everyone, or are there proximity limits on these? If there aren't proximity limits, I think that's where we need to start.

Anyways, just a thought...

[KLS]

Spawn info is sent on login and new spawn. Or if a spawn has to change shape for some reason, typically walking around you wont see many spawn infos. Movement updates are the big thing... which if it's a mob far away they only send movement updates once every 60 seconds, close npcs ie ones you can see send movement updates every few seconds.

[trevius]

Maybe if there was a way to adjust how far away the regular movement updates get sent. I use MQ which helps a ton with developing content and standing in the main part of Dreadspire, I can see probably 50 mobs in the entire main section all moving very accurately and seemingly sending multiple updates per second. It seems to be somewhere around 800 distance away. I can only ever see a max of 400 distance in the entire zone since it is all indoors. So, if I could somehow set that range on a per zone basis, that might help a bit. Also, if it was possible, I wouldn't mind the option to completely disable the zonewide 1 minute updates on a per zone basis. I don't see why my client even needs to be aware of mobs outside of my visual range other than for MQ purposes, which really has no part in actual game play.

I know that the rule for it could be adjusted, but for some zones, the location information might be more important than others. And I know for a fact that those updates are pretty heavy on the server or bandwidth. At one point, I tried setting those zonewide updates in the rules to be every 1 second, because I didn't know what it was doing exactly. With it set that way, my server was so insanely laggy that it was unplayable after a few people logged on. Even 1 time per minute, if you multiply that times 60 clients on the entire server, you are averaging a zone wide update to 1 client per second. Double the players to 120, which is about the max my server can handle and it goes up to an average of 2 zonewide updates per second.

Then, once you start factoring in combat spam that has to go out to each client, a big raid can cause a significant hit to bandwidth and performance. If all of those players were all fighting in separate areas, they would only be receiving their own combat spam, but all in a tight area, everyone gets everyone's combat spam, so it multiplies what needs to be sent by how many characters are there.

I think the AE issue with rampage is more of a server resource issue than it is on bandwidth, since AndMetal brought it up. The main reason to think this is because it is one of the few things that can an will crash a zone if you have too many mobs in the rampage pull. I don't think bandwidth can cause a zone crash, just cause lag and LDs. Only high running processes or bugs can cause crashes, or at least that is my guess. I think a rampage pull of 30+ mobs is just too much combat spam all hitting at the exact same time for the server to handle at once.

I am sure there are some solutions to help this stuff considerably. Even a 20% decrease in bandwidth could make a considerable difference on home run servers that don't have practically unlimited upload speeds. Right now, I think players seem to average about 5k per player on normal use. Though, that probably spikes up a lot during large raids.

[trevius]

Woot! I think I found part of what KLS is talking about:

zone/updatemgr.cpp

//squared distances for each level
//these values are pulled out of my ass, should be tuned some day
const float UpdateManager::level_distances2[UPDATE_LEVELS]
	= { 50*50, 250*250, 500*500, 800*800 };
	
//delay between sending packets in each level, in ms
//its best if they are all multiples of UPDATE_RESOLUTION
//these values are pulled out of my ass, should be tuned some day
const int32 UpdateManager::level_timers[UPDATE_LEVELS+1]
	= { UPDATE_RESOLUTION,		//.3s
		2*UPDATE_RESOLUTION, 	//.6s
		3*UPDATE_RESOLUTION,	//.9s
		9*UPDATE_RESOLUTION,	//~2s
		34*UPDATE_RESOLUTION	//~10s
	  };

I like how he says they are pulled out of his ass and need to be tuned lol. If there was a way to set these on a per zone basis, I think that would be amazing. If that would be too complex, I think these could each easily just have rules tied to them so you can adjust them all manually to find which is the best combination.

I think more could definitely be done, but I will have to look into finding other parts that could possibly be tweaked further.

[So_1337]

We have already been testing a quest system to depopall() on all trash mobs when a boss mob or named is being killed. It seems to completely stop all lag as compared to previously people would go LD and lag extremely bad in some cases.

Just a quick note on that. We used to have trouble on our server before we had our DS3 line installed. We'd have too many players whacking on things in heavily NPC-populated zones. Killing Yelinak in particular comes to mind. Now, for Skyshrine, our trash mobs were already all set to a 30 minute repop. We'd do a #depopzone and then manually spawn him so we'd have a chance at killing him without us all getting bugged out. That worked out fine, unless we were slow and the zone repopped before then. At that point, the zone is completely overwhelmed, and everything will freeze for a good long while. 500+ NPCs in Skyshrine =X

Anyway, just my thoughts. I'm sure that if you're doing this, you've already thought of it, but wanted to suggest that if you use quest::depopzone that you make sure to disable the respawn timers until the boss has died, and then re-enable them. Though that only sounds useful for fights that aren't intended to have the added difficulty of add control =)

Just my thoughts, this sounds like a really good suggestion if a reasonable way to implement it can be found.

[kayen85]

Working with Trev on this,

We have as system set up atm that if an event is engaged, we do a zone wide depop of all trash mobs using quest::depopall(npcid) for each trash, then pop some KOS guard mobs in specific locations so people can't just run around the zone with a few others things built in to avoid abuse. When event is over zone repops all the trash with quest. This works fine. But obviously the flaws are there which is it really affects any other group in the zone forcing a 8-10 minutes of down time. Plus if you want to do an event with smaller force it could take even longer, and then if your moving from event to event quickly you could basically have the zone locked up for a while making it unplayable by others. Is that worth it so everyone doesn't have a 70% chance to lock up/crash when fighting the event, probably but its not the ideal solution.

There are other ways around it which are better, but just require more advanced planning and coding useing standard scripts, set up certain areas of the zone with only certain NPCid's then have quest mobs spawn and check aggro if any players are in that area and if its clear then depop it, otherwise leave it up and hope by the time they are done in the area the event is over.

But really all these solutions are cumbersome and were praying we can get something more solid.

[ChaosSlayer]

anoher thing: you think turning OFF Npc buff friends would help with the lag?
Since npc needs to not only buff all their friends but also keep a watch when their buffs expire etc

[KLS]

Mob::SendPosUpdate(int8 iSendToSelf) is where you would want to look. We have it hardcoded to 800 dist but it wouldn't be too hard to load that from the zone table and use a custom value instead. Turning off more advanced ai like spell casting and buffing when people aren't close to npcs and npcs aren't engaged is probably a good idea too.

[cubber]

This sounds interesting but I have a few questions to think about.

1. If this despawns mobs that are not in a certian radius of the player then couldn't this approach be used as an exploit? If a player is trying to camp a named mob and the mob is not up, couldn't the player just venture out of the place holder's radius to despawn it? Triggering a respawn when they come back into it's radius, essentialy cycling the camp without killing anything? What about named mobs like CT that are only supposed to spawn on a schedual?

Or will the "despawned mob" basically be "cached" until it is revealed again, killed, or cycled by spawn timer?

2. What about respawn times in general? Could this be used to get around respawn times?

3. Also in comparison to WoW, I think that the spawn radius would have to be quite larger than what they impliment. It is quite annoying to be able to see a few hundred yards away but not see any mobs in that area until you are almost right on top of them.

[Rocker8956]

This does not effect if the mob is spawned. Basically the mob is still there but the player does not get updates on the mob's position unless the player is within so many feet of the mob.

One question I have though, does this effect track?

[cubber]

Ahh that makes more sense then.

[trevius]

Cubber, the original suggestion to actually depop mobs that were out of range would have worked perfectly fine for everything accept Track. The spawn timers would still be checked before a mob would be spawned if a player entered the spawn radius. The only thing you would need to take into account to avoid possible exploits would be to remember that you don't want to depop and repop named or anything with particularly nice loot tables, especially if they are set to equip the items. As long as they don't equip the items, the players would still have no way to tell what loot was or wasn't on the mob, so even named could potentially be depopped. The main thing is that there would be a per NPCID setting to allow you to enable the spawn to use the system or just to stay up like mobs currently do.

But, it looks like this change may not even be needed anyway as there seems to be a much better way to accomplish what I was originally wanting to do just by adjusting values in the source for setting the radius that the updates are sent.

The last code changes I posted in this thread show how to adjust the source to help reduce bandwidth. As far as I can tell and have heard so far, even just making the changes I posted has made a HUGE impact (in a positive way) on the amount of lag and performance problems on the server. The only time my players have seen noticeable lag since the change was when my server broke 100+ players this weekend, which hasn't happened in a while. So, I am fairly confident that if we can make some code that allows each of these settings to be tweaked in the source, I think we can potentially make a great reduction in bandwidth utilization and allow more players on servers with lower upload speeds and with less lag.

I think we should have settings added to the zones table to set the QueueCloseClients settings for both warping to location and for walking animations. I also think we should have the options to be able to adjust all 4 of the level_distances2 settings individually as well as all 5 of the UPDATE_RESOLUTION settings. Of course there should be defaults set if the fields are left at 0, which would be default for the new fields. But, by giving this option, I really think high traffic zones could be tweaked enough to really make a big difference on any server.

Also, I have set zone wide updates to 10 minutes and so far I haven't noticed any different at all in game other than less lag from when it was set to the default of 1 minute. I will have to try running around in an open zone at GM speed to see if I notice anything weird. But, unless I am wrong, maybe we should allow the globalpositionupdate rule to have a setting that will completely disable zone wide updates. Maybe setting it to something like 0 or so could make it disabled.

Either way, I think we are on the way to increasing server performance for servers with limited bandwidth. I would love to see home servers able to reach the 200 or 250+ player counts that are seen on PEQ or the CEQ Dragon Soul server. I know PEQ is/was hosted in a Data Center which essentially gave it unlimited upload speeds and I assume Dragon Soul is as well.