ok, so I figured it out! I'm guessing my replies aren't being looked at in this thread anymore, but I'll update for the heck of it.
It probably seems obvious at this point, but I can reproduce the problem at will now. I didn't realize I was completely running out of memory. I thought I still had a pretty safe buffer. When I am nearly full and start another EQ instance, all the times shoot up from the 20-40ms range to 500+ and start climbing to nearly 30k.
I also want to point out that I created a new server using Windows Server 2008 R2 instead of Windows Server 2012 R2 and 2008 seems waaaayyyyyyyy more stable. I don't have half the problems I was having in 2012 and my setup/installation procedure was exactly the same. (only this time I took several VM snapshots in case anything breaks and I need to go back to a stable state).
Earlier, you mentioned that EQ shouldn't have this large of a memory footprint. Any chance you know how I might go about reducing the amount of memory each instance is using? Here's a screenshot of what my host machine performance looked like while everything was stable. And things were stable all day yesterday, all through the night while I slept, all morning. But at 31.5GB usage when I start a new instance and load up that last toon to push it over the edge. All in game response times shot as described above.
And here is a snapshot of the processes, showing how much memory each instance is using. How can I reduce this?
You also mentioned something to me about possibly switching to ROF2 client and I noticed you made that recommendation to several others as well. If possible, I'd like to keep the underfoot client. It has been very stable when playing on the existing EMU servers and I like the version of MQ that I have that runs with it. MQ is causing none of these problems by the way. eqgame.exe is using about that much memory even with MQ turned off. I'm afraid that if I switch clients, I might lose some of the useful plugins that I make regular use of and playing without MQ just doesn't work for me. I'd rather quit.
If you have any advice, I'd love to hear it. Going to google this problem for now.
Thanks.
Edit: That large CPU spike you see in the first screenshot is not normal CPU usage. I just happened to be loading up another instance when I took it. Idle, the CPU sits anywhere from 4-15% depending on what it's doing, even when I have 20+ toons logged in.