Sony EverQuest Loot Drop System Research Thread
This thread will contain the research I have collected regarding Sony/Daybreak/Whatever's loot drop system and will otherwise serve to preserve Sony developer comments so they are not lost.
In 2007 Sony developer Zajeer posted this to one of the Shadowknight bulletin boards: (note that I have added a lot of whitespace for readability) Quote:
In 2018 Ngreth outlined the treasure table fields by pasting data to a user to prove that the drop chance for an item is 75%. Each denomination of coin had its own probability/frequency roll and die rolls. It's interesting/strange to me that apparently coin is bundled with Sony's analog of lootdrops instead of a parent NPC loot table, but maybe I'm interpreting this wrong? Quote:
In 2017 Ngreth posted a changelog for the item Obsidianwood Sap, which shows the current frequency, die size and item percent. The item table/list also includes a second item (should be Fire Arachnid Silk) and the changelog shows the percentages for both. So we can know how the tables currently are and compare it to Magelo. Quote:
In 2018 Prathun pasted part of Quarm's treasure table: Quote:
In 2018 Ngreth pasted an item table/list from a Time Rallos treasure table. This describes one of RZ's drops. The significance here is that the Darkblade may only drop from this item table and that RZ has two other item tables for his other 2 drops and those do not contain the Darkblade in them, meaning multiple Darkblades from a single kill are impossible. The Sandals were a newer item added late into the expansion and it's likely that all drops in the table were 20% before it was added. Quote:
In 2018 Ngreth pasted an item table/list from a recent expansion NPC. Note the drop percentiles, which are either 12 or 13. This strongly suggests that integers are always used here. Quote:
I pulled the Magelo data to compare it with. Magelo only had 95 kills for this NPC, but it's interesting to compare with the actual probabilities: Iron Sentinel's Spike 151153 53 55.79% Living Iron Torque 151121 32 33.68% Aura of Sky Shield 151147 6 6.32% Blessed Shield of Iron 151146 7 7.37% Impervious Guardian's Buckler 151145 6 6.32% Scale Touched Bracer Facet 151851 19 20.00% Scale Touched Gloves Facet 151852 6 6.32% Scale Touched Shoes Facet 151853 4 4.21% Scale Touched Cap Facet 151854 11 11.58% Scale Touched Sleeve Facet 151855 5 5.26% Scale Touched Pants Facet 151856 7 7.37% Scale Touched Tunic Facet 151857 6 6.32% Golem Mud 152558 17 17.89% Minor Clawed Tablet of the Ring of Scale 151901 14 14.74% Lesser Clawed Tablet of the Ring of Scale 151902 3 3.16% Median Clawed Tablet of the Ring of Scale 151903 5 5.26% Greater Clawed Tablet of the Ring of Scale 151904 3 3.16% Glowing Clawed Tablet of the Ring of Scale 151905 2 2.11% Woodcarved Symbol 149196 1 1.05% Woodcarved Ocean 149197 1 1.05% Woodcarved Thunderstorm 149199 2 2.11% Woodcarved Flower Bulb 149200 1 1.05% Woodcarved Butterfly 149202 1 1.05% In 2018 Ngreth outlines how the logic works. Note he says 'a d100' which more or less confirms the integers used in item tables. Quote:
In 2018 Prathun mentions how global loot worked: Quote:
2018 Ngreth post: Quote:
A 2021 Ngreth post. This seems contradictory to the previous post. 'this is NOT a treasure table' sounds like NPCs can be designed to spawn with worn gear that has a 100% chance to exist on spawn and this is not a treasure table. If that is the case then I don't understand the 22 number. Quote:
|
The key thing to understand is that Sony uses a XdY die system which is actually quite simple and flexible. EQEmu's system/schema cannot replicate Sony's output in all cases and is also less intuitive and more complex.
Treasure tables and item tables are essentially two components of EQEmu's singular lootdrop table. The treasure table contains the "frequency" which is the probability, plus the die size (which EQEmu lacks), die rolls (multiplier) and item table ID. Separating the tables this way allows item table reuse. EQEmu's analogs would simply be probability and muliplier; also I suppose mindrop and droplimit which Sony does not have. EQEmu does a probability roll for each multiplier iteration however so the functionality is not identical. When you reuse lootdrops in EQEmu, each master NPC loot table has their own copies of these lootdrop values to allow flexible reuse like Sony does. EQEmu's lootdrop primarily differs in how multiple items can drop from a single lootdrop. EQEmu has multiple ways/modes of interpreting the table list. The original way lootdrops were handled was to simply roll each item individually and possibly drop each one if the roll succeeded. Sony's way adds up all the drop chances which must add up to 100% then rolls once to determine which item drops. EQEmu added mindrop and droplimit to ensure a drop or enforce a maximum. Unfortuneately these parameters did not result in actual drop rates that matched the input drop percentiles and often resulted in highly unpredictable drop rates, so new ways of processing the tables had to be added. I bought up that issue in 2015 and KLS implemented a solution which improved things. KLS essentially added two new table processing modes. Instead of rolling each item individually, when mindrop>0 it will instead add up all the item percentiles together and do a single roll against it all and that roll decides the item, which is done mindrop number of times. This is much closer to how Sony does their tables. There is the minor issue of this being unintuitive in that you will end up with duplicate items when you would not expect them when you set mindrop>1, but it's otherwise more desirable because drops rates will end up more predictable. This is essentially a different table 'mode' because it's running an entirely different loop, but nothing in the editor form indicates this so it's unintuitive. If droplimit > mindrop, ANOTHER loop is run after the previous loop, or is run in place of it if mindrop=0. This loop does a single roll against either the sum of the item chances or 100, whichever is higher. In that way the roll can allow the possibility that nothing drops while ensuring drop chances are as they are input. This is effectively a third table mode. It works well if drop limit is 1 and mindrop is 0; however this creates some issues otherwise: 1) It becomes impossible for more than one item to drop at the same time without re-running the loop, but rerunning the loop is counterintuitive when not handling mindrops; after all you wouldn't re-run the loop when mindrop and droplimit are both 0. 2) If you re-run the loop, you end up with too many drops; sometimes WAY too many drops depending on the table parameters. 3) If you re-run the loop, you can end up with mindrop+1 drops of the same item (items with multiplier set to 1) which makes no sense. 4) If mindrop>0, then the dropped items average is higher than expected; i.e. mindrop=1 droplimit=2 will drop significantly more items on average (even while the limit is enforced; varies depending on the item drop rates) than mindrop=1 droplimit<=mindrop. In 2020 I implemented a somewhat crude solution for TAKP to mitigate the above problem by implementing a fourth loop which is run if droplimit is 2+ above mindrop. It is essentially the loop that would have been used if we were not ensuring that drop chances are literal to the table data. It randomizes the starting point of the items and rolls against them individually and exists once the limit is reached. This mostly works because the higher the droplimit above mindrop, the closer the results will be to the literal table percentages, so drops using +2 droplimit will be much closer to the table % than +1 droplimit. This does not fix problems 3 and 4 however, but will prevent high droplimits from creating a ton of items. I see that in 2021 noudess did something to the drop code but he did not document the change and it's too much of a headache to try and figure out. I can't say what EQEmu's current state is exactly. Just writing this post was a headache. You can see how complicated this got just trying to make the actual drop rate match the drop chance input by the content developer. Even after all that, it still isn't accurate in all cases. Best practice for content developers would be to avoid having a droplimit higher than mindrop if mindrop is non-zero and to not have a droplimit above one if mindrop is 0. Try to make drop chances in lootdrops add up to 100% and use multipliers and probabilities to drop multiples. I'll use Dozekar as an example to illustrate the primary issue EQEmu has in being unable to replicate Sony's drop rates. Dozekar can drop between one and seven tear items. The chance to drop any given number of tears should be about 14.3% each (100/7 or 1d7). But the only way to try and replicate this with EQEmu would be to create two identical lootdrops of the tears, make one table a 100% single drop, and the other table 50% probability with multiplier 6. However if you do that the result will end up like this: Kills with 1 total drops: 15334 (1.53%) Kills with 2 total drops: 94001 (9.40%) Kills with 3 total drops: 234776 (23.48%) Kills with 4 total drops: 312521 (31.25%) Kills with 5 total drops: 233834 (23.38%) Kills with 6 total drops: 94101 (9.41%) Kills with 7 total drops: 15433 (1.54%) That was the result of a loot simulation script I wrote simulating 1,000,000 kills. This problem occurs for any dX roll of 3 or higher, so will be a problem for items like spider silk. So not only do you need to make two tables/lootdrops, but you cannot get accurate results even with two tables. For TAKP I had Cavedude implement a 'multiplier minimum' field to lootdrops years ago which elimiated the need to make two tables, but the unequalness in drops probability remains for TAKP. This was not a good solution and I now know that a better solution would have been to add a boolean field which when toggled on would randomize the multiplier value when the NPC spawned from 1 to X. Even this would still only work if the probability is 100% because EQEmu does a probability roll for each multiplier, unlike Sony. That is also why simply adding a new integer field to serve as a die size wouldn't work for all cases. |
How can we improve our drop tables now that we know how Sony's system works? I have a couple of examples.
In 2017 Ngreth posted a changelog for the item Obsidianwood Sap, which shows the current frequency, die size and item percent. The item table/list also includes a second item and the changelog shows the percentages for both. So we can know how the tables currently are and compare it to Magelo to get a better idea of Magelo's bias. Quote:
The treasure table item_frequency is 40%, meaning the NPC has a 40% chance to check the item table. The treasure table item_die_size is 2, meaning it's 1d2 meaning it has one guaranteed drop if the frequency roll succeeds and a 50% chance for a second drop. The item table has two items. The item ID is not shown but given the drop chance this drop has to be Fire Arachnid Silk. Since the item table is 1d2, this means there should be a very small chance to get two saps from one mob. I checked old Al'Kabor logs for this and did find one case of somebody looting two saps within 1 second of each other which pretty much confirms it. Somebody could park a character in the zone with a merc to confirm this as well but I'm satisfied with the log I have. Magelo drop rates for sap shows 5%, and for the silk 34%. Magelo rounds down for these drop percentiles, and manually calculating Magelo's data shows the drop rate about 5.5%. I've checked a lot of Magelo pages and I can say that Magelo often (but not always) has a negative bias for drops which is often around -5 to -10%, so this is pretty much as expected. This bias is clearly seen when checking drop rates for items that have a 100% drop chance but Magelo lists ~90-95%. Also Magelo counts all drops of multiples of an item as one. For silk the real drop rate (assuming my math is correct) should be 37.8%. (0.4*0.9 + 0.4*0.5*0.1*0.9) For sap the drop rate should be 5.8%. (0.4*0.1 + 0.4*0.5*0.9*0.1) --- Recently I improved the drop rates for some tradeskill items on TAKP. These items were Swirling Shadows and Shadeling Silk. Swirling Shadows has a magelo drop rate of about 39% in Twilight Sea. I have some old AK logs of me farming these; I obtained about 2,000 of them and the average loot per kill was almost exactly 60%. Since these have a 1d2 item table die as they have a 50/50 chance to drop 1 or 2, this strongly implies a 40% chance frequency roll. It would make some sense for the Magelo bias to not be as bad there because the only reason to be there is to farm these things. Shadeling Silk has a die size of 3, meaning 3 can drop from a kill. Magelo lists drop chances of 34% to 44% for this. I checked my old AK logs of me farming this. I killed 364 lesser shades which resulted in: 56 dropped one silk; 57 dropped 2 silks; 39 dropped 3 silks; 152 (41.7%) dropped one or more silks. I also killed 508 xakra larva which resulted in 419 shadeling silks. That is a loot per kill of 82.5%. A frequency roll of 40% with a 1d3 item die would result in a loot per kill rate of 80%. It actually makes some sense for Magelo's data to be less accurate here because with a larger die size comes more streakiness. It's worth pointing out that often the higher drop rates on Magelo are from weaker NPCs, such is the case with Shadeling silk. This likely is the result of Sony copy and pasting the treasure tables along with the item tables for the NPCs and the lower level NPCs ending up higher due to margin of error. We can however see examples of Sony using different treasure tables for the same item table for tables of high level gems; i.e. Karnor's and City of Mist NPCs will not drop Fire Emeralds and such at the same rate as Plane of Fear NPCs even while using the same table. --- Since item tables must add up to 100% and Sony probably always used integers here, and presuming that Sony generally used multiples of 5 for frequency (it would be weird not to), this helps us improve the accuracy of drop tables. There is an item table containing the Mithril Amulet item which drops items meant to be sold to vendors for money. By checking out multiple NPCs that drop Mithril Amulets we can see that the item table is: Pearl Necklace, Fire Opal, Star Ruby, Fire Emerald, Sapphire, Emerald Ring, Opal Bracelet, Mithril Amulet I added up all the drop percentiles for these items on several NPCs with a high drop chance for this table and the results were about 13-14%. It seems likely that Sony had a 15% frequency/probabiliy for this table for NPCs known to have dropped them commonly, such as planar mobs. (as mentioned above, some zones dropped these with a lower frequency) We can attempt to get estimates for each individual item table drop percentile by dividing 1 by the sum of all of the table's drop chances and multiplying by the individual chance. Seafury Cyclopses in Ocean of Tears have the highest number of kills on Magelo, so that is the best NPC to look at: a seafury cyclops; item table sum = 13.66% Pearl Necklace 1.80% 13.14% Fire Opal 1.97% 14.44% Star Ruby 2.06% 15.08% Fire Emerald 1.58% 11.58% Sapphire 1.46% 10.69% Emerald Ring 2.07% 15.13% Opal Bracelet 1.42% 10.41% Mithril Amulet 1.30% 9.52% A lot of these values are close to either 10% or 15%. Sony's table may have used 10 and 15 values. The more valuable drops are the 10% values which makes this more plausible. Although Pearl Necklace and Fire Emerald are off by enough that those may have been different. I checked other tables and Fire Emerald was also significantly higher than 10% on those as well but randomness is notoriously streaky and narrowing down stuff to within one percent requires a LOT of trials, so some guesswork will always be required. Incidentally I checked the Ruby Crown table as well and those are uniform at around 11% each. |
I wrote a lua script to scrape drop data from Magelo NPC pages. The output is formatted to be pasted into spreadsheets, where I organize it in a way to try and figure out Sony's item tables and get a better idea of drop rates by comparing drop chances between multiple NPCs. It's also a way to preserve data in case Magelo goes offline.
The script requires some lua libraries to execute. I am using a lua distribution called 'Lua with batteries' from luadist.org but unfortunately the website in the last few months expired and a domain squatter took it. If you trust me I put it on my Google Drive, but if you don't want to trust me you can try to find the libraries yourself. https://drive.google.com/file/d/1AdG...usp=share_link I would paste the script in a CODE block here but the forum software is choking on it for some reason. So here's a pastebin link: https://pastebin.com/WXxwL85T |
| All times are GMT -4. The time now is 08:45 PM. |
Powered by vBulletin®, Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.