EQExtractor2

**trevius** · #1 05-30-2010, 07:33 PM

I don't know the details of how the tool works exactly, but it seems like it parses the .pcap when you first load it. Perhaps you could just give an option to generate a file of that parse, which could then be loaded and used to generate SQL from. Figured that may be another possible option for creating files anyone could use and share without worrying about private info. Another nice bonus to using a parse instead of some generated SQL is that a parse should be able to hold data that may not currently be set to be generated into SQL yet, so as the extractor tool is refined/expanded, those same parses could be ran through the tool again for an instant update with the latest features. This should prevent people from having to run their own .pcaps through the tool and sending them again. Instead, Cavedude could just run the parses through the tool, and have the latest with minimal work.

**gaeorn** · #2 05-31-2010, 01:56 AM

Quote:

Originally Posted by trevius

I don't know the details of how the tool works exactly, but it seems like it parses the .pcap when you first load it. Perhaps you could just give an option to generate a file of that parse, which could then be loaded and used to generate SQL from. Figured that may be another possible option for creating files anyone could use and share without worrying about private info. Another nice bonus to using a parse instead of some generated SQL is that a parse should be able to hold data that may not currently be set to be generated into SQL yet, so as the extractor tool is refined/expanded, those same parses could be ran through the tool again for an instant update with the latest features. This should prevent people from having to run their own .pcaps through the tool and sending them again. Instead, Cavedude could just run the parses through the tool, and have the latest with minimal work.

Why not just have a tool that zeros out any personal info from the .pcap files so they could be sent in as is. Then the full communication between client and server is complete in case we figure out something additional to extract from it later, but it won't contain any personal information that a player may wish kept from the general populace.

**trevius** · #3 05-31-2010, 04:08 AM

Quote:

Originally Posted by gaeorn

Why not just have a tool that zeros out any personal info from the .pcap files so they could be sent in as is. Then the full communication between client and server is complete in case we figure out something additional to extract from it later, but it won't contain any personal information that a player may wish kept from the general populace.

I think the old collectors originally used to do just that, but since then, SOE has added quite a few more packets with character names, and probably other information, and it might be hard to track them all down. I suppose we could put together a list of all possible packets we may ever need collected, and just filter out the rest of the unneeded ones. As-is, I think Derision already has the most important packets being collected. As far as I can think, the only other one we might even want at this point would be OP_SpawnAppearance.

To make use of many more packets would require a different kind of collecting, which would be more interactive. The collecting this tool currently does best works when there is no interaction at all with the zone other than maybe checking merchants for merchant lists. Maybe in the future, other types of collects could be made that collect things like tasks, loot drops, spell sets on NPCs, or any other number of things that require considerably more time to collect enough info for it to be useful. Most of that stuff would probably be better off just adding it manually after a little research from various websites. Though, tasks might be one thing we could get enough info from without much time, but Derision would be much better to answer that.

**cavedude** · #4 05-31-2010, 09:15 AM

Just a minor suggestion, can the zone data and zone_points query be given checkboxes as well? If they are checked, they will be in the log uncommented, if unchecked they will be completely ignored?

**Derision** · #5 05-31-2010, 01:03 PM

1.4 is up in the downloads section:

Changes:

Code:

==05/31/2010==
Populated Melee texture columns.
For playable races, equip_chest2 is set based on the equipped chest item, if one is equipped.
Set armour tint columns based on the colour field for the chest slot.

The starting IDs for database inserts are now set as variables at the top of the SQL:
  Note the Door ID has changed meaning. Instead of being the value for the id column in the doors table,
  it is now the base value for the doorid column. The id column will just use the next autoincrement id.

If the version number is none-zero, The base doorid = Version * 1000 and all other insert IDs are
  set to ZoneID * 1000 + (Version * 100).

If the version number is none-zero, deletes for npc_types and the spawn tables will only be generated from
  the starting InsertID to starting ID + 99.

Moved the Version NumericUpDown control to before the InsertID textboxes as changing the version will
 auto-change the InsertIDs.

Corrected a bug where IT10805 objects were flagged as ovens instead of brew barrels.
Added checkboxes (default unchecked) for zone config (fog, safe point, etc) and zone points.

**Akkadius** · #6 05-31-2010, 03:27 PM

I've done multiple zones and seen the quality extract, but PoK for some reason is just completely jacked up. There are multiples of spawns, and all on the guard pathing and I have redone the collect 4 times after the initial one which I took about 20-30 minutes to do to get the 170 merchant lists in the zone.

Here is the .pcap and the .sql, let me know what you think.

http://www.sendspace.com/file/wclue5

**Derision** · #7 05-31-2010, 03:49 PM

My first thought is you probably have the original PEQ spawns for POK which use different ID ranges than my app generates, and so it doesn't remove them.

If you do select id, spawngroupID from spawn2 where zone='poknowledge'; and you get ids not in the range 202000 - 202999, then that is the issue. You will need to manually trawl through your DB and delete the original PEQ data for that zone.

If that's not the problem, let me know.

**Derision** · #8 06-01-2010, 01:17 PM

Quote:

Originally Posted by Akkadius

PoK for some reason is just completely jacked up.

I blew away PoK from my PEQ database and sourced in your SQL and it looks fine to me.

The only duped spawn I could see is that Veeshan Emissary guy which Live is constantly spawning and immediately killing.

I have found it is not unusual to see mobs pathing around buildings or walls to actually walk through them on occasion, which I believe is due to you being out of the range at which you get constant movement updates for them, at some point during the packet collect.

The only other thing I noticed was the player mounts that got collected. I'll fix that in the next release.