Gotcha. But if I want to start a new one, simply zoning isn't enough, correct? I need to start from the character select each and every time for a new capture?

I've not tested it (I will later), but in theory, if you start a new capture while you are already in a zone, the next time you zone, the extractor should recognise the new zone and generate the SQL for that one.

Another option may be to start the capture before you zone, and as you zone, watch the capture and find a packet you are sure is the new zone communicating with your client and then right click that packet and select Conversation Filter > UDP. That will cause it to filter out everything accept for that zone's communication to you. You can normally tell fairly easily which packets are the zone communicating to you when you zone by watching for a ton of packets coming in at the same time from the same IP to you. Also, it is good to note that if you scroll up a bit while collecting, it will remain at that part of the collect, so you can select your packet to filter easier. You can scroll back to the bottom after if you wish to continue watching them come in real time again.

Then, when you stop the capture and save it, be SURE that you select the option for "Displayed" in the "Packet Range" section of the Save As window. That will cause it to save only the packets you filtered to see. Otherwise, if you don't save it with that option, it still saves the unfiltered collect.

By doing this, it should mean you can start a collect, then zone, then collect and save it. Then start another collect and zone again and so on.

I haven't really tested this yet, but I am pretty sure it would work fine.

1.8 is available
This should address blackdragonsdg's crashes.

The last change (npc_types_tint) generates a row in the npc_types_tint table for each armor slot, so NPCs with different colored items in each slot would look a bit more correct
(although I don't think there are many like that). I wouldn't recommend you select this option unless you know there is a specific NPC with multi-colored armor that you want to setup right.

I don't believe there are any existing NPCs in the standard PEQ database that use the npc_types_tint table, so it may not have been extensively tested.

There is more info about this table here: http://www.eqemulator.org/forums/showthread.php?t=28647

OK, now I've tested it and it doesn't work so well on the current version, so I don't recommend starting a capture unless you are at character select,
unless, as Trevius suggests, you are comfortable with using Wireshark filters to just save the packets relating to the zone you are interested in.

As it stands now, starting a capture while you are already in a zone and then zoning will either cause the extractor to miss data when entering the new zone
(including crucial packets which make the collect unusable), or if a mob happens to spawn in the zone you are in, after you start the capture, but before you
zone, then that mob in the previous zone will be included in the data for the zone you are going to.

Haha, that sounds like the conversations in the Monk chat channel during a raid. Apparently monkey math is different from what everyone else learned.

Derision, you seriously rock. I did fib a little bit and found one more thing... Do doors still send teleport data? I know they used to, because that's how PEQ got most of its ports. I wasn't too concerned about it at first until I remembered about intrazone portals. I would make my life 100% easier if we could grab dest_zone, dest_x, dest_y, dest_z, and dest_heading.

I second this motion.

No, they don't send that info unfortunately, but, it may be possible to get it by waiting for clickdoor and then check where it sends them. Though, I imagine that would take a bit of work to figure out fully and also mean you would have to click every door and have it port you to get them.

Looking at a couple of examples (a teleport disc in the bazaar and one of the PoK stones in ... PoK), the door param is a reference to one of the zone_points,
so by cross referencing the data, I may be able to populate the relevant fields in the doors table for ports.

Yeah Cavedude mentioned it to me, and I was gonna post that the door data does appear to be sent as a zone in. I easily got port working with the nek book that way, I could of without but it saved me 2 minutes.

Wow, now that is a good catch! Makes sense, heh.

It doesn't appear to be the case in all zone, e.g. a couple of zone have a door_param of -1 for the PoK book with no matching zone_point.

Also the Arcane Disks in freeportwest have an open_type of 57 which is not handled as a teleport type door by the server,
but it is easy enough to go in and change the open_type afterwards, to 58.

In 1.9, for doors with opentype 57 & 58, if there is a zone point with an id that matches with door_param, then the destination zone and co-ordinates from the zone point
are used to populate the door fields.

It would appear that this tool still functions after the patch!

Woot! I still have work to do.

Seems pretty fool proof. It worked on Test Server last week with no changes. Thought for sure it would at least need new opcodes... nope!

I have an improvement request...

Could you add an ability for the program to list the merchant NPC lists that were not collected, so they can be added on a second run through the zone? I'm assuming this is possible of course since something in the SQL must denote a merchant NPC.

I just uploaded sharvahl to the log repository. Not sure if I got all of the merchants since there's multiple levels to just about every house. Such a feature would have came in handy!

I've started refactoring the code to (amongst other things) make it easier to support multiple patch versions concurrently, for when a patch does break it,
and I don't really want to make any other changes until I've finished and tested it thoroughly.

In the meantime, you can check for merchants who you haven't collected data for with this SQL
changing 383000 and 383999 to the start and end ids of the NPCs of the zone you are interested in.

I'll just wait for you to add it in... I'm not currently running my own server so don't have the ability to check.

Found a bug using v1.9.

If a merchant only sells one item, it is not getting recorded in the sql generated. It just lists nothing for him in the generated sql.

Broker Fahaar in sharvahl only lists 'Wedding Band', but it's not recorded in the generated sql.

The reason the Wedding Bands he sells is not being recorded is because he only has a quantity of 5 in stock. To distinguish between proper merchant items and items that
players have sold to them, the extractor only includes items that the merchant has an infinite supply of (quantity of -1 in the item packet).

I don't really know of a way to fix this without including all items that players have sold to mechants.

Maybe an option to include them? In a lot of older zones that I've been collecting in, there's been nothing on the merchants whatsoever since the zones are hardly used.

Include them in the SQL dump, but have them commented out and then when sourcing, the person could uncomment any lines he did want imported.

Ive posted the newest PEQ database up that has the output of this tool merged in. Most (but not all) of the zones supported by SoD are now spawned, with doors, objects, etc.

Also, Gaeorn wrote a tool to parse EQTrader's merchantlist data and dump it to SQL compatible with our system. I've posted the SQL output of that script on the SVN also. Beeping and warnings are normal while sourcing so pay them no mind. Overall, I've found EQTraders has very high accuracy, but I have found missing/extra items on a couple of vendors so it is not 100% perfect. Still in my mind it's a nice shortcut to fill in missing gaps so we don't have to go around to every merchant in the game when collecting :) Of course, backup your current merchantlist table before running this especially if you have custom merchants, as it may overwrite existing IDs.

Both can be had here: http://code.google.com/p/projecteqdb/downloads/list

Edit: I should mention if you are just running a stock PEQ database, then there is no need to source the merchantlists. They have already been merged with the newest database version.

Sweet, thanks Cavedude and everyone else!!! Great work!!

Can anyone use this to update cshome?

Could someone please tell me how to set up EQExtracter so that i am able to use it... For some reason im am unable to get it to work.... Help would be appreciated

One thing I did notice Derision since I started doing a few more dumps, in zones like Gyrospire most noticeably... The doors are four pieced like Plane of Time. So when you click the door pieces they all go up vertically, when they are supposed to criss cross outwards to give the 'mechanic' feel. Now I know that you can't reproduce the walk-up opening of all 4 pieces of the door unless you forcedooropen but if its something that can be fixed in how the packets are decoded I am just pointing it out, no big deal.

You need to install the .NET Framework 4 from Microsoft first, I think this is the right link: http://www.microsoft.com/downloads/d...displaylang=en

After that, install Wireshark http://www.wireshark.org/download.html

And finally download EQExtractor2 http://projecteqemu.googlecode.com/f...actor2-211.rar and use Winrar to extract the files into a folder of your choosing.

To capture packets for a zone you are interested in, get your character to that zone and then camp to character select.

Start Wireshark and in the Capture/Interface List on the left hand side, click on the name of the network interface that you are using to connect to the Internet.

Back in your Live EQ client, enter the zone. If you just want to collect spawns, you can stop the Wireshark capture as soon as your character appears in the world. To collect pathing, just sit around for a while, or if you want to collect merchant inventory, move around the zone clicking on merchants.

When you are done, in Wireshark, select Capture/Stop then File/Save As to save the captured packets into a .pcap file.

Fire up EQExtractor2, select File/Load PCAP (or press CTRL-L) and select the .pcap file you saved in the previous step.

You should get a message that the client version that was used to make the .pcap was recognised and also the name of the zone you where in.

Next, select File/Generate SQL (or Ctrl-S) and then on the next screen click on the Generate SQL button and give it a file name to save the generated SQL.

You can then use the mysql command prompt or Navicat or whatever, to source this into your database, however be aware that the generated SQL may delete data already in your database.

If you still can't get it to work, let me know what error messages, etc, you are seeing. With the latest EQExtractor2 version, pressing Ctrl-D will bring up a sort of debug-window which may have more detail of what went wrong.

its now saying The capture ession could not be initiated (Failed to set hardware filter topromiscuous mode). please check that "\Device\NPF_{E0FE483E-2611-47B2-8A77-C89F7F08CDDC}" is the proper interface.

Is it a Wireless adapter ? Apparently some wireless adapters don't support promiscuous mode.

You could try going to Edit/Preferences, then clicking on Capture on the left hand site and then unticking 'Capture packets in promiscuous mode' on the right hand side. You don't need promiscuous mode in the case where you are running Wireshark on the same PC as Everquest.

that worked.. thanks

If you want to email me the .pcap for Gyrospire, I'll take a look at it.

I wrote the following little player.pl for GyrospireB to open all pieces of the doors at once. The doorids are from the latest PEQ database, but I guess they should
be the same if you build the zone yourself from the extractor output:

Derision sir,

Damn good job. Sorry I didn't get back to you earlier...

If you don't mind me asking, what else do you have left for Shared Tasks, and is there anything that I can try to do to assist you with it.

Just wondering on the eqextractor is there a way to get the AA's from live?

You can get some of the data.

The server does send the data that could be used to populate the altadv_vars table (except the AA name).

The aa_effects data (for passive effects) that is sent is only for the first rank of the AA (or the rank that the particular character you are capturing data for purchased).

Finally, the AA action data (for activated AAs) i.e. which spell to cast when you activate the AA, is not sent at all (until you activate them via a hotkey).

You can dump the AA data that is captured to a text file (Ctrl-A on the latest version), but from there you would need to manually create the database rows and figure out
yourself what aa_effect/aa_action rows are required to complete the implementation of a given AA.

Ill have to mess around with it.. I'm still new with using this whole eqextractor and still learning what it can. Do.. So, it'll take some time.

If it works with Beta, it would be a good place to get AA data, since there's an NPC that grants you around 2000aa to test with.

Is this still working, I tried tonight to use the extractor and got a "unsupported client" big red error message after reading my pcap file.