PDA

View Full Version : EQExtractor2


Derision
05-22-2010, 01:49 PM
I've been tooling around with Visual C# 2010 the past couple of days and one thing led to another and I wrote an EQ Packet decoder and extended it to dump out SQL statements for Doors and NPC Spawns.

I collected data for the revamped highholdpass and freeport west zones and thought I'd post the SQL for anyone interested:

<Link removed. This data is now incorporated into the PEQ database>

Note that this is just doors and static spawns, no grids or anything, and the mobs will spawn wherever they happened to be when I collected the data. It's all basic stuff, but I thought I'd throw it out there.

Akkadius
05-22-2010, 03:32 PM
Although this seemed to work fine, in West Freeport I could really notice that some NPC's were cut off beyond a certain radius. NPC's from out by commonlands and the temple were non-existent, as well as some other NPC's in the outer perimeter.

Are you in the middle of developing this packet collector further hence just the SQL files?

Thanks for this Derision.

Derision
05-22-2010, 03:55 PM
Although this seemed to work fine, in West Freeport I could really notice that some NPC's were cut off beyond a certain radius. NPC's from out by commonlands
and the temple were non-existent, as well as some other NPC's in the outer perimeter.


It would help a lot if you could give me a #loc where you are standing, so I can go on live and compare the mobs I collected with what are on live and see what are missing.

It maybe that Live only sends mobs in a certain radius of the player, so I need to run around the zone to get them all.


Are you in the middle of developing this packet collector further hence just the SQL files?


This is my first C# program of any size, and certainly isn't anywhere near release quality (hard coded file names, very fragile etc.). I just thought it was cool to see spawns in those
two revamped zones which is why I posted the .SQL for them.

Akkadius
05-22-2010, 04:21 PM
It would help a lot if you could give me a #loc where you are standing, so I can go on live and compare the mobs I collected with what are on live and see what are missing.

It maybe that Live only sends mobs in a certain radius of the player, so I need to run around the zone to get them all.



This is my first C# program of any size, and certainly isn't anywhere near release quality (hard coded file names, very fragile etc.). I just thought it was cool to see spawns in those
two revamped zones which is why I posted the .SQL for them.

Aye I was just wondering where you were on this because I could have alot of fun with it.

And yes, I am pretty positive for lag reasons there is a certain radius that you cache info wise. But from what it looked like in freeport is that you packet collected from the center. Basically every mob outside the same radius is missing. I would just run your query to check it out, I pulled it up on a MQ2 map read to see all the mob positioning. And yes, I only use MQ for map purposes.

joligario
05-22-2010, 07:38 PM
Yeah, on live they implemented an anti-MQ2Map feature a while back where they now only send data on NPCs within a certain radius.

trevius
05-22-2010, 08:49 PM
Well, that is only halfway true. It does help cut down the bandwidth usage and the usefulness of ShowEQ and MQ2 Maps. But, they do still send zone-wide spawns, they just don't send position updates after that point, and maybe not even new spawns, I am not sure. But, if you zone into Blightfire for example, you will get spawns for the entire zone on ShowEQ, so I am pretty sure it still sends all of them on the initial zone in.

Derision
05-23-2010, 12:16 PM
I had a look at freeportwest with MQ2Map and also moved around the zone and I didn't see any obvious missing spawns.

As Trevius says, Live seems to send all the spawns when you zone in, then for mobs within a certain radius, you get rapid position updates for them (with deltas),
and for mobs outside that radius, you get less frequent updates.

I've extended my collector to create grids. Here is an updated .SQL for freeportwest with grids:

<Link removed. This data is now incorporated into the PEQ database>

You will see that for mobs inside the city, near to where I was, e.g the patrolling knights and guards, there are lots of grid entries, because the position updates come every couple
of seconds, whereas for the mobs outside the gates (snakes, bats and the like), there are considerably fewer updates.

Automatic grids created from packet collects are never going to be much good for that reason, although it does give the world builder a starting point I suppose.

Anyway, I guess the next step is for me to get the tool to the point where it is ready for release as an executable at least, so other people can get to play with it.
Probably have time to do that next weekend.

I should add that while in theory it should be possible to have it record a live EQ stream as you are playing, the way I have it setup is to use capture files collected with Wireshark.
So you go to char select, start Wireshark capturing, zone in with a character, come back 20 minutes later stop the capture, save the captured packets and run the extractor
on the .pcap file.

robinreg
05-24-2010, 01:29 AM
This version of highpasshold would be good to replace the old highpass for the SoF/SoD that currently has problems with it. I tried replacing it with the new version in PEQ database. Only problem I am having is the zone point to zoning out of HighPassHold. I can zone in from Kithicor, Keep and East Karana but I can not get the zone out to work.

Akkadius
05-28-2010, 10:55 PM
Derision, would you please please please packet collect the doors/objects for Oldhighpass? Could that be doable? Or are you getting closer to a working version of this for others?

Derision
05-29-2010, 03:22 AM
Derision, would you please please please packet collect the doors/objects for Oldhighpass? Could that be doable? Or are you getting closer to a working version of this for others?

Is oldhighpass accessable on Live ? It seems like it should be part of Seeds Of Destruction, but I don't see it listed here: http://everquest.allakhazam.com/db/zone.html?zcont=Seeds+of+Destruction

Anyway, assuming it is a SoD zone and it has pre-req tasks for entrance, I don't have any way of getting in there.

I'll try and put a version of my tool out some time this weekend.

Akkadius
05-29-2010, 04:03 AM
Is oldhighpass accessable on Live ? It seems like it should be part of Seeds Of Destruction, but I don't see it listed here: http://everquest.allakhazam.com/db/zone.html?zcont=Seeds+of+Destruction

Anyway, assuming it is a SoD zone and it has pre-req tasks for entrance, I don't have any way of getting in there.

I'll try and put a version of my tool out some time this weekend.

I had a feeling it wasn't either, I was about to check myself 8 /

Derision
05-29-2010, 02:25 PM
I've uploaded the executable here: http://code.google.com/p/projecteqemu/downloads/detail?name=EQExtractor2.rar&can=2&q=

Prereqs: .NET Framework 4, a Live EQ account and Wireshark to capture the .pcap files that this program processes.

When capturing with Wireshark, ALWAYS start the capture with your character at char select, then zone-in once the capture has started.

The program will only process data for a single zone, so if you move to another zone, only the data for the first zone you entered after starting the capture will be processed.

This is still very much a work-in-progress, but feel free to report any bugs or suggestions here.

cavedude
05-29-2010, 08:03 PM
Tried it out on moors, works perfectly! You can see the end results in the next PEQ database :)

trevius
05-29-2010, 09:07 PM
Tried it out on moors, works perfectly! You can see the end results in the next PEQ database :)

Haha, I tried it out on the same zone and yeah, it worked pretty awesomely. I have a few pcaps saved from random collects here and there and figure I might as well use those through this tool first, then will probably start collecting more zones later. Excellent work as always, Derision :D I can't wait to play with this a bit more. It will definitely save me a ton of time on doors and ground spawns at the very least.

Some other info that might be good to pull would be zone points and maybe even newzone for zone settings, since most of the new zones are just copy/paste of a single example in the zone table. Now that the zone table can have multiple entries for the same zone, I think we could just add an id field as the primary key.

Akkadius
05-30-2010, 12:42 AM
I've uploaded the executable here: http://code.google.com/p/projecteqemu/downloads/detail?name=EQExtractor2.rar&can=2&q=

Prereqs: .NET Framework 4, a Live EQ account and Wireshark to capture the .pcap files that this program processes.

When capturing with Wireshark, ALWAYS start the capture with your character at char select, then zone-in once the capture has started.

The program will only process data for a single zone, so if you move to another zone, only the data for the first zone you entered after starting the capture will be processed.

This is still very much a work-in-progress, but feel free to report any bugs or suggestions here.

There are 655 npc's in Dragonscale Hills. Simply amazing Derision. Wow.

Akkadius
05-30-2010, 02:08 AM
I've done about 20 live zones so far at about 1-2 minute captures. I will share them when I get a decent collection.


One small suggestion: Having the option to add version type in the SQL create so you can capture and use those captures for different versions (Instances) for everything spawn2/objects/doors

Other than that this is phenominal.

Derision
05-30-2010, 08:48 AM
Version 1.2 is up: http://code.google.com/p/projecteqemu/downloads/detail?name=EQExtractor2v1.2.rar&can=2&q=

Changes:

Version field in the UI to populate the version field of any tables that have a version column.

SQL to update the zone table from values in the OP_NewZone packet, as well as to insert zone_points is automatically generated at the top of the output file, but left commented out.

The 'findable' column in npc_types is now set correctly.

Looking forward to your Dragonscale Hills .SQL, Akkadius ... never been in that zone on live, so will be interesting to see it populated :)

steve
05-30-2010, 09:27 AM
Awesome work, Derision! You never cease to amaze me with all of the quality work you put out. We're lucky to have you!

I can't wait until I get home from work and can start collecting! All I need to do is install WireShark and then your application to get started?

Derision
05-30-2010, 10:03 AM
The extractor app is written using C# 2010 Express and needs .NET Framework version 4 which you can get from http://msdn.microsoft.com/en-us/netframework/default.aspx (Click on the Download .NET 4 link on that page).

Akkadius
05-30-2010, 11:37 AM
I've populated about 20-25 zones from SoF to SoD. I will be more than glad to post them here. And I will continue to keep on populating.

steve
05-30-2010, 01:04 PM
I've populated about 20-25 zones from SoF to SoD. I will be more than glad to post them here. And I will continue to keep on populating.

Did you verify them in-game too? :) Anxious to see this in action!

cavedude
05-30-2010, 01:08 PM
Found a problem, spawngroup is using the npcid for its id instead of the one you specify. The funny thing is, spawnentry is using the proper spawngroupID (meaning, the one you specify, and not the npcid.) In my testing, all other IDs look to be OK.

Just a friendly word of warning, if you are building zones against the PEQ database, the program defaults often conflict with existing data, which is destroyed when you source in the output.

Nevertheless, this is the most exciting thing to come out in a long time for EQEmu in my mind.

Akkadius
05-30-2010, 01:12 PM
Did you verify them in-game too? :) Anxious to see this in action!

Yes I did verify this, I've dumped all of them. Beauty of the program is too is that you can always recapture and redump.

Nevertheless, this is the most exciting thing to come out in a long time for EQEmu in my mind.

I fully agree. I believe we should get together a collection of dumps and keep them posted somewhere.

Derision, I am very thankful of your contributions to this community. Likewise every other developer.

Another suggestion: Incrementing the ID's by adding a hundred or something for versions. For example, Zone 6 (Highkeep) Version 1, instead of 6000 for all the increments, have it be 6100 or something of that nature. Another thing, when you export another version, the door ID's need to increment differently, otherwise you are going to get conflicting doors. Version 1 needs to change the door ID increments to something like 1000 cause I know there are some zones with a couple hundred doors.

steve
05-30-2010, 02:12 PM
Just a friendly word of warning, if you are building zones against the PEQ database, the program defaults often conflict with existing data, which is destroyed when you source in the output.


What would be your suggestion in order to keep everything compatible and to be able to submit it to PEQ?

Also, this just gets doors, zone entries, NPC entries in-game. What about adding in special attacks, spells, loot, etc? What's the easiest way to help with this?

Akkadius
05-30-2010, 02:13 PM
What would be your suggestion in order to keep everything compatible and to be able to submit it to PEQ?

Also, this just gets doors, zone entries, NPC entries in-game. What about adding in special attacks, spells, loot, etc? What's the easiest way to help with this?

He did say it was just an early version, I am sure he is going to add quite a bit more to it.

EDIT: For those who want it, here are some zones that I collected, not all of them because I did some for personal.

I wouldn't say this is PEQ standard, but doors are all there, spawns, pathing for about 1-4 minutes. Merchant lists as much as possible.

http://www.sendspace.com/file/ivch22

steve
05-30-2010, 02:16 PM
He did say it was just an early version, I am sure he is going to add quite a bit more to it.

Oh, I understand that :) I was talking more procedurally, for submitting to PEQ. Would I need to hunt through my year+ of eqlog files and search for NPC names to find any spells they used and manually add them? Just wondering how this works?

Derision
05-30-2010, 03:04 PM
Found a problem, spawngroup is using the npcid for its id instead of the one you specify. The funny thing is, spawnentry is using the proper spawngroupID (meaning, the one you specify, and not the npcid.) In my testing, all other IDs look to be OK.

Hopefully that is fixed in v1.3 (just uploaded it).

Akkadius
05-30-2010, 03:14 PM
There's got to be a way we could manipulate as script to pull raw loot tables from Lucy. And modify from there.

Tricyclethief
05-30-2010, 03:50 PM
Anyone able to get the rest or more of the sod zones? Would really love to see more sod stuff in eqemu.

Akkadius
05-30-2010, 03:52 PM
Anyone able to get the rest or more of the sod zones? Would really love to see more sod stuff in eqemu.

Working on them

cavedude
05-30-2010, 04:01 PM
There's got to be a way we could manipulate as script to pull raw loot tables from Lucy. And modify from there.

Gaeorn is working on that, actually. He did merchant lists, though there is a major bug with the output I need to talk to him about before the sql released.

Hopefully that is fixed in v1.3 (just uploaded it).

Yep, Derision that fixed it, I can now build! I did find one more tiny problem, though. You'll need to add a doorid field next to version. The doorid currently always starts at 1, and that will conflict if the zone already has doors in another version.

What would be your suggestion in order to keep everything compatible and to be able to submit it to PEQ?

Also, this just gets doors, zone entries, NPC entries in-game. What about adding in special attacks, spells, loot, etc? What's the easiest way to help with this?

To merge into PEQ is pretty tough, because for everything except for NPCs, we use the next available ID. If you feel comfortable, you can send any pcap logs you gather to me at npcswithoutfaces@gmail.com. Otherwise, I don't know a way for multiple people to successfully merge into PEQ, especially since our IDs are going to change QUICK now. SQLs won't help much in this case because the IDs will almost certainly conflict.

Akkadius
05-30-2010, 04:13 PM
Gaeorn is working on that, actually. He did merchant lists, though there is a major bug with the output I need to talk to him about before the sql released.



Yep, Derision that fixed it, I can now build! I did find one more tiny problem, though. You'll need to add a doorid field next to version. The doorid currently always starts at 1, and that will conflict if the zone already has doors in another version.



To merge into PEQ is pretty tough, because for everything except for NPCs, we use the next available ID. If you feel comfortable, you can send any pcap logs you gather to me at npcswithoutfaces@gmail.com. Otherwise, I don't know a way for multiple people to successfully merge into PEQ, especially since our IDs are going to change QUICK now. SQLs won't help much in this case because the IDs will almost certainly conflict.

A loot table script would be phenomenal. I wouldn't mind helping with this.

Derision
05-30-2010, 04:21 PM
our IDs are going to change QUICK now. SQLs won't help much in this case because the IDs will almost certainly conflict.

I did wonder if I could set variables for the starting Insert IDs at the start of the generated SQL and just reference them with increments afterwards, so you could just change
a few variables at the start to the next free IDs. I didn't know if I could do that in SQL, but I just tested it with select statements, and it seems to be possible:


set @myinsertid = 1001;
select id, name from npc_types where id = @myinsertid;
select id, name from npc_types where id = @myinsertid + 1;


Output:

mysql> source test.sql
Query OK, 0 rows affected (0.00 sec)

+------+-------------+
| id | name |
+------+-------------+
| 1001 | Guard_Mezzt |
+------+-------------+
1 row in set (0.00 sec)

+------+--------------+
| id | name |
+------+--------------+
| 1002 | Guard_Jerith |
+------+--------------+
1 row in set (0.00 sec)


So maybe that is the way to go.

cavedude
05-30-2010, 04:37 PM
So maybe that is the way to go.

Yes! I think you got it! It would mean everybody would need to re-dump their SQLs to share, but it would also mean that with a couple changes all those files will be universal over any database.

Derision
05-30-2010, 05:24 PM
Yes! I think you got it! It would mean everybody would need to re-dump their SQLs to share, but it would also mean that with a couple changes all those files will be universal
over any database.

How would it be if I removed all the INSERT ID fields from the Extractor UI, so you didn't have to worry about them when generating the SQL, and could just go in afterwards
and set them as required, i.e. I would put a template at the start of the generated SQL like:



set @NPCTypesStartingInsertID = XXXXXXX
set @SpawnEntryStartingInsertID = XXXXXXX
...
...
-- Set the starting Insert IDs above and remove the exit statement below before executing this SQL
exit
<Generated Insert statements follow, referencing the variables defined above>



I.e. you would just have to press the Load .pcap button, select the .pcap file, alter the check boxes if you wanted, not have to worry about IDs at that point, and then just click 'Generate SQL'.

Secrets
05-30-2010, 05:27 PM
How would it be if I removed all the INSERT ID fields from the Extractor UI, so you didn't have to worry about them when generating the SQL, and could just go in afterwards
and set them as required, i.e. I would put a template at the start of the generated SQL like:



set @NPCTypesStartingInsertID = XXXXXXX
set @SpawnEntryStartingInsertID = XXXXXXX
...
...
-- Set the starting Insert IDs above and remove the exit statement below before executing this SQL
exit
<Generated Insert statements follow, referencing the variables defined above>



I.e. you would just have to press the Load .pcap button, select the .pcap file, alter the check boxes if you wanted, not have to worry about IDs at that point, and then just click 'Generate SQL'.

That seems ideal to me. That way it's compatable with any database.

Secrets
05-30-2010, 05:31 PM
As for loot drops and everything, I found that Magelo is a better resource than Lucy. I don't know why or how, but it is. I think Magelo uses the same ID in items too.

edit: yeah it does. http://eq.magelo.com/item/1001

trevius
05-30-2010, 05:45 PM
How would it be if I removed all the INSERT ID fields from the Extractor UI, so you didn't have to worry about them when generating the SQL, and could just go in afterwards
and set them as required, i.e. I would put a template at the start of the generated SQL like:



set @NPCTypesStartingInsertID = XXXXXXX
set @SpawnEntryStartingInsertID = XXXXXXX
...
...
-- Set the starting Insert IDs above and remove the exit statement below before executing this SQL
exit
<Generated Insert statements follow, referencing the variables defined above>



I.e. you would just have to press the Load .pcap button, select the .pcap file, alter the check boxes if you wanted, not have to worry about IDs at that point, and then just click 'Generate SQL'.

I prefer having the fields there, but a simple toggle box to toggle them enabled/disabled would work well. That way, people could toggle them off when collecting for PEQ or leave them on if collecting for their own DB. A simple toggle would make it nice and quick to generate one of each type of you wanted. I think it is nice to be able to see the actual range of numbers that the SQL is planning to use so it can be easily compared against the DB to make sure it will be ok.

cavedude
05-30-2010, 06:07 PM
I agree with Trevius. Although I would personally disable the IDs in the program, I think it should still be an option for others.

trevius
05-30-2010, 07:33 PM
I don't know the details of how the tool works exactly, but it seems like it parses the .pcap when you first load it. Perhaps you could just give an option to generate a file of that parse, which could then be loaded and used to generate SQL from. Figured that may be another possible option for creating files anyone could use and share without worrying about private info. Another nice bonus to using a parse instead of some generated SQL is that a parse should be able to hold data that may not currently be set to be generated into SQL yet, so as the extractor tool is refined/expanded, those same parses could be ran through the tool again for an instant update with the latest features. This should prevent people from having to run their own .pcaps through the tool and sending them again. Instead, Cavedude could just run the parses through the tool, and have the latest with minimal work.

Akkadius
05-30-2010, 08:55 PM
I agree with Trevius. Although I would personally disable the IDs in the program, I think it should still be an option for others.

I just got back from ball, I agree completely that this should be a universal application for dumping collects, but should also be customizable if need be.

gaeorn
05-31-2010, 01:49 AM
A loot table script would be phenomenal. I wouldn't mind helping with this.

I already farmed all the data from magelo for their loot info for all mobs in all zones. I suppose the same could be done for lucy, but I believe the magelo data is more complete. I can write a script to parse out the usable data without an issue. The problem comes in when it's time to write out that data in some usable form. I haven't started on the script yet because I haven't yet decided how to write out the data. I'm sure I'll either write out SQL or I'll create temp tables and dump the data there, but I'm sure it'll take a human to merge it in cleanly with the PEQ db.

If you want to help, look at the magelo drop data and see what ideas you come up with for using it. I'd love to hear more ideas about how best to use that data.

gaeorn
05-31-2010, 01:52 AM
I did wonder if I could set variables for the starting Insert IDs at the start of the generated SQL and just reference them with increments afterwards, so you could just change
a few variables at the start to the next free IDs. I didn't know if I could do that in SQL, but I just tested it with select statements, and it seems to be possible:


set @myinsertid = 1001;
select id, name from npc_types where id = @myinsertid;
select id, name from npc_types where id = @myinsertid + 1;


Output:

mysql> source test.sql
Query OK, 0 rows affected (0.00 sec)

+------+-------------+
| id | name |
+------+-------------+
| 1001 | Guard_Mezzt |
+------+-------------+
1 row in set (0.00 sec)

+------+--------------+
| id | name |
+------+--------------+
| 1002 | Guard_Jerith |
+------+--------------+
1 row in set (0.00 sec)


So maybe that is the way to go.

Why not just fill in the ID with a sub select? You could simply do a (select max(id) from table)+1 where you would otherwise put in the ID. Or you could at least use a select to automatically set the variable at the start and otherwise do it as you have above.

gaeorn
05-31-2010, 01:56 AM
I don't know the details of how the tool works exactly, but it seems like it parses the .pcap when you first load it. Perhaps you could just give an option to generate a file of that parse, which could then be loaded and used to generate SQL from. Figured that may be another possible option for creating files anyone could use and share without worrying about private info. Another nice bonus to using a parse instead of some generated SQL is that a parse should be able to hold data that may not currently be set to be generated into SQL yet, so as the extractor tool is refined/expanded, those same parses could be ran through the tool again for an instant update with the latest features. This should prevent people from having to run their own .pcaps through the tool and sending them again. Instead, Cavedude could just run the parses through the tool, and have the latest with minimal work.

Why not just have a tool that zeros out any personal info from the .pcap files so they could be sent in as is. Then the full communication between client and server is complete in case we figure out something additional to extract from it later, but it won't contain any personal information that a player may wish kept from the general populace.

trevius
05-31-2010, 04:08 AM
Why not just have a tool that zeros out any personal info from the .pcap files so they could be sent in as is. Then the full communication between client and server is complete in case we figure out something additional to extract from it later, but it won't contain any personal information that a player may wish kept from the general populace.

I think the old collectors originally used to do just that, but since then, SOE has added quite a few more packets with character names, and probably other information, and it might be hard to track them all down. I suppose we could put together a list of all possible packets we may ever need collected, and just filter out the rest of the unneeded ones. As-is, I think Derision already has the most important packets being collected. As far as I can think, the only other one we might even want at this point would be OP_SpawnAppearance.

To make use of many more packets would require a different kind of collecting, which would be more interactive. The collecting this tool currently does best works when there is no interaction at all with the zone other than maybe checking merchants for merchant lists. Maybe in the future, other types of collects could be made that collect things like tasks, loot drops, spell sets on NPCs, or any other number of things that require considerably more time to collect enough info for it to be useful. Most of that stuff would probably be better off just adding it manually after a little research from various websites. Though, tasks might be one thing we could get enough info from without much time, but Derision would be much better to answer that.

cavedude
05-31-2010, 09:15 AM
Just a minor suggestion, can the zone data and zone_points query be given checkboxes as well? If they are checked, they will be in the log uncommented, if unchecked they will be completely ignored?

Derision
05-31-2010, 01:03 PM
1.4 is up in the downloads section:

Changes:


==05/31/2010==
Populated Melee texture columns.
For playable races, equip_chest2 is set based on the equipped chest item, if one is equipped.
Set armour tint columns based on the colour field for the chest slot.

The starting IDs for database inserts are now set as variables at the top of the SQL:
Note the Door ID has changed meaning. Instead of being the value for the id column in the doors table,
it is now the base value for the doorid column. The id column will just use the next autoincrement id.

If the version number is none-zero, The base doorid = Version * 1000 and all other insert IDs are
set to ZoneID * 1000 + (Version * 100).

If the version number is none-zero, deletes for npc_types and the spawn tables will only be generated from
the starting InsertID to starting ID + 99.

Moved the Version NumericUpDown control to before the InsertID textboxes as changing the version will
auto-change the InsertIDs.

Corrected a bug where IT10805 objects were flagged as ovens instead of brew barrels.
Added checkboxes (default unchecked) for zone config (fog, safe point, etc) and zone points.

Akkadius
05-31-2010, 03:27 PM
I've done multiple zones and seen the quality extract, but PoK for some reason is just completely jacked up. There are multiples of spawns, and all on the guard pathing and I have redone the collect 4 times after the initial one which I took about 20-30 minutes to do to get the 170 merchant lists in the zone.

Here is the .pcap and the .sql, let me know what you think.

http://www.sendspace.com/file/wclue5

Derision
05-31-2010, 03:49 PM
My first thought is you probably have the original PEQ spawns for POK which use different ID ranges than my app generates, and so it doesn't remove them.

If you do select id, spawngroupID from spawn2 where zone='poknowledge'; and you get ids not in the range 202000 - 202999, then that is the issue. You will need to manually trawl through your DB and delete the original PEQ data for that zone.

If that's not the problem, let me know.

Akkadius
05-31-2010, 03:55 PM
My first thought is you probably have the original PEQ spawns for POK which use different ID ranges than my app generates, and so it doesn't remove them.

If you do select id, spawngroupID from spawn2 where zone='poknowledge'; and you get ids not in the range 202000 - 202999, then that is the issue. You will need to manually trawl through your DB and delete the original PEQ data for that zone.

If that's not the problem, let me know.

I forgot to mention that that isn't the problem.

Derision
05-31-2010, 04:17 PM
I sourced the SQL into an otherwise empty test DB and browsed through it, and didn't see any obvious duped spawns.

I don't feel inclined to take the time tonight to create a DB cleansed of PEQ PoK data so I can actually source the SQL in and test it in game, but if you don't figure out
what the problem is, I'll see if I can get to it during the week ahead.

Derision
05-31-2010, 04:24 PM
Actually, I do see dupes, for #Tzudani Emissary of Veeshan (lots of them). The packet dump shows lots of OP_ZoneEntry packets for him, as if he spawned
mulitple times during the collect. I'll look at it further tomorrow.

Akkadius
05-31-2010, 04:35 PM
Actually, I do see dupes, for #Tzudani Emissary of Veeshan (lots of them). The packet dump shows lots of OP_ZoneEntry packets for him, as if he spawned
mulitple times during the collect. I'll look at it further tomorrow.

Sounds good, I had to go to work so I wasn't going to troubleshoot it further.

cavedude
05-31-2010, 07:35 PM
I've brushed off the old PEQ repo for logs and started posting what we have so far at http://www.projecteq.net/logs

You'll need to create a PEQ forum name and log in (http://www.projecteq.net/phpBB2) first, but then anybody is free to upload and download any logs they have, or logs in the repo.

All of the logs currently up are zipped/rarred SQLs using EQExtractor 1.4. However, if you wish feel free to post the pcaps. Please don't post SQLs created with anything less than 1.4, as they aren't very versatile among different databases.

Tricyclethief
05-31-2010, 09:48 PM
Thanks cavedude! i should be finished by tonight and have about all new SOD zones collected.

steve
05-31-2010, 10:46 PM
Actually, I do see dupes, for #Tzudani Emissary of Veeshan (lots of them). The packet dump shows lots of OP_ZoneEntry packets for him, as if he spawned
mulitple times during the collect. I'll look at it further tomorrow.

#Tzudani Emissary of Veeshan was permakilled by GMs on Live. Meaning the instant he appears in-game, he automagically self-kills. This was the Dragons, Dragons, Dragons event that is disabled on Live. His spawn rate must be set very low, hence the numerous dupes of him spawning. Dunno why they would depop him that way instead of their scripted means, but ah well... If you have a fast enough connection, you can sit outside the Guild Lobby zonein, facing the tree and see him pop into view for a millisecond and then poof. He's right behind the portal spot for PoK.

Akkadius
06-01-2010, 03:31 AM
Just uploaded entire TSS and almost entire SoD minus some Void versions.

8 D

robinreg
06-01-2010, 04:25 AM
uploaded the entire DoD zones minus instances. I sourced in the Dreadspire Keep into my test server and it really populates everything. even the pathing. Cool to even see Master Vule there haha. That's is so awesome. I will try to get some DoN in also. I may even open up a 2nd account to start doing instances for DoN and DoD.

steve
06-01-2010, 01:11 PM
Just uploaded entire TSS and almost entire SoD minus some Void versions.

8 D

Which Void zones did you miss? I can see about grabbing them.

Derision
06-01-2010, 01:17 PM
PoK for some reason is just completely jacked up.

I blew away PoK from my PEQ database and sourced in your SQL and it looks fine to me.

The only duped spawn I could see is that Veeshan Emissary guy which Live is constantly spawning and immediately killing.

I have found it is not unusual to see mobs pathing around buildings or walls to actually walk through them on occasion, which I believe is due to you being out of the range at which you get constant movement updates for them, at some point during the packet collect.

The only other thing I noticed was the player mounts that got collected. I'll fix that in the next release.

Akkadius
06-01-2010, 02:32 PM
I blew away PoK from my PEQ database and sourced in your SQL and it looks fine to me.

The only duped spawn I could see is that Veeshan Emissary guy which Live is constantly spawning and immediately killing.

I have found it is not unusual to see mobs pathing around buildings or walls to actually walk through them on occasion, which I believe is due to you being out of the range at which you get constant movement updates for them, at some point during the packet collect.

The only other thing I noticed was the player mounts that got collected. I'll fix that in the next release.

Player mounts have been in there yes, but not a huge deal imo, I think it helps to get the new versions of mounts collected.

I've wiped my PoK slate, #repop and 0 NPC's. When I simply reenter my 35 minute collect with the 170 merchants I get duplicates of NPC's everywhere, NPC's pathing everywhere, NPC's not even close to their remote location way across the zone. I thought it may have been pre-existing grids but that's not it.

Derision
06-01-2010, 02:39 PM
Odd. What I did was shutdown the server completely, executed:

delete from spawn2 where zone like '%poknowledge%';
delete from doors where zone like '%poknowledge%';
delete from grid where zoneid = 202;
delete from grid_entries where zoneid = 202;
delete from ground_spawns where zoneid = 202;
delete from object where zoneid = 202;

Then sourced in your SQL and restarted the server.

If you do #npcstats on each occurrence of the same duped spawns, are they both the same npcid and spawngroup (and in the 202nnn range ?).

cavedude
06-01-2010, 02:56 PM
Sorry to keep being a bother, but checking Zones Points and Zone Config outputs the entries commented out. Any chance that can be changed?

Also, how difficult would it be to add an option to UPDATE npc_types instead of INSERT? If checked, the SQL may look like:


UPDATE npc_types set field1 = $field1, field2 = $field2 where name = $name and id >= @StartingNPCTypeID and id <= @StartingNPCTypeID + 999;

That would help me to use new collects to update already built zones, since you've added proper texture and face parsing! Of course, the default option should probably create INSERTS for new zones :)

I can't thank you enough, this seriously is the best parser EQEmu has ever seen! My hope is the next version of the PEQ database will have every zone supported by our client at least spawned with doors, objects, fog, etc.

Akkadius
06-01-2010, 03:05 PM
Sorry to keep being a bother, but checking Zones Points and Zone Config outputs the entries commented out. Any chance that can be changed?

Also, how difficult would it be to add an option to UPDATE npc_types instead of INSERT? If checked, the SQL may look like:


UPDATE npc_types set field1 = $field1, field2 = $field2 where name = $name and id >= @StartingNPCTypeID and id <= @StartingNPCTypeID + 999;

That would help me to use new collects to update already built zones, since you've added proper texture and face parsing! Of course, the default option should probably create INSERTS for new zones :)

I can't thank you enough, this seriously is the best parser EQEmu has ever seen! My hope is the next version of the PEQ database will have every zone supported by our client at least spawned with doors, objects, fog, etc.

Should be an option between Insert and Update really.

Derision
06-01-2010, 03:20 PM
Sorry to keep being a bother, but checking Zones Points and Zone Config outputs the entries commented out. Any chance that can be changed?

I had misread your original request which is why I left them commented out. I've changed it locally and it will be in the next release, which won't be until tomorrow at least.


Also, how difficult would it be to add an option to UPDATE npc_types instead of INSERT? If checked, the SQL may look like:


UPDATE npc_types set field1 = $field1, field2 = $field2 where name = $name and id >= @StartingNPCTypeID and id <= @StartingNPCTypeID + 999;

That would help me to use new collects to update already built zones


I assume that if the option to generate UPDATEs like that is selected, then you wll also not want the INSERTs generating for the spawn tables ?

Finally, you are not being a bother at all. Of anyone, you know best as to how a tool like this needs to work from a user's perspective :)

Derision
06-01-2010, 03:24 PM
UPDATE npc_types set field1 = $field1, field2 = $field2 where name = $name and id >= @StartingNPCTypeID and id <= @StartingNPCTypeID + 999;


I've just thought of an issue with this. What about the case where multiple mobs exist with the same name with different attributes (level, hp, possibly other attributes) ?

Do you only want a selected number of fields included in the update, like face, texture, etc ?

cavedude
06-01-2010, 04:23 PM
Yes, I know in my case if I choose the UPDATE option, I'll only be looking to update the npc_types, and will leave all other options unchecked.

That's a good thought, about NPCs with the same name and different levels or other critical stats. What I am mainly looking for is the appearance data, so yes if the query could be limited to textures, face, the various color and hair columns, armortints, and melee columns, etc that would be ideal! Also, I noticed the Drakkin specific appearance columns aren't parsed, are they not sent? No big deal if not. The data I am working with now is far more detailed than ever!

Derision
06-01-2010, 04:35 PM
Good catch about the drakkin data. I decode it from the packet, just don't include it in the INSERT. I'll include that in the list of things to add.

Akkadius
06-01-2010, 05:31 PM
I've only posted TSS and SoD because I felt I had all of the zones. I have quite a few bits and pieces of others

EDIT: Yes Derision I got PoK to work after some trial and error, but there was a dramatic difference which makes no sense but yeah 8 D.

steve
06-01-2010, 06:51 PM
Will this work for Test Server? If not, is there a way to get it to work?

Was thinking best way might be to testcopy and do some suicide runs.

Akkadius
06-01-2010, 07:00 PM
Will this work for Test Server? If not, is there a way to get it to work?

Was thinking best way might be to testcopy and do some suicide runs.



What are you trying to say?

steve
06-01-2010, 07:40 PM
Answered my own question. Appears to work fine on Test Server.

Anyway, I noticed when running collects in The Void, the spawn of Zebuxoruk is not logged. Verified with Akkadius' collects that he uploaded as well. He's a special merchant that you buy from using Chronobines.

Andrew80k
06-01-2010, 09:13 PM
Answered my own question. Appears to work fine on Test Server.

Anyway, I noticed when running collects in The Void, the spawn of Zebuxoruk is not logged. Verified with Akkadius' collects that he uploaded as well. He's a special merchant that you buy from using Chronobines.
He also does some of the void access quests.

Akkadius
06-01-2010, 10:40 PM
He probably has classification of a different merchant type which isn't in Derision's script to filter and capture.

Akkadius
06-02-2010, 12:28 AM
I'm gathering as much of PoR as I possibly can without the instances. I will have all of the statics tonight. As well as DoN statics. So whoever is doing collects I would work on another expansion.

Although there are some very very dynamic zones such as "The Devastation" where there are several factions of NPC's within a zone and they take each other over, I am collecting it as well as I possibly can. Because sitting in Devastation isn't going to get the zone collected thoroughly because we need working faction ID's to make it all work. And then they have weapons and armory that spawn as a result of a certain faction taking over a zone, such as walls and catapults. Rage is similar.

robinreg
06-02-2010, 12:48 AM
I got all the DoD and PoR zones. the one I got for devastation is the Bolvark spawns. I labeled it on the PEQ log site. Also got most of the revamps zones. I'm currently working on GoD instance zones.

Akkadius
06-02-2010, 01:14 AM
I got all the DoD and PoR zones. the one I got for devastation is the Bolvark spawns. I labeled it on the PEQ log site. Also got most of the revamps zones. I'm currently working on GoD instance zones.

I'm grabbing quite a few of the PoR instance zones.

Derision
06-02-2010, 03:57 AM
He probably has classification of a different merchant type which isn't in Derision's script to filter and capture.

The only spawns that should not be included are players, pets and mercenaries. I'd be interested in looking at the .pcap or the hex dump produced by
the 'Dump Packets' button for the Void zone to figure out why Zebuxorok isn't being included, if you or Steve could email it to me at DerisionEQ at gmail.com

Akkadius
06-02-2010, 04:15 AM
The only spawns that should not be included are players, pets and mercenaries. I'd be interested in looking at the .pcap or the hex dump produced by
the 'Dump Packets' button for the Void zone to figure out why Zebuxorok isn't being included, if you or Steve could email it to me at DerisionEQ at gmail.com

Done, let me know what you think.

Derision
06-02-2010, 05:17 AM
Done, let me know what you think.

There is nothing unusual about his spawn packet, and if I run that .pcap through an older, command-line version of the extractor, it generates SQL for Zebuxoruk just fine.

I'm at work right now and don't have the current source code with me, but it must be a simple bug that I will fix tonight. I'm thinking that maybe the extractor is always missing
the first NPC in the zone, and since Zeb is the only NPC in the Void, then it is noticeable.

Andrew80k
06-02-2010, 10:40 AM
Anyone doing TBS?

cavedude
06-02-2010, 12:03 PM
I did the first couple of zones, but have since started working on other zones to get some zone_points and npc appearances working.

Akkadius
06-02-2010, 01:22 PM
Anyone doing TBS?

I have been doing TBS, but most of the zones are instanced, I have most of the statics though.

I've got almost PoR (is done I think), TBS and DoN statics completely done. I don't post them all until they are done otherwise if I upload them gradually I have no idea where I am with the way you have to upload to CD's repo.

Derision
06-02-2010, 02:14 PM
1.5 is up with these changes:

Fixed bug which caused the first spawn to always be skipped.
Update generated for fog_density column in zone table.
Populated Drakkin specific fields in npc_types.
If zone config and/or zone points are selected, the generated SQL will no longer be commented out.


I implemented the update for npc_types, but I have disabled it for now in the UI as I found some issues with it ...

First I was updating the size, then I found multiple NPCs with the same name but different sizes, so I took the size out of the update. Then I found multiple NPCs with the same name,
but some were wielding weapons and some not, e.g.

mysql> select id, name, d_meele_texture1, d_meele_texture2 from npc_types where name like 'a_Nokk_elder';
+--------+--------------+------------------+------------------+
| id | name | d_meele_texture1 | d_meele_texture2 |
+--------+--------------+------------------+------------------+
| 394319 | a_Nokk_elder | 0 | 0 |
| 394320 | a_Nokk_elder | 10608 | 0 |
| 394363 | a_Nokk_elder | 0 | 0 |
+--------+--------------+------------------+------------------+
3 rows in set (0.25 sec)


(this is with current live data, not existing PEQ tables).

Now my guess is there are multiple mobs with the same name but different facial features, but I didn't check that, so I'm really not sure what to do and await Cavedude's words
of wisdom on the matter :)

cavedude
06-02-2010, 02:30 PM
The melee textures, could that possibly be an item the NPC has on their loottable? If so, I don't think it would be a big deal to 0 out those columns after they have been sourced into the db. I'm mainly looking for textures on guards and player races as everything else is usually handled by loot. It's no big deal if I have to go back and manually fiddle with the melee columns.

About the size and face data... That's tricky. How about a way to check which columns you want to dump in the UPDATE? Maybe have the default the "safe" options, but also have choices for things like level or size which may not be as safe. Not sure how much work that would be, but I think that would make everybody happy.

Derision
06-02-2010, 03:08 PM
Done, let me know what you think.

Thanks for the .pcap. The problem with Zeb not appearing is fixed in v1.5. Could I possibly trouble you for a .pcap for Ocean Green Hills ? There are 'Zombie Catapults' in there
that are showing as humans with outstretched arms, so there must be something else in the spawn structure that we are not currently handling.

(Example of the human zombie catapult is at -631, 4868, 9).

Akkadius
06-02-2010, 03:31 PM
Thanks for the .pcap. The problem with Zeb not appearing is fixed in v1.5. Could I possibly trouble you for a .pcap for Ocean Green Hills ? There are 'Zombie Catapults' in there
that are showing as humans with outstretched arms, so there must be something else in the spawn structure that we are not currently handling.

(Example of the human zombie catapult is at -631, 4868, 9).

Derision I sent you my entire collection, the same link I sent to Cavedude. At your Gmail account.

Derision
06-02-2010, 03:40 PM
Derision I sent you my entire collection, the same link I sent to Cavedude. At your Gmail account.

Thanks. Much appreciated :)

Akkadius
06-02-2010, 03:48 PM
Thanks. Much appreciated :)

No sir, thank you, this is amazing, and I can do so so much with this, even though my server content is usually from scratch. And this greatly helps the entire community in any of their efforts.

EDIT: BTW your new compile is erroring out MYSQL syntax:

This is one of two that I have encountered, the same error as followed:

UPDATE zone set fog_density = 0.33 WHERE zoneidnumber = 342;
[Err] 1054 - Unknown column 'fog_density' in 'field list'

Derision
06-02-2010, 04:23 PM
EDIT: BTW your new compile is erroring out MYSQL syntax:

This is one of two that I have encountered, the same error as followed:

UPDATE zone set fog_density = 0.33 WHERE zoneidnumber = 342;
[Err] 1054 - Unknown column 'fog_density' in 'field list'



Yeah, that is a new column that KLS added to the zone table very recently (Rev1513), and the reason I put it in a separate update statement, so the first,
main, zone table row update will go through if you don't have that column, and the separate update for fog_density will error if you don't have it, but can be
safely ignored.

I meant to put a comment preceding the fog_density update to that effect, but forgot.

Akkadius
06-02-2010, 04:25 PM
Yeah, that is a new column that KLS added to the zone table very recently, and the reason I put it in a separate update statement, so the first, main, zone table row update will go through
if you don't have that column, and the separate update for fog_density will error if you don't have it, but can be safely ignored.

I meant to put a comment preceding the fog_density update to that effect, but forgot.

Yeah I haven't updated binaries in a few weeks only because I was scared of some of the repercussions some of the changes might have had.

trevius
06-03-2010, 04:47 AM
Yeah I haven't updated binaries in a few weeks only because I was scared of some of the repercussions some of the changes might have had.

Everything actually seems to work pretty smoothly after the recent large updates BTW.

Derision
06-03-2010, 02:12 PM
1.6 is up with a couple of bug fixes

The zone long name should no longer be truncated at 32 characters.
Apostrophes in the zone long name are escaped, i.e. Dranik's Hollows should be Dranik\'s Hollows.


And I enabled the feature to update existing NPCs in the database, which doesn't generate new npc_types or spawn table entries, but updates existing npc_types as follows:



If selected, the following columns:

texture, helmtexture, size, face, luclin_hairstyle, luclin_haircolor, luclin_eyecolor, luclin_eyecolor2, luclin_beardcolor,
luclin_beard, drakkin_heritage, drakkin_tattoo, drakkin_details, armortint_red, armortint_green, armortint_blue,
d_meele_texture1, d_meele_texture2, findable

will be updated for existing rows in the npc_types table that meet the following criteria:

Their ID falls within the range specified.
The NPC is a playable race (Race <= 12 || Race == 128 || Race == 130 || Race == 330 || Race == 522)
The NPC name is unique in the sense that only one NPC with the given first name was seen in the packet collect.


In other news :)

The zombie_catapults I mentioned in a previous post that were showing as humans are actually 'Destructable objects' that I think were introduced in Serpent's spine.

I can get the catapults to display properly if I hard code the same data as is in a live spawn packet (three strings and 53 bytes of as yet unidentified data).

I plan to look at the destructable objects some more at the weekend to see if I can figure out what those 53 bytes control so we can make some meaningful fields for them in
the npc_types table.

If anyone has any info on destructable objects, particuarly do some behave differently than others, do their graphics change as they get to various %s of hit points, are some immune
to melee, or spells, or certain types of spells, etc, that might possibly help identify what some of the data associated with them controls.

Domel
06-03-2010, 02:32 PM
Here is a bit from Eqplayers :

Destructible Objects - Destructible objects, like walls, doors, and siege engines, are used in many of the new expansion zones. These are terrain objects that have a health points which can be depleted and, when the object has reached low enough health, it will crumble and break. As you damage the objects you will see it break down until it is completely destroyed. You can identify an object that is destructible because it will pulse when you put your mouse over it. Source: http://eqplayers.station.sony.com/game_updates.vm?date=2/21/2006

I do remember breaking up catapults and watching them look more damaged, it was pretty cool :)

Unfortunately, I don't recall any specifics like what % the graphics changed, etc.

cavedude
06-03-2010, 04:25 PM
Here I am again! :)

Could gender also be added to the update query? I have that Shroudkeeper in PoK and also many Crescent guards in mind with that request. Also, could you add the guard races to the list? Their weapons are a key piece of data I am looking for, and in most cases the guard races differ from the base race. Lastly, could you make it so that version also gets added to the WHERE statement? So if I select version 1, it would tack on "and version = 1" to the end of the query.

The list of guard races:

44 Freeport Guards
67 Highpass Guards
71 Qeynos Guards
77 Neriak Guards
78 Erud Guards
81 Rivervale Guards
90 Halas Guards
92 Grobb Guards
93 Oggok Guards
94 Kaladim Guards
106 Felguards
112 Fayguards
139 Cabilis Guards
183 Coldain
239 Vah Shir Guards

I believe that's it!

I have to say, I just updated Crescent and it looks beautiful now! All the NPCs have their proper robes, faces, heritage, and weapons. Thank you!

I think there may be a bug with certain zoneids parsing incorrectly, but I want to gather details before officially reporting it. Once I get that, I'll PM you with a log I see the issue with.

Derision
06-03-2010, 04:59 PM
I think there may be a bug with certain zoneids parsing incorrectly, but I want to gather details before officially reporting it. Once I get that, I'll PM you with a log I see the issue with.

I noticed the zoneids with the dranikhollowsc and ikkinz zones on the PEQ logs repo seems totally out of whack. I really need to see the .pcaps or logs produced by the 'Dump Packets' button to figure out what the issue is.

robinreg
06-03-2010, 05:32 PM
I sent the pcaps to CD. you can get it from him for debugging purpose. the ikky zones are basically the same zone but they are different instances with different npcs for progression.

Akkadius
06-03-2010, 06:26 PM
I posted the rest that I have for the most part, I halted doing any captures to get quite a few things done on my server. I will do more capturing when it's appropriate.

cavedude
06-03-2010, 07:16 PM
I noticed the zoneids with the dranikhollowsc and ikkinz zones on the PEQ logs repo seems totally out of whack. I really need to see the .pcaps or logs produced by the 'Dump Packets' button to figure out what the issue is.

Was just sending you a link in fact, and also it appears 1.6 broke merchant dumps. I'm including a crescent log that dumps merchants for 1.5 and below, but not 1.6. It'll be in your PMs :)

Derision
06-04-2010, 01:07 PM
1.7 up

Fixed bug introduced in v1.6 that broke merchant SQL generation.
Gender is now updated for existing npc_types.
Guard races are now included when updating existing npc_types.
Obtained the ZoneID number from the Player Profile instead of the NewZone_Struct.

cavedude
06-04-2010, 01:57 PM
The zombie_catapults I mentioned in a previous post that were showing as humans are actually 'Destructable objects' that I think were introduced in Serpent's spine.

I can get the catapults to display properly if I hard code the same data as is in a live spawn packet (three strings and 53 bytes of as yet unidentified data).

I plan to look at the destructable objects some more at the weekend to see if I can figure out what those 53 bytes control so we can make some meaningful fields for them in
the npc_types table.

If anyone has any info on destructable objects, particuarly do some behave differently than others, do their graphics change as they get to various %s of hit points, are some immune
to melee, or spells, or certain types of spells, etc, that might possibly help identify what some of the data associated with them controls.

If it helps any, I think a_training_dummy in the crescent log I sent is in the same boat. On live it appears as a training dummy and seems to be a chest object, but in the collect its race and bodytype is 0.

Thank you for getting those changes in. I can't think of a thing wrong now!

Derision
06-04-2010, 02:23 PM
If it helps any, I think a_training_dummy in the crescent log I sent is in the same boat. On live it appears as a training dummy and seems to be a chest object, but in the collect its race and bodytype is 0.


Yes they are, thanks! I wish I'd known that earlier, it would have saved me the trouble of sneaking my way across Blightfire Moors and Goru'kar Mesa to find an object to test with :)

blackdragonsdg
06-04-2010, 03:49 PM
I keep getting an error on some zones when using the extractor to read the pcap files. City of Dranik and The Void are the only zones I have seen this error occur on. I have tried extractor versions 1.4, 1.5 & 1.7 and all encounter the same problem.


Unhandled exception has occurred in your application. If you click Continue, the application will ignore this error and attempt to continue. If you click Quit, the application will close immediately.

Source array was not long enough. Check srcIndex and length, and the array's lower bounds.


See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.ArgumentException: Source array was not long enough. Check srcIndex and length, and the array's lower bounds.
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length)
at EQPacket.PacketManager.ProcessPacket(IPAddress srcIp, IPAddress dstIp, UInt16 srcPort, UInt16 dstPort, Byte[] Payload, Boolean SubPacket, Boolean Cached)
at EQExtractor02.EQExtractor2Form1.device_OnPacketArr ival(Object sender, CaptureEventArgs e)
at SharpPcap.PcapDevice.SendPacketArrivalEvent(RawPac ket p)
at SharpPcap.PcapDevice.PacketHandler(IntPtr param, IntPtr header, IntPtr data)
at SharpPcap.SafeNativeMethods.pcap_dispatch(IntPtr adaptHandle, Int32 count, pcap_handler callback, IntPtr ptr)
at SharpPcap.PcapDevice.CaptureThread()
at SharpPcap.PcapDevice.Capture()
at EQExtractor02.EQExtractor2Form1.ProcessFileButton_ Click(Object sender, EventArgs e)
at System.Windows.Forms.Control.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnClick(EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventAr gs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.O nMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.W ndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
EQExtractor2
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///D:/zzTemp/EQExtractorv2v1.7/EQExtractor2.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
SharpPcap
Assembly Version: 3.1.0.0
Win32 Version: 3.1.0.0
CodeBase: file:///D:/zzTemp/EQExtractorv2v1.7/SharpPcap.DLL
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 (RTMRel.030319-0100)
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
PacketDotNet
Assembly Version: 0.4.0.0
Win32 Version: 0.4.0.0
CodeBase: file:///D:/zzTemp/EQExtractorv2v1.7/PacketDotNet.DLL
----------------------------------------
System.Core
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.1 built by: RTMRel
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
----------------------------------------
zlib.net
Assembly Version: 1.0.3.0
Win32 Version: 1.0.3.0
CodeBase: file:///D:/zzTemp/EQExtractorv2v1.7/zlib.net.DLL
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

steve
06-04-2010, 03:55 PM
What version of The Void did you use it in? It worked fine for me in Void A, B, & G.

Derision
06-04-2010, 04:01 PM
I keep getting an error on some zones when using the extractor to read the pcap files. City of Dranik and The Void are the only zones I have seen this error occur on. I have tried extractor versions 1.4, 1.5 & 1.7 and all encounter the same problem.


If you are willing to email me one of the .pcap files that is causing the crash (DerisionEQ at gmail.com) I'll see what's causing the crash and fix it.

Derision
06-04-2010, 04:05 PM
Lastly, could you make it so that version also gets added to the WHERE statement? So if I select version 1, it would tack on "and version = 1" to the end of the query.


Oops, I forgot this request. I'll put this in tomorrow.

blackdragonsdg
06-04-2010, 04:57 PM
What version of The Void did you use it in? It worked fine for me in Void A, B, & G.

It is the one when you zone in from PoTime...no idea which version it is.

blackdragonsdg
06-04-2010, 05:06 PM
If you are willing to email me one of the .pcap files that is causing the crash (DerisionEQ at gmail.com) I'll see what's causing the crash and fix it.

I just sent the file.

steve
06-04-2010, 05:17 PM
It is the one when you zone in from PoTime...no idea which version it is.

If you do a /who on yourself, it will say in the zonename. Depending on how far along you are with SoD progression determines what version of the zone you get sent to. Void A is the default version with no progression completed.

steve
06-04-2010, 09:54 PM
Are you guys camping out to character select after getting to each zone? Just wanted to make sure I wasn't doing something unneeded. And will multiboxing confuse the capture?

robinreg
06-04-2010, 10:09 PM
I just go to the zone that I want to do capture with. try to find a good spot preferebly in the center of the zone. then camp out to character select screen, start your sharkwire, then log on. Just sit around for 30 to 45 mins. then stop your sharkwire save your pcap. then generate the sql with the extractor. The program filters out all players characters, mercs, and pets. Just need to make sure there are no mob aggroing from anyone or you during the data collects. I did most SoF today and most zones were empty so I had the zone to myself. works out perfectly. The only 2 zone I had trouble with is Dragonscale and Loping Plains as there were other players killing mobs and such. So I just left that until I find a time that I have the whole zone to myself. Crystallos I need to get keyed for it. I just need a couple of items from Zeka and MMM then I can do the combine and I'll be able to access that zone.

Akkadius
06-04-2010, 10:52 PM
I just go to the zone that I want to do capture with. try to find a good spot preferebly in the center of the zone. then camp out to character select screen, start your sharkwire, then log on. Just sit around for 30 to 45 mins. then stop your sharkwire save your pcap. then generate the sql with the extractor. The program filters out all players characters, mercs, and pets. Just need to make sure there are no mob aggroing from anyone or you during the data collects. I did most SoF today and most zones were empty so I had the zone to myself. works out perfectly. The only 2 zone I had trouble with is Dragonscale and Loping Plains as there were other players killing mobs and such. So I just left that until I find a time that I have the whole zone to myself. Crystallos I need to get keyed for it. I just need a couple of items from Zeka and MMM then I can do the combine and I'll be able to access that zone.

I can get them all too when I get done with a few campaign things on my server.

steve
06-04-2010, 10:57 PM
I'm just nervous about how long I should stay in a zone to be sure I get all of the pathing. Are you staying 20 minutes in each zone?

robinreg
06-04-2010, 11:06 PM
I usually stay sometime between 30 to 45 mins to make sure I get all the pathing. move around a bit if it's big zone like commonlands.

Akkadius
06-04-2010, 11:22 PM
I'm just nervous about how long I should stay in a zone to be sure I get all of the pathing. Are you staying 20 minutes in each zone?

5-10 minutes is fine really, to get the gist of pathing. Otherwise there's a new grid system function that KLS made that is phenomenal.

robinreg
06-05-2010, 12:02 AM
ah I was just following the suggested time that was posted on log repository. Or was for the old collecting?

trevius
06-05-2010, 01:36 AM
Ideally, collects should have no one else in the zone if possible. Otherwise, pathing is going to be messed up by people pulling and killing spawns (one of the Relic collects shows this right when you zone in due to a player obviously training the zone in). Another thing I have noticed is that mounts show up pathing all over if people are in the zone and using them. That is easy enough to correct, though.

For doors, any door that is set to opentype 157 needs to be changed to 105 and have invertstate set to 1 for them to work correctly (steamfontmts and dragonscale for example).

As for collecting good pathing grids, it is a really good idea to use a rogue for collecting if possible. Many zones are accessible via a level 51 rogue on the Mayong Server. The reason it is good to use a rogue, is because you can move around most zones fairly freely as long as you consider anything you get near to make sure it can't see you. If you want good pathing, you will need to wander through the zone one section at a time so you get good pathing collects. The further away you are from an NPC, the less updates you get and the more choppy your pathing will be for them. But, if you move from spot to spot in a zone, you can get some pretty nice pathing collects. It does take a while to do that properly.

I am sure pathing will need to be cleaned up no matter what, but with a really good collect, that cleanup could be minimal. The collects we have so far are really great either way. If someone was planning to do some major work on a zone to make it fully playable, I would highly suggest making sure you have as good a collect as possible though.

Derision
06-05-2010, 03:37 AM
Are you guys camping out to character select after getting to each zone? Just wanted to make sure I wasn't doing something unneeded. And will multiboxing confuse the capture?

Multiboxing can confuse the extractor and appears to be the cause of blackdragonsdg's crash.

What happens is the extractor reads each packet in the .pcap file looking for an SOE 'Session Request' packet (that the client send to the server).
When it sees a session request, it 'tentatively locks on' to that session and waits until it sees a particular packet (which was OP_NewZone and is
the packet in which the server sends various details about the zone) before deciding this is a valid EQ connection and permanently locking onto
this stream and ignoring all others.

What can happen, is if you zone in a second character before the server has sent the OP_NewZone to the first one, the extractor get's confused
by the crossed streams (oblig. Ghostbusters reference).

I've altered the extractor locally to 'permalock' to a stream when it sees an OP_ZoneEntry packet from the client to the server. This is the second
packet sent and so it is much less likely that multi-boxing will cause an issue, although of course, only the packets to/from the first client to
zone-in will be processed, but that shouldn't be an issue.

I'll push this version out later today when I've looked at another issue.

EDIT: Oh, and you don't need to camp out when you have finished. Just stop the capture and save it.

joligario
06-05-2010, 04:03 AM
Derision: There's something very important I forgot to tell you.
Blackdragonsdg: What?
Derision: Don't cross the streams.
Blackdragonsdg: Why?
Derision: It would be bad.
Blackdragonsdg: I'm fuzzy on the whole good/bad thing. What do you mean, "bad"?
Derision: Try to imagine all life as you know it stopping instantaneously and every molecule in your body exploding at the speed of light.
Trevius: Total protonic reversal.
Blackdragonsdg: Right. That's bad. Okay. All right. Important safety tip. Thanks, Derision.

robinreg
06-05-2010, 04:14 AM
yeah I figured that it would not be a good idea to box on the same computer that is doing the collecting or running the wireshark. I have my 2nd account on my laptop while I go run my wireshark on my computer. Only reason I have 2nd account is to be able to do instances as you need 3 to get a group mission or expedition and the mercenary would count toward it. However, the mercenary would not be counted toward a raid expedition unfortunately.

steve
06-05-2010, 04:49 AM
Oh, and you don't need to camp out when you have finished. Just stop the capture and save it.

Gotcha. But if I want to start a new one, simply zoning isn't enough, correct? I need to start from the character select each and every time for a new capture?

Derision
06-05-2010, 04:54 AM
I've not tested it (I will later), but in theory, if you start a new capture while you are already in a zone, the next time you zone, the extractor should recognise the new zone and generate the SQL for that one.

trevius
06-05-2010, 08:17 AM
Another option may be to start the capture before you zone, and as you zone, watch the capture and find a packet you are sure is the new zone communicating with your client and then right click that packet and select Conversation Filter > UDP. That will cause it to filter out everything accept for that zone's communication to you. You can normally tell fairly easily which packets are the zone communicating to you when you zone by watching for a ton of packets coming in at the same time from the same IP to you. Also, it is good to note that if you scroll up a bit while collecting, it will remain at that part of the collect, so you can select your packet to filter easier. You can scroll back to the bottom after if you wish to continue watching them come in real time again.

Then, when you stop the capture and save it, be SURE that you select the option for "Displayed" in the "Packet Range" section of the Save As window. That will cause it to save only the packets you filtered to see. Otherwise, if you don't save it with that option, it still saves the unfiltered collect.

By doing this, it should mean you can start a collect, then zone, then collect and save it. Then start another collect and zone again and so on.

I haven't really tested this yet, but I am pretty sure it would work fine.

Derision
06-05-2010, 11:43 AM
1.8 is available

Permalocked onto stream using C->S OP_ZoneEntry to reduce chance of multi-boxing messing the collect up.
Version clause is appended to query when updating existing npc_types.
Selecting the update existing npc types option will automatically uncheck all other options.
Reworked texture/helm parsing, so hopefully NPCs should look as close as possible to live now (as close as we can without having textures for each armor slot in npc_types).
Added 'experimental option to add texture colors to npc_types_tint so that different armor slots can have different colors.


This should address blackdragonsdg's crashes.

The last change (npc_types_tint) generates a row in the npc_types_tint table for each armor slot, so NPCs with different colored items in each slot would look a bit more correct
(although I don't think there are many like that). I wouldn't recommend you select this option unless you know there is a specific NPC with multi-colored armor that you want to setup right.

I don't believe there are any existing NPCs in the standard PEQ database that use the npc_types_tint table, so it may not have been extensively tested.

There is more info about this table here: http://www.eqemulator.org/forums/showthread.php?t=28647

Derision
06-05-2010, 12:12 PM
I've not tested it (I will later), but in theory, if you start a new capture while you are already in a zone, the next time you zone, the extractor should
recognise the new zone and generate the SQL for that one.

OK, now I've tested it and it doesn't work so well on the current version, so I don't recommend starting a capture unless you are at character select,
unless, as Trevius suggests, you are comfortable with using Wireshark filters to just save the packets relating to the zone you are interested in.

As it stands now, starting a capture while you are already in a zone and then zoning will either cause the extractor to miss data when entering the new zone
(including crucial packets which make the collect unusable), or if a mob happens to spawn in the zone you are in, after you start the capture, but before you
zone, then that mob in the previous zone will be included in the data for the zone you are going to.

blackdragonsdg
06-05-2010, 01:57 PM
Derision: There's something very important I forgot to tell you.
Blackdragonsdg: What?
Derision: Don't cross the streams.
Blackdragonsdg: Why?
Derision: It would be bad.
Blackdragonsdg: I'm fuzzy on the whole good/bad thing. What do you mean, "bad"?
Derision: Try to imagine all life as you know it stopping instantaneously and every molecule in your body exploding at the speed of light.
Trevius: Total protonic reversal.
Blackdragonsdg: Right. That's bad. Okay. All right. Important safety tip. Thanks, Derision.

Haha, that sounds like the conversations in the Monk chat channel during a raid. Apparently monkey math is different from what everyone else learned.

cavedude
06-05-2010, 04:15 PM
Derision, you seriously rock. I did fib a little bit and found one more thing... Do doors still send teleport data? I know they used to, because that's how PEQ got most of its ports. I wasn't too concerned about it at first until I remembered about intrazone portals. I would make my life 100% easier if we could grab dest_zone, dest_x, dest_y, dest_z, and dest_heading.

Akkadius
06-05-2010, 05:58 PM
Derision, you seriously rock. I did fib a little bit and found one more thing... Do doors still send teleport data? I know they used to, because that's how PEQ got most of its ports. I wasn't too concerned about it at first until I remembered about intrazone portals. I would make my life 100% easier if we could grab dest_zone, dest_x, dest_y, dest_z, and dest_heading.

I second this motion.

trevius
06-05-2010, 10:18 PM
No, they don't send that info unfortunately, but, it may be possible to get it by waiting for clickdoor and then check where it sends them. Though, I imagine that would take a bit of work to figure out fully and also mean you would have to click every door and have it port you to get them.

Derision
06-06-2010, 04:07 AM
Looking at a couple of examples (a teleport disc in the bazaar and one of the PoK stones in ... PoK), the door param is a reference to one of the zone_points,
so by cross referencing the data, I may be able to populate the relevant fields in the doors table for ports.

KLS
06-06-2010, 05:09 AM
Yeah Cavedude mentioned it to me, and I was gonna post that the door data does appear to be sent as a zone in. I easily got port working with the nek book that way, I could of without but it saved me 2 minutes.

trevius
06-06-2010, 05:41 AM
Wow, now that is a good catch! Makes sense, heh.

Derision
06-06-2010, 05:51 AM
It doesn't appear to be the case in all zone, e.g. a couple of zone have a door_param of -1 for the PoK book with no matching zone_point.

Also the Arcane Disks in freeportwest have an open_type of 57 which is not handled as a teleport type door by the server,
but it is easy enough to go in and change the open_type afterwards, to 58.

Derision
06-06-2010, 09:58 AM
In 1.9, for doors with opentype 57 & 58, if there is a zone point with an id that matches with door_param, then the destination zone and co-ordinates from the zone point
are used to populate the door fields.

cavedude
06-09-2010, 04:11 PM
It would appear that this tool still functions after the patch!

Akkadius
06-09-2010, 04:41 PM
It would appear that this tool still functions after the patch!

Woot! I still have work to do.

steve
06-09-2010, 05:22 PM
Seems pretty fool proof. It worked on Test Server last week with no changes. Thought for sure it would at least need new opcodes... nope!

steve
06-10-2010, 12:42 PM
I have an improvement request...

Could you add an ability for the program to list the merchant NPC lists that were not collected, so they can be added on a second run through the zone? I'm assuming this is possible of course since something in the SQL must denote a merchant NPC.

I just uploaded sharvahl to the log repository. Not sure if I got all of the merchants since there's multiple levels to just about every house. Such a feature would have came in handy!

Derision
06-10-2010, 02:08 PM
I've started refactoring the code to (amongst other things) make it easier to support multiple patch versions concurrently, for when a patch does break it,
and I don't really want to make any other changes until I've finished and tested it thoroughly.

In the meantime, you can check for merchants who you haven't collected data for with this SQL

select id, name from npc_types where id >= 383000 and id <=383999 and class = 41 and merchant_id = 0;


changing 383000 and 383999 to the start and end ids of the NPCs of the zone you are interested in.

steve
06-10-2010, 03:23 PM
I'll just wait for you to add it in... I'm not currently running my own server so don't have the ability to check.

steve
06-11-2010, 11:55 AM
Found a bug using v1.9.

If a merchant only sells one item, it is not getting recorded in the sql generated. It just lists nothing for him in the generated sql.

Broker Fahaar in sharvahl only lists 'Wedding Band', but it's not recorded in the generated sql.

Derision
06-11-2010, 12:35 PM
Found a bug using v1.9.

If a merchant only sells one item, it is not getting recorded in the sql generated. It just lists nothing for him in the generated sql.

Broker Fahaar in sharvahl only lists 'Wedding Band', but it's not recorded in the generated sql.

The reason the Wedding Bands he sells is not being recorded is because he only has a quantity of 5 in stock. To distinguish between proper merchant items and items that
players have sold to them, the extractor only includes items that the merchant has an infinite supply of (quantity of -1 in the item packet).

I don't really know of a way to fix this without including all items that players have sold to mechants.

steve
06-11-2010, 01:21 PM
Maybe an option to include them? In a lot of older zones that I've been collecting in, there's been nothing on the merchants whatsoever since the zones are hardly used.

Include them in the SQL dump, but have them commented out and then when sourcing, the person could uncomment any lines he did want imported.

cavedude
06-11-2010, 05:06 PM
Ive posted the newest PEQ database up that has the output of this tool merged in. Most (but not all) of the zones supported by SoD are now spawned, with doors, objects, etc.

Also, Gaeorn wrote a tool to parse EQTrader's merchantlist data and dump it to SQL compatible with our system. I've posted the SQL output of that script on the SVN also. Beeping and warnings are normal while sourcing so pay them no mind. Overall, I've found EQTraders has very high accuracy, but I have found missing/extra items on a couple of vendors so it is not 100% perfect. Still in my mind it's a nice shortcut to fill in missing gaps so we don't have to go around to every merchant in the game when collecting :) Of course, backup your current merchantlist table before running this especially if you have custom merchants, as it may overwrite existing IDs.

Both can be had here: http://code.google.com/p/projecteqdb/downloads/list

Edit: I should mention if you are just running a stock PEQ database, then there is no need to source the merchantlists. They have already been merged with the newest database version.

WkimWes
06-14-2010, 02:31 PM
Ive posted the newest PEQ database up that has the output of this tool merged in. Most (but not all) of the zones supported by SoD are now spawned, with doors, objects, etc.

Also, Gaeorn wrote a tool to parse EQTrader's merchantlist data and dump it to SQL compatible with our system. I've posted the SQL output of that script on the SVN also. Beeping and warnings are normal while sourcing so pay them no mind. Overall, I've found EQTraders has very high accuracy, but I have found missing/extra items on a couple of vendors so it is not 100% perfect. Still in my mind it's a nice shortcut to fill in missing gaps so we don't have to go around to every merchant in the game when collecting :) Of course, backup your current merchantlist table before running this especially if you have custom merchants, as it may overwrite existing IDs.

Both can be had here: http://code.google.com/p/projecteqdb/downloads/list

Edit: I should mention if you are just running a stock PEQ database, then there is no need to source the merchantlists. They have already been merged with the newest database version.

Sweet, thanks Cavedude and everyone else!!! Great work!!

An_Evil_Enchanter01
07-20-2010, 12:57 AM
Can anyone use this to update cshome?

Dylan1994
08-28-2010, 12:31 AM
Could someone please tell me how to set up EQExtracter so that i am able to use it... For some reason im am unable to get it to work.... Help would be appreciated

Akkadius
08-28-2010, 12:43 AM
One thing I did notice Derision since I started doing a few more dumps, in zones like Gyrospire most noticeably... The doors are four pieced like Plane of Time. So when you click the door pieces they all go up vertically, when they are supposed to criss cross outwards to give the 'mechanic' feel. Now I know that you can't reproduce the walk-up opening of all 4 pieces of the door unless you forcedooropen but if its something that can be fixed in how the packets are decoded I am just pointing it out, no big deal.

Derision
08-28-2010, 05:14 AM
Could someone please tell me how to set up EQExtracter so that i am able to use it... For some reason im am unable to get it to work.... Help would be appreciated

You need to install the .NET Framework 4 from Microsoft first, I think this is the right link: http://www.microsoft.com/downloads/details.aspx?FamilyID=9cfb2d51-5ff4-4491-b0e5-b386f32c0992&displaylang=en

After that, install Wireshark http://www.wireshark.org/download.html

And finally download EQExtractor2 http://projecteqemu.googlecode.com/files/EQExtractor2-211.rar and use Winrar to extract the files into a folder of your choosing.

To capture packets for a zone you are interested in, get your character to that zone and then camp to character select.

Start Wireshark and in the Capture/Interface List on the left hand side, click on the name of the network interface that you are using to connect to the Internet.

Back in your Live EQ client, enter the zone. If you just want to collect spawns, you can stop the Wireshark capture as soon as your character appears in the world. To collect pathing, just sit around for a while, or if you want to collect merchant inventory, move around the zone clicking on merchants.

When you are done, in Wireshark, select Capture/Stop then File/Save As to save the captured packets into a .pcap file.

Fire up EQExtractor2, select File/Load PCAP (or press CTRL-L) and select the .pcap file you saved in the previous step.

You should get a message that the client version that was used to make the .pcap was recognised and also the name of the zone you where in.

Next, select File/Generate SQL (or Ctrl-S) and then on the next screen click on the Generate SQL button and give it a file name to save the generated SQL.

You can then use the mysql command prompt or Navicat or whatever, to source this into your database, however be aware that the generated SQL may delete data already in your database.

If you still can't get it to work, let me know what error messages, etc, you are seeing. With the latest EQExtractor2 version, pressing Ctrl-D will bring up a sort of debug-window which may have more detail of what went wrong.

Dylan1994
08-28-2010, 08:40 AM
its now saying The capture ession could not be initiated (Failed to set hardware filter topromiscuous mode). please check that "\Device\NPF_{E0FE483E-2611-47B2-8A77-C89F7F08CDDC}" is the proper interface.

Derision
08-28-2010, 08:53 AM
Is it a Wireless adapter ? Apparently some wireless adapters don't support promiscuous mode.

You could try going to Edit/Preferences, then clicking on Capture on the left hand site and then unticking 'Capture packets in promiscuous mode' on the right hand side. You don't need promiscuous mode in the case where you are running Wireshark on the same PC as Everquest.

Dylan1994
08-28-2010, 01:08 PM
that worked.. thanks

Derision
08-29-2010, 02:32 PM
One thing I did notice Derision since I started doing a few more dumps, in zones like Gyrospire most noticeably... The doors are four pieced like Plane of Time. So when you click the door pieces they all go up vertically, when they are supposed to criss cross outwards to give the 'mechanic' feel. Now I know that you can't reproduce the walk-up opening of all 4 pieces of the door unless you forcedooropen but if its something that can be fixed in how the packets are decoded I am just pointing it out, no big deal.

If you want to email me the .pcap for Gyrospire, I'll take a look at it.

Derision
08-30-2010, 11:22 AM
I wrote the following little player.pl for GyrospireB to open all pieces of the doors at once. The doorids are from the latest PEQ database, but I guess they should
be the same if you build the zone yourself from the extractor output:



#
# quests/gyrospireb/player.pl
#
# Script for opening all parts of the four part doors when a player clicks on any part.
#
sub EVENT_CLICKDOOR
{
@multipartdoors = ([58, 59, 60, 61], [62, 63, 64, 65], [66, 67, 68, 69], [70, 71, 72, 73], [78, 79, 80, 81],
[82, 83, 84, 85], [86, 87, 88, 89], [90, 91, 92, 93], [94, 95, 96, 97], [98, 99, 100, 101],
[102, 103, 104, 105], [106, 107, 108, 109], [110, 111, 112, 113], [120, 121, 122, 123],
[125, 126, 127, 128], [129, 130, 131, 132], [133, 134, 135, 136], [137, 138, 139, 140],
[141, 142, 143, 144]);

for $row ( @multipartdoors )
{
if(($doorid >= @$row[0]) && ($doorid <= @$row[3]))
{
for $d ( @$row)
{
if($d != $doorid)
{
quest::forcedooropen($d);
}
}
return;
}
}
}

Akkadius
08-30-2010, 11:51 AM
I wrote the following little player.pl for GyrospireB to open all pieces of the doors at once. The doorids are from the latest PEQ database, but I guess they should
be the same if you build the zone yourself from the extractor output:



#
# quests/gyrospireb/player.pl
#
# Script for opening all parts of the four part doors when a player clicks on any part.
#
sub EVENT_CLICKDOOR
{
@multipartdoors = ([58, 59, 60, 61], [62, 63, 64, 65], [66, 67, 68, 69], [70, 71, 72, 73], [78, 79, 80, 81],
[82, 83, 84, 85], [86, 87, 88, 89], [90, 91, 92, 93], [94, 95, 96, 97], [98, 99, 100, 101],
[102, 103, 104, 105], [106, 107, 108, 109], [110, 111, 112, 113], [120, 121, 122, 123],
[125, 126, 127, 128], [129, 130, 131, 132], [133, 134, 135, 136], [137, 138, 139, 140],
[141, 142, 143, 144]);

for $row ( @multipartdoors )
{
if(($doorid >= @$row[0]) && ($doorid <= @$row[3]))
{
for $d ( @$row)
{
if($d != $doorid)
{
quest::forcedooropen($d);
}
}
return;
}
}
}


Derision sir,

Damn good job. Sorry I didn't get back to you earlier...

If you don't mind me asking, what else do you have left for Shared Tasks, and is there anything that I can try to do to assist you with it.

Dylan1994
09-14-2010, 10:26 PM
Just wondering on the eqextractor is there a way to get the AA's from live?

Derision
09-15-2010, 02:36 PM
Just wondering on the eqextractor is there a way to get the AA's from live?

You can get some of the data.

The server does send the data that could be used to populate the altadv_vars table (except the AA name).

The aa_effects data (for passive effects) that is sent is only for the first rank of the AA (or the rank that the particular character you are capturing data for purchased).

Finally, the AA action data (for activated AAs) i.e. which spell to cast when you activate the AA, is not sent at all (until you activate them via a hotkey).

You can dump the AA data that is captured to a text file (Ctrl-A on the latest version), but from there you would need to manually create the database rows and figure out
yourself what aa_effect/aa_action rows are required to complete the implementation of a given AA.

Dylan1994
09-15-2010, 10:26 PM
You can get some of the data.

The server does send the data that could be used to populate the altadv_vars table (except the AA name).

The aa_effects data (for passive effects) that is sent is only for the first rank of the AA (or the rank that the particular character you are capturing data for purchased).

Finally, the AA action data (for activated AAs) i.e. which spell to cast when you activate the AA, is not sent at all (until you activate them via a hotkey).

You can dump the AA data that is captured to a text file (Ctrl-A on the latest version), but from there you would need to manually create the database rows and figure out
yourself what aa_effect/aa_action rows are required to complete the implementation of a given AA.

Ill have to mess around with it.. I'm still new with using this whole eqextractor and still learning what it can. Do.. So, it'll take some time.

steve
09-16-2010, 07:58 PM
If it works with Beta, it would be a good place to get AA data, since there's an NPC that grants you around 2000aa to test with.

provocating
12-16-2010, 10:05 PM
Is this still working, I tried tonight to use the extractor and got a "unsupported client" big red error message after reading my pcap file.

Derision
12-17-2010, 02:44 PM
The version on the source code SVN (2.2.9) works fine (I tested it last night). The compiled version in the download area on the SVN hasn't been updated for quite a while, so I have deprecated it.

You need Visual C# 2010 Express (freeware) to compile the source.

provocating
12-17-2010, 03:01 PM
Yeah, I used the precompiled version. I will install from source then.

provocating
12-21-2010, 11:42 AM
I am curious, on the notes for collecting it says to leave the zone as untouched as possible. Let's say you get aggro and kill a mob or two, what are the consequences ? Should you start over or wait for the mob to spawn again ? Just curious on this.

Derision
12-21-2010, 01:57 PM
Aggroing mobs has two consequences.

First, if you source in the collect, the pathing data will include the movement the mob made after it was aggroed.

Second, if you kill the mob and let it respawn, then the collect will include the data for both spawns of the mob and you will have duplicate data.

If it's just one or two mobs, I wouldn't worry about it. As I said in the other thread, the data produced by the collector is just your starting point for building a zone. Given you have to add all the loot, faction, fine-tune pathing etc, manually deleting a duplicate mob or two is not a big deal.

provocating
12-21-2010, 01:59 PM
Thanks for answering, I am still learning.

I will try to stay invisible and out of aggro range.

robinreg
02-03-2011, 06:03 PM
I know this was designed for everquest on PC due to opcodes. But is it possible to be able to do collects on EQMac? I was just curious since I am tempted to get a cheap mac to play on EQMac.

Secrets
02-03-2011, 06:35 PM
I know this was designed for everquest on PC due to opcodes. But is it possible to be able to do collects on EQMac? I was just curious since I am tempted to get a cheap mac to play on EQMac.

Short answer: No.

Long answer: It uses a different packet protocol than 0.6.1 and later EQEmu clients. Even if you got it up and running, you would need a packet collector for the old version of EverQuest rather than the new client packet protocol. There's a few of them out there, but I can't think of any that would compile, let alone work on a mac. You would need a hub running linux to capture traffic.

Funkey Monkey
04-04-2011, 10:08 PM
I've brushed off the old PEQ repo for logs and started posting what we have so far at http://www.projecteq.net/logs

You'll need to create a PEQ forum name and log in (http://www.projecteq.net/phpBB2) first, but then anybody is free to upload and download any logs they have, or logs in the repo.

All of the logs currently up are zipped/rarred SQLs using EQExtractor 1.4. However, if you wish feel free to post the pcaps. Please don't post SQLs created with anything less than 1.4, as they aren't very versatile among different databases.

Sorry to rez a dead thread but been while since i used this and was wondering if logs that have been uploaded are hosted somewhere else now? Links are dead. Any help is appreciated, would like to get some of the logs i uploaded before and others as I no longer have them saved on HD. Thanks

sorvani
04-04-2011, 10:30 PM
PEQ changed website addresses last fall change your links from projecteq.net to peqtgc.com and they work.

Funkey Monkey
04-05-2011, 12:25 AM
nm figured it out, wont let ya log in from the logs page have to log in from the phpbb2 first then go to logs.