EQEmulator Forums - EQExtractor2

Page 2 of 5

Show 40 post(s) from this thread on one page

EQEmulator Forums (https://www.eqemulator.org/forums/index.php)

- Support::Packetcollector (https://www.eqemulator.org/forums/forumdisplay.php?f=602)

- - EQExtractor2 (https://www.eqemulator.org/forums/showthread.php?t=31354)

Akkadius

05-30-2010 08:55 PM

Quote:

Originally Posted by cavedude (Post 188463)

I agree with Trevius. Although I would personally disable the IDs in the program, I think it should still be an option for others.

I just got back from ball, I agree completely that this should be a universal application for dumping collects, but should also be customizable if need be.

gaeorn

05-31-2010 01:49 AM

Quote:

Originally Posted by Akkadius (Post 188456)

A loot table script would be phenomenal. I wouldn't mind helping with this.

I already farmed all the data from magelo for their loot info for all mobs in all zones. I suppose the same could be done for lucy, but I believe the magelo data is more complete. I can write a script to parse out the usable data without an issue. The problem comes in when it's time to write out that data in some usable form. I haven't started on the script yet because I haven't yet decided how to write out the data. I'm sure I'll either write out SQL or I'll create temp tables and dump the data there, but I'm sure it'll take a human to merge it in cleanly with the PEQ db.

If you want to help, look at the magelo drop data and see what ideas you come up with for using it. I'd love to hear more ideas about how best to use that data.

gaeorn

05-31-2010 01:52 AM

Quote:

Originally Posted by Derision (Post 188457)

I did wonder if I could set variables for the starting Insert IDs at the start of the generated SQL and just reference them with increments afterwards, so you could just change
a few variables at the start to the next free IDs. I didn't know if I could do that in SQL, but I just tested it with select statements, and it seems to be possible:

Code:

set @myinsertid = 1001;

select id, name from npc_types where id = @myinsertid;

select id, name from npc_types where id = @myinsertid + 1;

Output:

Code:

mysql> source test.sql

Query OK, 0 rows affected (0.00 sec)



+------+-------------+

| id   | name        |

+------+-------------+

| 1001 | Guard_Mezzt |

+------+-------------+

1 row in set (0.00 sec)



+------+--------------+

| id   | name         |

+------+--------------+

| 1002 | Guard_Jerith |

+------+--------------+

1 row in set (0.00 sec)

So maybe that is the way to go.

Why not just fill in the ID with a sub select? You could simply do a (select max(id) from table)+1 where you would otherwise put in the ID. Or you could at least use a select to automatically set the variable at the start and otherwise do it as you have above.

gaeorn

05-31-2010 01:56 AM

Quote:

Originally Posted by trevius (Post 188468)

I don't know the details of how the tool works exactly, but it seems like it parses the .pcap when you first load it. Perhaps you could just give an option to generate a file of that parse, which could then be loaded and used to generate SQL from. Figured that may be another possible option for creating files anyone could use and share without worrying about private info. Another nice bonus to using a parse instead of some generated SQL is that a parse should be able to hold data that may not currently be set to be generated into SQL yet, so as the extractor tool is refined/expanded, those same parses could be ran through the tool again for an instant update with the latest features. This should prevent people from having to run their own .pcaps through the tool and sending them again. Instead, Cavedude could just run the parses through the tool, and have the latest with minimal work.

Why not just have a tool that zeros out any personal info from the .pcap files so they could be sent in as is. Then the full communication between client and server is complete in case we figure out something additional to extract from it later, but it won't contain any personal information that a player may wish kept from the general populace.

trevius

05-31-2010 04:08 AM

Quote:

Originally Posted by gaeorn (Post 188490)

I think the old collectors originally used to do just that, but since then, SOE has added quite a few more packets with character names, and probably other information, and it might be hard to track them all down. I suppose we could put together a list of all possible packets we may ever need collected, and just filter out the rest of the unneeded ones. As-is, I think Derision already has the most important packets being collected. As far as I can think, the only other one we might even want at this point would be OP_SpawnAppearance.

To make use of many more packets would require a different kind of collecting, which would be more interactive. The collecting this tool currently does best works when there is no interaction at all with the zone other than maybe checking merchants for merchant lists. Maybe in the future, other types of collects could be made that collect things like tasks, loot drops, spell sets on NPCs, or any other number of things that require considerably more time to collect enough info for it to be useful. Most of that stuff would probably be better off just adding it manually after a little research from various websites. Though, tasks might be one thing we could get enough info from without much time, but Derision would be much better to answer that.

cavedude

05-31-2010 09:15 AM

Just a minor suggestion, can the zone data and zone_points query be given checkboxes as well? If they are checked, they will be in the log uncommented, if unchecked they will be completely ignored?

Derision

05-31-2010 01:03 PM

1.4 is up in the downloads section:

Changes:

Code:

==05/31/2010==

Populated Melee texture columns.

For playable races, equip_chest2 is set based on the equipped chest item, if one is equipped.

Set armour tint columns based on the colour field for the chest slot.



The starting IDs for database inserts are now set as variables at the top of the SQL:

  Note the Door ID has changed meaning. Instead of being the value for the id column in the doors table,

  it is now the base value for the doorid column. The id column will just use the next autoincrement id.



If the version number is none-zero, The base doorid = Version * 1000 and all other insert IDs are

  set to ZoneID * 1000 + (Version * 100).



If the version number is none-zero, deletes for npc_types and the spawn tables will only be generated from

  the starting InsertID to starting ID + 99.



Moved the Version NumericUpDown control to before the InsertID textboxes as changing the version will

 auto-change the InsertIDs.



Corrected a bug where IT10805 objects were flagged as ovens instead of brew barrels.

Added checkboxes (default unchecked) for zone config (fog, safe point, etc) and zone points.

Akkadius

05-31-2010 03:27 PM

I've done multiple zones and seen the quality extract, but PoK for some reason is just completely jacked up. There are multiples of spawns, and all on the guard pathing and I have redone the collect 4 times after the initial one which I took about 20-30 minutes to do to get the 170 merchant lists in the zone.

Here is the .pcap and the .sql, let me know what you think.

http://www.sendspace.com/file/wclue5

Derision

05-31-2010 03:49 PM

My first thought is you probably have the original PEQ spawns for POK which use different ID ranges than my app generates, and so it doesn't remove them.

If you do select id, spawngroupID from spawn2 where zone='poknowledge'; and you get ids not in the range 202000 - 202999, then that is the issue. You will need to manually trawl through your DB and delete the original PEQ data for that zone.

If that's not the problem, let me know.

Akkadius

05-31-2010 03:55 PM

Quote:

Originally Posted by Derision (Post 188499)

I forgot to mention that that isn't the problem.

Derision

05-31-2010 04:17 PM

I sourced the SQL into an otherwise empty test DB and browsed through it, and didn't see any obvious duped spawns.

I don't feel inclined to take the time tonight to create a DB cleansed of PEQ PoK data so I can actually source the SQL in and test it in game, but if you don't figure out
what the problem is, I'll see if I can get to it during the week ahead.

Derision

05-31-2010 04:24 PM

Actually, I do see dupes, for #Tzudani Emissary of Veeshan (lots of them). The packet dump shows lots of OP_ZoneEntry packets for him, as if he spawned
mulitple times during the collect. I'll look at it further tomorrow.

Akkadius

05-31-2010 04:35 PM

Quote:

Originally Posted by Derision (Post 188502)

Sounds good, I had to go to work so I wasn't going to troubleshoot it further.

cavedude

05-31-2010 07:35 PM

I've brushed off the old PEQ repo for logs and started posting what we have so far at http://www.projecteq.net/logs

You'll need to create a PEQ forum name and log in (http://www.projecteq.net/phpBB2) first, but then anybody is free to upload and download any logs they have, or logs in the repo.

All of the logs currently up are zipped/rarred SQLs using EQExtractor 1.4. However, if you wish feel free to post the pcaps. Please don't post SQLs created with anything less than 1.4, as they aren't very versatile among different databases.

Tricyclethief

05-31-2010 09:48 PM

Thanks cavedude! i should be finished by tonight and have about all new SOD zones collected.

steve

05-31-2010 10:46 PM

Quote:

Originally Posted by Derision (Post 188502)

#Tzudani Emissary of Veeshan was permakilled by GMs on Live. Meaning the instant he appears in-game, he automagically self-kills. This was the Dragons, Dragons, Dragons event that is disabled on Live. His spawn rate must be set very low, hence the numerous dupes of him spawning. Dunno why they would depop him that way instead of their scripted means, but ah well... If you have a fast enough connection, you can sit outside the Guild Lobby zonein, facing the tree and see him pop into view for a millisecond and then poof. He's right behind the portal spot for PoK.

Akkadius

06-01-2010 03:31 AM

Just uploaded entire TSS and almost entire SoD minus some Void versions.

8 D

robinreg

06-01-2010 04:25 AM

uploaded the entire DoD zones minus instances. I sourced in the Dreadspire Keep into my test server and it really populates everything. even the pathing. Cool to even see Master Vule there haha. That's is so awesome. I will try to get some DoN in also. I may even open up a 2nd account to start doing instances for DoN and DoD.

steve

06-01-2010 01:11 PM

Quote:

Originally Posted by Akkadius (Post 188517)

Just uploaded entire TSS and almost entire SoD minus some Void versions.

8 D

Which Void zones did you miss? I can see about grabbing them.

Derision

06-01-2010 01:17 PM

Quote:

Originally Posted by Akkadius (Post 188498)

PoK for some reason is just completely jacked up.

I blew away PoK from my PEQ database and sourced in your SQL and it looks fine to me.

The only duped spawn I could see is that Veeshan Emissary guy which Live is constantly spawning and immediately killing.

I have found it is not unusual to see mobs pathing around buildings or walls to actually walk through them on occasion, which I believe is due to you being out of the range at which you get constant movement updates for them, at some point during the packet collect.

The only other thing I noticed was the player mounts that got collected. I'll fix that in the next release.

Akkadius

06-01-2010 02:32 PM

Quote:

Originally Posted by Derision (Post 188531)

Player mounts have been in there yes, but not a huge deal imo, I think it helps to get the new versions of mounts collected.

I've wiped my PoK slate, #repop and 0 NPC's. When I simply reenter my 35 minute collect with the 170 merchants I get duplicates of NPC's everywhere, NPC's pathing everywhere, NPC's not even close to their remote location way across the zone. I thought it may have been pre-existing grids but that's not it.

Derision

06-01-2010 02:39 PM

Odd. What I did was shutdown the server completely, executed:

Code:

delete from spawn2 where zone like '%poknowledge%';

delete from doors where zone like '%poknowledge%';

delete from grid where zoneid = 202;

delete from grid_entries where zoneid = 202;

delete from ground_spawns where zoneid = 202;

delete from object where zoneid = 202;

Then sourced in your SQL and restarted the server.

If you do #npcstats on each occurrence of the same duped spawns, are they both the same npcid and spawngroup (and in the 202nnn range ?).

cavedude

06-01-2010 02:56 PM

Sorry to keep being a bother, but checking Zones Points and Zone Config outputs the entries commented out. Any chance that can be changed?

Also, how difficult would it be to add an option to UPDATE npc_types instead of INSERT? If checked, the SQL may look like:

Code:

UPDATE npc_types set field1 = $field1, field2 = $field2 where name = $name and id >= @StartingNPCTypeID and id <= @StartingNPCTypeID + 999;

That would help me to use new collects to update already built zones, since you've added proper texture and face parsing! Of course, the default option should probably create INSERTS for new zones :)

I can't thank you enough, this seriously is the best parser EQEmu has ever seen! My hope is the next version of the PEQ database will have every zone supported by our client at least spawned with doors, objects, fog, etc.

Akkadius

06-01-2010 03:05 PM

Quote:

Originally Posted by cavedude (Post 188537)

Code:

UPDATE npc_types set field1 = $field1, field2 = $field2 where name = $name and id >= @StartingNPCTypeID and id <= @StartingNPCTypeID + 999;

Should be an option between Insert and Update really.

Derision

06-01-2010 03:20 PM

Quote:

Originally Posted by cavedude (Post 188537)

Sorry to keep being a bother, but checking Zones Points and Zone Config outputs the entries commented out. Any chance that can be changed?

I had misread your original request which is why I left them commented out. I've changed it locally and it will be in the next release, which won't be until tomorrow at least.

Quote:

Also, how difficult would it be to add an option to UPDATE npc_types instead of INSERT? If checked, the SQL may look like:

Code:

UPDATE npc_types set field1 = $field1, field2 = $field2 where name = $name and id >= @StartingNPCTypeID and id <= @StartingNPCTypeID + 999;

That would help me to use new collects to update already built zones

I assume that if the option to generate UPDATEs like that is selected, then you wll also not want the INSERTs generating for the spawn tables ?

Finally, you are not being a bother at all. Of anyone, you know best as to how a tool like this needs to work from a user's perspective :)

Derision

06-01-2010 03:24 PM

Code:

UPDATE npc_types set field1 = $field1, field2 = $field2 where name = $name and id >= @StartingNPCTypeID and id <= @StartingNPCTypeID + 999;

I've just thought of an issue with this. What about the case where multiple mobs exist with the same name with different attributes (level, hp, possibly other attributes) ?

Do you only want a selected number of fields included in the update, like face, texture, etc ?

cavedude

06-01-2010 04:23 PM

Yes, I know in my case if I choose the UPDATE option, I'll only be looking to update the npc_types, and will leave all other options unchecked.

That's a good thought, about NPCs with the same name and different levels or other critical stats. What I am mainly looking for is the appearance data, so yes if the query could be limited to textures, face, the various color and hair columns, armortints, and melee columns, etc that would be ideal! Also, I noticed the Drakkin specific appearance columns aren't parsed, are they not sent? No big deal if not. The data I am working with now is far more detailed than ever!

Derision

06-01-2010 04:35 PM

Good catch about the drakkin data. I decode it from the packet, just don't include it in the INSERT. I'll include that in the list of things to add.

Akkadius

06-01-2010 05:31 PM

I've only posted TSS and SoD because I felt I had all of the zones. I have quite a few bits and pieces of others

EDIT: Yes Derision I got PoK to work after some trial and error, but there was a dramatic difference which makes no sense but yeah 8 D.

steve

06-01-2010 06:51 PM

Will this work for Test Server? If not, is there a way to get it to work?

Was thinking best way might be to testcopy and do some suicide runs.

Akkadius

06-01-2010 07:00 PM

Quote:

Originally Posted by steve (Post 188553)

Will this work for Test Server? If not, is there a way to get it to work?

Was thinking best way might be to testcopy and do some suicide runs.

What are you trying to say?

steve

06-01-2010 07:40 PM

Answered my own question. Appears to work fine on Test Server.

Anyway, I noticed when running collects in The Void, the spawn of Zebuxoruk is not logged. Verified with Akkadius' collects that he uploaded as well. He's a special merchant that you buy from using Chronobines.

Andrew80k

06-01-2010 09:13 PM

Quote:

Originally Posted by steve (Post 188555)

He also does some of the void access quests.

Akkadius

06-01-2010 10:40 PM

He probably has classification of a different merchant type which isn't in Derision's script to filter and capture.

Akkadius

06-02-2010 12:28 AM

I'm gathering as much of PoR as I possibly can without the instances. I will have all of the statics tonight. As well as DoN statics. So whoever is doing collects I would work on another expansion.

Although there are some very very dynamic zones such as "The Devastation" where there are several factions of NPC's within a zone and they take each other over, I am collecting it as well as I possibly can. Because sitting in Devastation isn't going to get the zone collected thoroughly because we need working faction ID's to make it all work. And then they have weapons and armory that spawn as a result of a certain faction taking over a zone, such as walls and catapults. Rage is similar.

robinreg

06-02-2010 12:48 AM

I got all the DoD and PoR zones. the one I got for devastation is the Bolvark spawns. I labeled it on the PEQ log site. Also got most of the revamps zones. I'm currently working on GoD instance zones.

Akkadius

06-02-2010 01:14 AM

Quote:

Originally Posted by robinreg (Post 188563)

I'm grabbing quite a few of the PoR instance zones.

Derision

06-02-2010 03:57 AM

Quote:

Originally Posted by Akkadius (Post 188559)

He probably has classification of a different merchant type which isn't in Derision's script to filter and capture.

The only spawns that should not be included are players, pets and mercenaries. I'd be interested in looking at the .pcap or the hex dump produced by
the 'Dump Packets' button for the Void zone to figure out why Zebuxorok isn't being included, if you or Steve could email it to me at DerisionEQ at gmail.com

Akkadius

06-02-2010 04:15 AM

Quote:

Originally Posted by Derision (Post 188565)

Done, let me know what you think.

Derision

06-02-2010 05:17 AM

Quote:

Originally Posted by Akkadius (Post 188566)

Done, let me know what you think.

There is nothing unusual about his spawn packet, and if I run that .pcap through an older, command-line version of the extractor, it generates SQL for Zebuxoruk just fine.

I'm at work right now and don't have the current source code with me, but it must be a simple bug that I will fix tonight. I'm thinking that maybe the extractor is always missing
the first NPC in the zone, and since Zeb is the only NPC in the Void, then it is noticeable.

All times are GMT -4. The time now is 05:05 AM.

Page 2 of 5

Show 40 post(s) from this thread on one page