ruby extractor tool
Hello, i am working on a ruby extension to read the packet logs. This is very experimental, currently ;)
I am not sure what structs are important! to extract What format should the tool output? Here is some sample output Log created: 2009-12-19 15:00:29 +0100 Code:
Opcode: 43ac OP_NewZone Code:
require 'ext/EQEmu' |
Excellent work :)
I think some of the opcodes Cavedude would be most interested in are: OP_NewSpawn OP_SpawnDoor OP_GroundSpawn OP_NewZone with a command line flag for the extractor tool to spit out SQL ready to populate the database. Also: OP_ClientUpdate to create mob pathing grids. However this is more problematic to create automatically as they would really need visual inspection to see where the mob completed a circuit of it's patrol route in order to avoid redundant waypoints. |
Yeah, that is some nice work, Erde :)
What are the 2 files you are requiring at the beginning there? Have you considered using ShowEQ logs instead of the .pf log? ShowEQ log files might be a bit easier to read and I am pretty sure we can post accurate structures for them to update and they will update them so we can use their struct files with the tool. That might make upkeep a bit easier. Then we would have 2 projects helping to keep structs and opcodes up to date so the tool continues to work instead of getting outdated if people from here stop updating it. I think Derision already hit most of the important packets we would want. You would just need to have it do MySQL queries to get them into the database. Though, until it was refined, it would probably be best to only update test databases and then copy over the good updates into the production database after verifying all went well. I don't know how the old system handled pathing, but as Derision mentioned, it isn't an easy task to do. Ideally, you would want to be the only char in the zone while collecting pathing information or you risk people pulling mobs and throwing off your pathing info. We probably also want zone points: OP_SendZonepoints And, I wouldn't mind having AAs to help get the SoF AA stuff worked out: OP_SendAATable Since for basic database population we really only require a few packet types, it probably wouldn't be too hard to keep this tool updated. As long as it can see Null Terminators and be able to do the proper conversions for 8bit, 16bit, 32bit, Strings, and floats, I don't think it would be too hard to add new fields or adjust structures and such when needed. Then, we would just need a config file for updating the few Opcodes we want to collect info from and another config file for updating the actual structures. Just being able to collect doors, objects and spawns alone would be a huge step in the right direction :) |
Thanks Derision and Trevius
Quote:
But after getting all to work, i have decided to start from scratch ;) To many problems, specialy type conversation uint16 to Ruby::Integer ... The current implementation is completly written in ruby and seems to work very well. Quote:
This tool needs ruby 1.9 ! Sorry ruby 1.8 isn't supported yet! You need to install the 'BinData' gem Code:
gem install bindata http://www.eqdaemons.de/?page_id=6 Ok, this is a work in progress so dont expect too much ;) Usage: Code:
EQEmulator Extractor Version 0.1.1 Handled opcodes so far: OP_ClientUpdate OP_NewZone OP_GroundSpawn // Partially working ops, more work on the Spawn_Struct needed OP_NewSpawn OP_ZoneSpawns OP_ZoneEntry in utils/eqopcodes is a tool to update the patch_Live.conf from ShowEQ's worldopcodes.xml and zoneopcodes.xml copy patch_Live.conf,worldopcodes.xml and zoneopcodes.xml into that directory Code:
ruby merge.rb patch_Live.conf.new = updated patch_Live.conf showeq_ops.txt = opcodes from seq not found in patch_Live.conf eqemu_ops.txt = opcodes from eqemu not found in seq files This is collected data from the OP_ClientUpdate opcode, could this values be valid? Code:
spawnId, padding0000, deltaX, padding0005, deltaHeading, deltaY, padding0006, y, animation, padding0010, heading, x, padding0014, z, deltaZ |
No, looking at your client update collected data looks all wrong to me. Most likely, your packet structure is way off. Unfortunately, clientupdate is one of the packets that SOE likes to completely jumble around every patch or so to throw off ShowEQ people for a couple of days lol. So, most likely the structure you are using is for an older version of the client.
For testing purposes while creating your tool, you might be better off using structs from EQEmu to test it collecting data for either Titanium or SoF, whichever you chose. Both should be considerably more accurate packet structures than what ShowEQ has for them. Also, you would probably want to test on the SoF client, since it is by far closer to Live than Titanium is. Then, once you know your tool is able to collect everything how you want it, it is just a matter of adjusting the packet structures and opcodes to match Live. For only a few packets like this, it should be very easy. I can figure out almost all of the packet structures we might need very quickly. Though, the clientupdate one is a bit harder for me, due to how they seem to throw in crap we don't care about as what we call padding. I think clientupdate is one of the only (or very few) packets that actually breaks int32s into separate sections, and only uses some of the bits. Without looking at the ISM output from IDA, it is hard to figure out what goes where. And, I don't know how to read ISM well enough to figure that stuff out yet. I had to lean on what info ShowEQ had and do some testing to finally get it working properly for SoF. Here is the OP_ClientUpdate struct we use in SoF: Code:
struct PlayerPositionUpdateServer_Struct I believe this is the current ShowEQ version for EQLive (as of 12/18) for OP_ClientUpdate: Code:
struct playerSpawnPosStruct |
Hello,
i have updated the extractor, you could download it here: http://www.eqdaemons.de/?page_id=6 currently generated csv files: door.csv = Doors (OP_SpawnDoor) object.csv = Objects (OP_GroundSpawn) spawn.csv = Spawns (OP_ZoneEntry) zone.csv = Zone (OP_NewZone) zonepoints.csv = Zone Points (OP_SendZonepoints) Still in development: ( i am still fighting with this ) MobPos.csv = Mob position updates, not usefull yet ! SpawnPos.csv = Spawn position updates, not usefull yet ! This is still in development! I will try to add sql output like the old extractor had. greetings Stefan |
Hi,
new release => http://www.eqdaemons.de/?page_id=6 Changes: - new csv export (all fields are exported now) |
Hi,
new release => http://www.eqdaemons.de/?page_id=6 Changes: - csv export fixes - better spawn handling( OP_ZoneEntry, OP_ZoneSpawns ) |
I might have to pay to get my EQLive account running again so I can test this tool out. Looks like some very nice work :)
|
Does anyone have a copy of the source? The links appear to be dead.
|
All times are GMT -4. The time now is 10:22 AM. |
Powered by vBulletin®, Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.