EQEmulator Forums

EQEmulator Forums (https://www.eqemulator.org/forums/index.php)
-   Support::Packetcollector (https://www.eqemulator.org/forums/forumdisplay.php?f=602)
-   -   ruby extractor tool (https://www.eqemulator.org/forums/showthread.php?t=30271)

erde 01-08-2010 02:09 PM

ruby extractor tool
 
Hello, i am working on a ruby extension to read the packet logs. This is very experimental, currently ;)

I am not sure what structs are important! to extract
What format should the tool output?

Here is some sample output
Log created: 2009-12-19 15:00:29 +0100

Code:

Opcode: 43ac OP_NewZone
Char Name: xxx
Long Name: The Plane of Knowledge
Short Name: PoKnowledge
zone: poknowledge
Type: 0
fog_red: [50, 50, 50, 50]
fog_green: [50, 50, 50, 50]
fog_blue: [155, 155, 150, 150]
fog_minclip: [400.0, 400.0, 400.0, 400.0]
fog_maxclip: [2000.0, 2000.0, 2000.0, 2000.0]
gravity: 0.400000005960464
time_type: 2
Sky: 0
zone_exp_multiplier: 1.0
safe_y: -148.0
safe_x: -285.0
safe_z: -159.0
max_z: 0.0
underworld: -5000.0
minclip: 50.0
maxclip: 2000.0
Short Name2: poknowledge
zone_id: 202
zone_instance: 0
fall_damage: 1

Opcode: 5c85 OP_GroundSpawn
Name: IT10645_ACTORDEF
Linked list [0, 142262936]
Drop Id 21
Zone Id 202
Zone Instance 0
Heading 255.0
Size 1.0
X 538.0
Y -405.0
Z -88.0
Object Type 1
Spawn Id 0
unknown096 [255, 255, 255, 255]

This ist the current source of the tool:
Code:

require 'ext/EQEmu'
require 'Live_Structs'

puts "GroundSpawn Test"

begin
        pfr = PacketFile::RbPacketFileReader.new
rescue Exception => e
        puts "#{ e } (#{ e.class })!"
end

begin
        pfr.OpenFile( "packetlog-156738727.pf" )
rescue Exception => e
        puts "#{ e } (#{ e.class })!"
end

begin
        puts pfr.GetStamp()
rescue Exception => e
        puts "#{ e } (#{ e.class })!"
end

@op = OpcodeMgr::RegularOpcodeManager.new
@op.LoadOpcodes( "patch_Live.conf" )
begin
        while pfr.ReadPacket()
                opcode = @op.EQToEmu( pfr.getOpCode() )

                if opcode == EmuOpcodes::OP_GroundSpawn
                        puts "Opcode: " + pfr.getOpCode().to_s(16) + " " + EmuOpcodes::OpcodeName(@op.EQToEmu(pfr.getOpCode()))
                        p = pfr.getPacket()
                        begin
                          a = Object_Struct.new
                                a.read(p)
                                puts "Name: #{a.object_name}"
                                puts "Linked list #{a.linked_list_addr}"
                                puts "Drop Id #{a.drop_id}"
                                puts "Zone Id #{a.zone_id}"
                                puts "Zone Instance #{a.zone_instance}"
                                puts "Heading #{a.heading}"
                                puts "Size #{a.osize}"
                                puts "X #{a.x}"
                                puts "Y #{a.y}"
                                puts "Z #{a.z}"
                                puts "Object Type #{a.object_type}"
                                puts "Spawn Id #{a.spawn_id}"
                                puts "unknown096 #{a.unknown096}"
                                puts
                        rescue Exception => e
                                puts "#{ e } (#{ e.class })!"
                        end
                end
                if opcode == EmuOpcodes::OP_NewZone
                        puts "Opcode: " + pfr.getOpCode().to_s(16) + " " + EmuOpcodes::OpcodeName(@op.EQToEmu(pfr.getOpCode()))
                        p = pfr.getPacket()
                        begin
                                a = NewZone_Struct.new
                                a.read(p)
                                puts "Char Name: #{a.char_name}"
                                puts "Long Name: #{a.zone_long_name}"
                                puts "Short Name: #{a.zone_short_name}"
                                puts "zone: #{a.zone_short_name2}"
                                puts "Type: #{a.ztype}"
                                puts "fog_red: #{a.fog_red}"
                                puts "fog_green: #{a.fog_green}"
                                puts "fog_blue: #{a.fog_blue}"
                                puts "fog_minclip: #{a.fog_minclip}"
                                puts "fog_maxclip: #{a.fog_maxclip}"
                                puts "gravity: #{a.gravity}"
                                puts "time_type: #{a.time_type}"
                                puts "Sky: #{a.sky}"
                                puts "zone_exp_multiplier: #{a.zone_exp_multiplier}"
                                puts "safe_y: #{a.safe_y}"
                                puts "safe_x: #{a.safe_x}"
                                puts "safe_z: #{a.safe_z}"
                                puts "max_z: #{a.max_z}"
                                puts "underworld: #{a.underworld}"
                                puts "minclip: #{a.minclip}"
                                puts "maxclip: #{a.maxclip}"
                                puts "Short Name2: #{a.zone_short_name2}"
                                puts "zone_id: #{a.zone_id}"
                                puts "zone_instance: #{a.zone_instance}"
                                puts "fall_damage: #{a.fall_damage}"
                                puts
                        rescue Exception => e
                                puts "#{ e } (#{ e.class })!"
                        end
                end
                if opcode == EmuOpcodes::OP_PlayerProfile
                  p = pfr.getPacket()
                        begin
                                a = CharProfileStruct.new
                                a.read(p)
                                puts

                        rescue Exception => e
                                puts "#{ e } (#{ e.class })!"
                        end
                end
        end
rescue Exception => e
        puts "#{ e } (#{ e.class })!"
end

pfr.CloseFile()
puts "End"


Derision 01-09-2010 02:20 PM

Excellent work :)

I think some of the opcodes Cavedude would be most interested in are:

OP_NewSpawn
OP_SpawnDoor
OP_GroundSpawn
OP_NewZone

with a command line flag for the extractor tool to spit out SQL ready to populate the database.

Also:

OP_ClientUpdate

to create mob pathing grids. However this is more problematic to create automatically as they would really
need visual inspection to see where the mob completed a circuit of it's patrol route in order to avoid
redundant waypoints.

trevius 01-10-2010 12:22 AM

Yeah, that is some nice work, Erde :)

What are the 2 files you are requiring at the beginning there?

Have you considered using ShowEQ logs instead of the .pf log? ShowEQ log files might be a bit easier to read and I am pretty sure we can post accurate structures for them to update and they will update them so we can use their struct files with the tool. That might make upkeep a bit easier. Then we would have 2 projects helping to keep structs and opcodes up to date so the tool continues to work instead of getting outdated if people from here stop updating it.

I think Derision already hit most of the important packets we would want. You would just need to have it do MySQL queries to get them into the database. Though, until it was refined, it would probably be best to only update test databases and then copy over the good updates into the production database after verifying all went well.

I don't know how the old system handled pathing, but as Derision mentioned, it isn't an easy task to do. Ideally, you would want to be the only char in the zone while collecting pathing information or you risk people pulling mobs and throwing off your pathing info.

We probably also want zone points:
OP_SendZonepoints

And, I wouldn't mind having AAs to help get the SoF AA stuff worked out:
OP_SendAATable

Since for basic database population we really only require a few packet types, it probably wouldn't be too hard to keep this tool updated. As long as it can see Null Terminators and be able to do the proper conversions for 8bit, 16bit, 32bit, Strings, and floats, I don't think it would be too hard to add new fields or adjust structures and such when needed. Then, we would just need a config file for updating the few Opcodes we want to collect info from and another config file for updating the actual structures.

Just being able to collect doors, objects and spawns alone would be a huge step in the right direction :)

erde 01-10-2010 02:25 PM

Thanks Derision and Trevius

Quote:

What are the 2 files you are requiring at the beginning there?
I have started to create a extension to ruby, i have used swig to wrap c++ to ruby. This tells ruby to use my extension.

But after getting all to work, i have decided to start from scratch ;) To many problems, specialy type conversation uint16 to Ruby::Integer ...
The current implementation is completly written in ruby and seems to work very well.

Quote:

Have you considered using ShowEQ logs instead of the .pf log?
I thought there is already a tool? The main reason was, i wanted to know if i am capable of writting such a tool in ruby.

This tool needs ruby 1.9 ! Sorry ruby 1.8 isn't supported yet!
You need to install the 'BinData' gem
Code:

gem install bindata
You could get the source here

http://www.eqdaemons.de/?page_id=6

Ok, this is a work in progress so dont expect too much ;)
Usage:
Code:

EQEmulator Extractor Version 0.1.1
Usage: extractor [options] logfile
Options:
        --csv        Export to CSV-file

The export function generates: door.csv, object.csv, pos.csv, spawn.csv and zone.csv

Handled opcodes so far:
OP_ClientUpdate
OP_NewZone
OP_GroundSpawn

// Partially working ops, more work on the Spawn_Struct needed
OP_NewSpawn
OP_ZoneSpawns
OP_ZoneEntry

in utils/eqopcodes is a tool to update the patch_Live.conf from ShowEQ's worldopcodes.xml and zoneopcodes.xml
copy patch_Live.conf,worldopcodes.xml and zoneopcodes.xml into that directory
Code:

ruby merge.rb
will generate 3 files
patch_Live.conf.new = updated patch_Live.conf
showeq_ops.txt = opcodes from seq not found in patch_Live.conf
eqemu_ops.txt = opcodes from eqemu not found in seq files

This is collected data from the OP_ClientUpdate opcode, could this values be valid?
Code:

spawnId, padding0000, deltaX, padding0005, deltaHeading, deltaY, padding0006, y, animation, padding0010, heading, x, padding0014, z, deltaZ
0815, 0, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 16386, 0, 28160, 0
0815, 48, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 16386, 0, 28160, 0
0815, 1056, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 16386, 0, 28160, 0
0815, 2448, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 16386, 0, 28160, 0
0815, 3104, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 17698, 0, 28160, 0
0815, 3520, 0, 96, 96, 0, 128, 43716, 8, 0, 332, 16674, 0, 28160, 0
0815, 3952, 0, 96, 96, 117, 247, 43460, 340, 0, 316, 16386, 0, 28160, 524
2056, 0, 0, 126, 0, 64, 160, 40200, 8, 0, 2854, 7040, 1, 24322, 0
0815, 272, 128, 96, 96, 127, 196, 43204, 778, 3, 300, 20226, 0, 28160, 683
0815, 688, 128, 96, 96, 39, 352, 42948, 619, 4, 300, 16386, 0, 28160, 1621
0815, 1168, 128, 96, 96, 60, 254, 42436, 914, 5, 300, 16386, 0, 28160, 1206
0815, 1632, 128, 96, 96, 51, 416, 42180, 132, 5, 300, 16386, 0, 28160, 853
0815, 2912, 128, 96, 96, 88, 185, 41156, 92, 3, 268, 16386, 0, 28160, 853
0815, 48, 256, 96, 96, 101, 248, 40132, 54, 0, 252, 17986, 0, 28160, 1102
0815, 480, 256, 96, 96, 63, 416, 39876, 147, 4, 252, 16386, 0, 28160, 862
0815, 1776, 256, 96, 96, 73, 231, 38596, 463, 6, 252, 16386, 0, 28160, 1556
2056, 0, 0, 0, 264, 0, 0, 40200, 512, 2, 2853, 7040, 1, 24322, 0


trevius 01-10-2010 10:29 PM

No, looking at your client update collected data looks all wrong to me. Most likely, your packet structure is way off. Unfortunately, clientupdate is one of the packets that SOE likes to completely jumble around every patch or so to throw off ShowEQ people for a couple of days lol. So, most likely the structure you are using is for an older version of the client.

For testing purposes while creating your tool, you might be better off using structs from EQEmu to test it collecting data for either Titanium or SoF, whichever you chose. Both should be considerably more accurate packet structures than what ShowEQ has for them. Also, you would probably want to test on the SoF client, since it is by far closer to Live than Titanium is.

Then, once you know your tool is able to collect everything how you want it, it is just a matter of adjusting the packet structures and opcodes to match Live. For only a few packets like this, it should be very easy. I can figure out almost all of the packet structures we might need very quickly. Though, the clientupdate one is a bit harder for me, due to how they seem to throw in crap we don't care about as what we call padding. I think clientupdate is one of the only (or very few) packets that actually breaks int32s into separate sections, and only uses some of the bits. Without looking at the ISM output from IDA, it is hard to figure out what goes where. And, I don't know how to read ISM well enough to figure that stuff out yet. I had to lean on what info ShowEQ had and do some testing to finally get it working properly for SoF.

Here is the OP_ClientUpdate struct we use in SoF:

Code:

struct PlayerPositionUpdateServer_Struct
{
/*0000*/ uint16  spawn_id;                        // Entity ID of the Spawn/Player
/*0002*/ signed  padding0000:12;        // ***Placeholder
                signed  x_pos:19;                        // x coord
                signed  padding0290:1;        // ***Placeholder
/*0006*/ signed  delta_x:13;                // change in x
                signed  delta_y:13;                // change in y
                signed  padding0294:6;        // ***Placeholder
/*0010*/ signed  z_pos:19;                        // z coord
                signed  delta_heading:10;        // change in heading
                signed  padding0298:3;        // ***Placeholder
/*0014*/ signed  y_pos:19;                        // y coord
                signed  delta_z:13;                // change in z
/*0022*/ signed  animation:10;                // animation
                unsigned heading:12;                // heading
                signed  padding0302:10;        // ***Placeholder
/*0026*/
};

I am sure that has been completely re-arranged by now on Live. ShowEQ does require at least some of these fields to be correct for their tool to work at all, so their structure should be at least partially correct if ShowEQ works with Live currently to show NPC movement.

I believe this is the current ShowEQ version for EQLive (as of 12/18) for OP_ClientUpdate:

Code:

struct playerSpawnPosStruct
{
/*0000*/ uint16_t spawnId;
/*0002*/ signed  padding0000:12; // ***Placeholder
        signed  deltaX:13;      // change in x
        signed  padding0005:7;  // ***Placeholder
/*0006*/ signed  deltaHeading:10;// change in heading
        signed  deltaY:13;      // change in y
        signed  padding0006:9;  // ***Placeholder
/*0010*/ signed  y:19;          // y coord
        signed  animation:10;  // animation
        signed  padding0010:3;  // ***Placeholder
/*0014*/ unsigned heading:12;    // heading
        signed  x:19;          // x coord
        signed  padding0014:1;  // ***Placeholder
/*0018*/ signed  z:19;          // z coord
        signed  deltaZ:13;      // change in z
/*0022*/
};

If you know the exact loc of an NPC, you can break down the packets using the suggested packet structs from ShowEQ and see if it matches the loc info. Delta is hard to figure out this way, but X, Y and Z aren't too bad. The only problem there is that there is no way to get a #loc of an NPC from Live. It is best to just have someone that is good with IDA to figure that particular struct out. The other structs are all pretty straight forward, and don't normally change much either.

erde 01-15-2010 05:40 PM

Hello,

i have updated the extractor, you could download it here: http://www.eqdaemons.de/?page_id=6

currently generated csv files:
door.csv = Doors (OP_SpawnDoor)
object.csv = Objects (OP_GroundSpawn)
spawn.csv = Spawns (OP_ZoneEntry)
zone.csv = Zone (OP_NewZone)
zonepoints.csv = Zone Points (OP_SendZonepoints)

Still in development: ( i am still fighting with this )
MobPos.csv = Mob position updates, not usefull yet !
SpawnPos.csv = Spawn position updates, not usefull yet !

This is still in development! I will try to add sql output like the old extractor had.

greetings Stefan

erde 01-16-2010 08:04 AM

Hi,

new release => http://www.eqdaemons.de/?page_id=6

Changes:
- new csv export (all fields are exported now)

erde 01-18-2010 05:47 AM

Hi,

new release => http://www.eqdaemons.de/?page_id=6

Changes:
- csv export fixes
- better spawn handling( OP_ZoneEntry, OP_ZoneSpawns )

trevius 01-18-2010 05:54 AM

I might have to pay to get my EQLive account running again so I can test this tool out. Looks like some very nice work :)

sereal 10-18-2017 05:36 PM

Does anyone have a copy of the source? The links appear to be dead.


All times are GMT -4. The time now is 11:15 PM.

Powered by vBulletin®, Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.