Update

[Andrew80k]

I'm making some progress. I got all the zone stuff fixed last night, and started working on the player profile and AA's. I pulled out all the Spawn stuff that I did and have gone back to the drawing board on that. Going to work on some other stuff while I'm figuring out more of how it works.

There are 2 major things that the extractor does. It extracts data that it knows about, and "explores" data that it doesn't. The extraction part isn't too hard. Some of the newer variable length stuff is going to be a bit of a challenge, but nothing too tough. The explorer stuff is a little strange and is going to take a bit of work on my part to figure out.

So far the 2 collects that I made with the collector have been fine. It's getting the data out that is taking all the work.

[Akkadius]

This is very cool Andrew80k, I'm sure I will be using the crap out of this, other as well.

[trevius]

I think AA info will be useful and probably not to hard to extract. Though, I am not sure what we would need from the PP, unless you are just wanting to collect it to be able to identify stuff in it easier or something. I have actually started working a bit on a Live client for EQEmu, but haven't gotten too far just yet. I don't plan to put too much work into it like our other clients, but doing it more for fun and just to see how functional it would be without much work. But, when I am done, I should have the PP all lined up perfectly, or at least close, which if you still need it, you could use what I get on the SVN in the next day or 2.

[Andrew80k]

It doesn't seem to be. I don't think the PP is very useful except where it pertains to AA's to be honest, currently. Using the Explorers function it might be kind of useful for filling in the gaps at some point in the future. It's one of the easier things to get right in the extractor though so I'm starting at the easy stuff while I get a handle on how it works and the structs.

[Andrew80k]

I worked on the Collector most of the night last night as I don't seem to be getting AA's in the collections. I did a collection with my Ranger on live and I'm making progress with the spawns, they're still off some, but getting there. Player profile was just used in getting buffs, but there is a disconnect somewhere in the code for that. Despite me updating the PP struct, it was still not right and I'm not certain where it is off and I'm not going to spend many cycles fixing that at this point. More important stuff is needed. Not sure what the buff stuff is used for at this point anyway. Maybe someone else knows.

[trevius]

I am working on Live still and am now able to get to character select as of last night. I am hoping to make it in-game tonight if it isn't too much trouble. Once I get in-game, it should be easy for me to update all opcodes in the patch_Live.conf file to be 95%+ accurate pretty quickly.

Either way, I went through and updated the PP as best as I can to prepare for my attempts of getting in-game later. Here is how I have it set currently:

// Live May 5 2010 - Size 25312
struct PlayerProfile_Struct
{
/*00000*/ uint32 checksum; //
//BEGIN SUB-STRUCT used for shrouding stuff...
/*00004*/ uint32 gender; // Player Gender - 0 Male, 1 Female
/*00008*/ uint32 race; // Player race
/*00012*/ uint32 class_; // Player class
/*00016*/ uint8 unknown00016[40]; // #### uint32 unknown00016; in Titanium ####uint8[40]
/*00056*/ uint8 level; // Level of player
/*00057*/ uint8 level1; // Level of player (again?)
/*00058*/ uint8 unknown00022[2]; // ***Placeholder
/*00060*/ BindStruct binds[5]; // Bind points (primary is first)
/*00160*/ uint32 deity; // deity
/*00164*/ uint32 intoxication; // Alcohol level (in ticks till sober?)
/*00168*/ uint32 spellSlotRefresh[MAX_PP_MEMSPELL]; // Refresh time (millis) - 4 Octets Each
/*00208*/ uint8 unknown00218[14]; // Seen 00 00 00 00 00 00 00 00 00 00 00 00 02 01
/*00222*/ uint32 abilitySlotRefresh;
/*00226*/ uint8 haircolor; // Player hair color
/*00227*/ uint8 beardcolor; // Player beard color
/*00228*/ uint8 eyecolor1; // Player left eye color
/*00229*/ uint8 eyecolor2; // Player right eye color
/*00230*/ uint8 hairstyle; // Player hair style
/*00231*/ uint8 beard; // Player beard type
/*00232*/ uint8 unknown00236[4]; // was 14
/*00236*/ union
{
struct
{
/*00236*/ EquipStruct equip_helmet; // Equiptment: Helmet visual
/*00248*/ EquipStruct equip_chest; // Equiptment: Chest visual
/*00260*/ EquipStruct equip_arms; // Equiptment: Arms visual
/*00272*/ EquipStruct equip_bracers; // Equiptment: Wrist visual
/*00284*/ EquipStruct equip_hands; // Equiptment: Hands visual
/*00296*/ EquipStruct equip_legs; // Equiptment: Legs visual
/*00308*/ EquipStruct equip_feet; // Equiptment: Boots visual
/*00320*/ EquipStruct equip_primary; // Equiptment: Main visual
/*00332*/ EquipStruct equip_secondary; // Equiptment: Off visual
} equip;
/*00236*/ EquipStruct equipment[9]; //Live Shows [108] for this part
};
/*00344*/ uint8 unknown00224[168]; // Live Shows [160]
/*00512*/ Color_Struct item_tint[9]; // RR GG BB 00
/*00548*/ AA_Array aa_array[MAX_PP_AA_ARRAY]; // [3600] AAs 12 bytes each
/*04148*/ uint32 points; // Unspent Practice points - RELOCATED???
/*04152*/ uint32 mana; // Current mana
/*04156*/ uint32 cur_hp; // Current HP without +HP equipment
/*04160*/ uint32 STR; // Strength - 6e 00 00 00 - 110
/*04164*/ uint32 STA; // Stamina - 73 00 00 00 - 115
/*04168*/ uint32 CHA; // Charisma - 37 00 00 00 - 55
/*04172*/ uint32 DEX; // Dexterity - 50 00 00 00 - 80
/*04176*/ uint32 INT; // Intelligence - 3c 00 00 00 - 60
/*04180*/ uint32 AGI; // Agility - 5f 00 00 00 - 95
/*04184*/ uint32 WIS; // Wisdom - 46 00 00 00 - 70
/*04188*/ uint8 unknown04172[28]; //
/*04216*/ uint8 face; // Player face - Actually int32?
/*04217*/ uint8 unknown02264[147]; // was [175]
/*04364*/ int32 spell_book[MAX_PP_SPELLBOOK]; // List of the Spells in spellbook 720 = 90 pages [2880] was [1920]
/*06284*/ uint8 unknown06284[960]; // Spacer for the end of the book for now (pages 60 to 90)
/*07244*/ int32 mem_spells[MAX_PP_MEMSPELL]; // List of spells memorized
/*07284*/ uint8 unknown04396[28]; //#### uint8 unknown04396[32]; in Titanium ####[28]
/*07312*/ uint32 platinum; // Platinum Pieces on player
/*07316*/ uint32 gold; // Gold Pieces on player
/*07320*/ uint32 silver; // Silver Pieces on player
/*07324*/ uint32 copper; // Copper Pieces on player
/*07328*/ uint32 platinum_cursor; // Platinum Pieces on cursor
/*07332*/ uint32 gold_cursor; // Gold Pieces on cursor
/*07336*/ uint32 silver_cursor; // Silver Pieces on cursor
/*07340*/ uint32 copper_cursor; // Copper Pieces on cursor
/*07344*/ uint32 skills[MAX_PP_SKILL]; // [300] List of skills
/*07644*/ uint8 unknown04760[236];
/*07880*/ uint32 toxicity; // Potion Toxicity (15=too toxic, each potion adds 3)
/*07884*/ uint32 thirst_level; // Drink (ticks till next drink)
/*07888*/ uint32 hunger_level; // Food (ticks till next eat)
/*07892*/ SpellBuff_Struct buffs[BUFF_COUNT]; // [960] Buffs currently on the player (60 Max) - was 600
/*08492*/ uint8 unknown08492[360]; // End of Buffs
/*08852*/ Disciplines_Struct disciplines; // [400] Known disciplines
/*09252*/ uint8 unknown09252[400]; // Discs?
/*09652*/ uint32 recastTimers[MAX_RECAST_TYPES]; // Timers (UNIX Time of last use)
/*09732*/ uint8 unknown08124[160]; // Some type of Timers
/*09892*/ uint32 endurance; // Current endurance
/*09896*/ uint8 unknown09896[20]; // ?
/*09916*/ uint32 aapoints_spent; // Number of spent AA points
/*09920*/ uint32 aapoints; // Unspent AA points
/*09924*/ uint8 unknown06160[4];
/*09928*/ Bandolier_Struct bandoliers[MAX_PLAYER_BANDOLIER]; // [6400] bandolier contents
/*16328*/ PotionBelt_Struct potionbelt; // [360] potion belt 72 extra octets by adding 1 more belt slot
/*16688*/ uint8 unknown12852[8];
/*16696*/ uint32 available_slots;
/*16700*/ uint8 unknown12864[80]; //#### uint8 uint8 unknown12864[76]; in Titanium ####[80]
//END SUB-STRUCT used for shrouding.
/*16780*/ char name[64]; // Name of player
/*16844*/ char last_name[32]; // Last name of player
/*16876*/ uint8 unknown19588[8]; //#### Not In Titanium #### new to SoF[12]
/*16884*/ int32 guild_id; // guildid
/*16888*/ uint32 birthday; // character birthday
/*16892*/ uint32 lastlogin; // character last save time
/*16896*/ uint32 account_startdate; // Date the Account was started - New Field for Live***
/*16900*/ uint32 timePlayedMin; // time character played
/*16904*/ uint8 pvp; // 1=pvp, 0=not pvp
/*16905*/ uint8 anon; // 2=roleplay, 1=anon, 0=not anon
/*16906*/ uint8 gm; // 0=no, 1=yes (guessing!)
/*16907*/ int8 guildrank; // 0=member, 1=officer, 2=guildleader -1=no guild
/*16908*/ uint32 guildbanker;
/*16912*/ uint8 unknown13054[4]; //was [8]
/*16916*/ uint32 exp; // Current Experience
/*16920*/ uint8 unknown13072[8];
/*16928*/ uint32 timeentitledonaccount;
/*16932*/ uint8 languages[MAX_PP_LANGUAGE]; // List of languages
/*16957*/ uint8 unknown13109[7]; //#### uint8 unknown13109[4]; in Titanium ####[7]
/*16964*/ float y; // Players y position (NOT positive about this switch)
/*16968*/ float x; // Players x position
/*16972*/ float z; // Players z position
/*16976*/ float heading; // Players heading
/*16980*/ uint8 unknown13132[4]; // ***Placeholder
/*16984*/ uint32 platinum_bank; // Platinum Pieces in Bank
/*16988*/ uint32 gold_bank; // Gold Pieces in Bank
/*16992*/ uint32 silver_bank; // Silver Pieces in Bank
/*16996*/ uint32 copper_bank; // Copper Pieces in Bank
/*17000*/ uint32 platinum_shared; // Shared platinum pieces
/*17004*/ uint8 unknown13156[1036]; // was [716]
/*18040*/ uint32 expansions; // Bitmask for expansions ff 7f 00 00 - SoD
/*18044*/ uint8 unknown13244[12];
/*18056*/ uint32 autosplit; // 0 = off, 1 = on
/*18060*/ uint8 unknown13260[16];
/*18076*/ uint16 zone_id; // see zones.h
/*18074*/ uint16 zoneInstance; // Instance id
/*18080*/ char groupMembers[MAX_GROUP_MEMBERS][64];// 384 all the members in group, including self
/*18464*/ char groupLeader[64]; // Leader of the group ?
/*18528*/ uint8 unknown13728[604]; // was [348]
/*19132*/ uint32 entityid;
/*19136*/ uint32 leadAAActive; // 0 = leader AA off, 1 = leader AA on
/*19140*/ uint8 unknown14392[4];
/*19144*/ sint32 ldon_points_guk; // Earned GUK points
/*19148*/ sint32 ldon_points_mir; // Earned MIR points
/*19152*/ sint32 ldon_points_mmc; // Earned MMC points
/*19156*/ sint32 ldon_points_ruj; // Earned RUJ points
/*19160*/ sint32 ldon_points_tak; // Earned TAK points
/*19164*/ sint32 ldon_points_available; // Available LDON points
/*19168*/ uint8 unknown14420[68]; // was [136]
/*19236*/ float tribute_time_remaining; // Time remaining on tribute (millisecs)
/*19240*/ uint32 career_tribute_points; // Total favor points for this char
/*19244*/ uint32 unknown7208; // *** Placeholder
/*19248*/ uint32 tribute_points; // Current tribute points
/*19252*/ uint32 unknown7216; // *** Placeholder
/*19256*/ uint32 tribute_active; // 0 = off, 1=on
/*19260*/ Tribute_Struct tributes[MAX_PLAYER_TRIBUTES]; // [40] Current tribute loadout
/*19300*/ uint8 unknown14616[4];
/*19304*/ double group_leadership_exp; // Current group lead exp points
/*19312*/ double raid_leadership_exp; // Current raid lead AA exp points
/*19320*/ uint32 group_leadership_points; // Unspent group lead AA points
/*19324*/ uint32 raid_leadership_points; // Unspent raid lead AA points
/*19328*/ LeadershipAA_Struct leader_abilities; // [128]Leader AA ranks 19332
/*19456*/ uint8 unknown14772[132]; // was [128]
/*19588*/ uint32 air_remaining; // Air supply (seconds)
/*19592*/ uint32 PVPKills;
/*19596*/ uint32 PVPDeaths;
/*19600*/ uint32 PVPCurrentPoints;
/*19604*/ uint32 PVPCareerPoints;
/*19608*/ uint32 PVPBestKillStreak;
/*19612*/ uint32 PVPWorstDeathStreak;
/*19616*/ uint32 PVPCurrentKillStreak;
/*19620*/ PVPStatsEntry_Struct PVPLastKill; // size 88
/*19708*/ PVPStatsEntry_Struct PVPLastDeath; // size 88
/*19796*/ uint32 PVPNumberOfKillsInLast24Hours;
/*19800*/ PVPStatsEntry_Struct PVPRecentKills[50]; // size 4400 - 88 each
/*24200*/ uint32 expAA; // Exp earned in current AA point
/*24204*/ uint8 unknown19516[40];
/*24244*/ uint32 currentRadCrystals; // Current count of radiant crystals
/*24248*/ uint32 careerRadCrystals; // Total count of radiant crystals ever
/*24252*/ uint32 currentEbonCrystals; // Current count of ebon crystals
/*24256*/ uint32 careerEbonCrystals; // Total count of ebon crystals ever
/*24260*/ uint8 groupAutoconsent; // 0=off, 1=on
/*24261*/ uint8 raidAutoconsent; // 0=off, 1=on
/*24262*/ uint8 guildAutoconsent; // 0=off, 1=on
/*24263*/ uint8 unknown19575; // ***Placeholder (6/29/2005)
/*24264*/ uint32 level3; // SoF looks at the level here to determine how many leadership AA you can bank.
/*24268*/ uint32 showhelm; // 0=no, 1=yes
/*24272*/ uint32 RestTimer;
/*24276*/ uint8 unknown19584[1036]; // ***Placeholder (2/13/2007) was[1028]or[940]or[1380] - END of Struct
/*25312*/
};

Given the large number of fields, it is easy for the struct to get bumped off a bit. But, once I can start working on getting in-game, I should be able to iron out most of the kinks. And once I am all of the way in-game, I can probably get the PP looking pretty good if something is off a bit. Thought you might be able to make use of this if you do care to try messing with the PP at some point. If you give it a couple of days and work on spawns and such in the mean-time, I should have the PP looking pretty good by the time you come back to it again (if you ever feel like it).

[Andrew80k]

I played with it some, but it's giving me something weird. It keeps saying the size is wrong. I get the right size in the collect but it keeps saying the sizeof(PlayerProfile_Struct) is 19538. I can't seem to figure that one out just yet.

[Andrew80k]

Spent most of the night looking at packet dumps of spawns. Still not ready yet but I think I've figured out how to get there. Ugh.

[cavedude]

I could have sworn you had access to the collector's SVN, Andrew80k but it looks like not. I've added you so you can make changes as you have them. http://code.google.com/p/eqemutool/

[gaeorn]

I could have sworn you had access to the collector's SVN, Andrew80k but it looks like not. I've added you so you can make changes as you have them. http://code.google.com/p/eqemutool/

Yeah Andrew80k, it'd be awesome if you upload the code so we can get a group effort going on this. And there's nothing in this repo to harm so you don't have to worry about anything you changed in common.

[Andrew80k]

Thanks, CD.

So far there's not really anything to upload. I've spent a LOT of time in the extractor trying figure out the spawn structs. It's not far off but the coords are not right and I've been painfully adjusting the structure one character at a time over the last week or so. The collector seems to be pretty good but no way really to validate it without fixing the extractor first. Doors in both the collector and extractor seem pretty good. I haven't validated the zone points yet. I've noticed that Derision seems to be working a bit on the live stucts and that will definitely help.

[trevius]

They just patched Live again today, and at least a few things (like the PP) have changed for sure. Since the next patch isn't scheduled until June 9th (I believe), I am hoping to get most of the rest of the opcodes updated for Live pretty soon. Derision and I have been working on it a bit here and there. Once we have it pretty filled out, you may even be able to have the tools use the Live.conf and struct files directly from the EQEmuServer/Utils and EQEmuServer/common/patches/ folders. That way, as long as we somewhat keep up with each patch or so, the Live client and the Collector/Extractor tools should both remain functional without duplicate efforts (or at least not much).

As far as the SVN goes, what is on there currently isn't functional at all yet from what I have heard. So, any improvement is better than none. Since you have them both so they can at least compile, it wouldn't hurt to get what you have on the SVN. If you prefer to wait until you have it more functional, that is no problem. But, one of the nice things about using an SVN is that it lets others work together to finish things off so it doesn't all get put on a single person.

BTW, what compiler are you currently using to compile it? Last I heard, it was made for VS Pro or something.

[Andrew80k]

I'm using VS 2008. Not Express. As far as I know it will not compile with express though I haven't tried it since erde got it going. Straight out of the SVN it pretty much compiles and runs for me. I had to tweak the solution just a bit to point to the correct lib locations, but I think that was all.

The collector "works". It will collect packet information and some of it is solid. Some not so much. I do keep getting an unknown stream type in the collector and I haven't figured it out yet. I have put a bunch of debug info into the collector and have started doing more collects with it to help me figure it out. Mostly I'm just learning how it works.

What's working:

Zone info.
Doors
Player data
Some spawn info but some of it isn't right. I think that's more of an adjustment to the extractor. The collector just picks up packets and lays them down with OP_code "tags".

The extractor "works" for some things as well.

What is working:
Doors -- seems good
Zone info -- same here. seems good.
Partial spawn data

What's not working:

AA's -- not seeing this at all
Tasks -- all commented out in the extractor, but the data is there.
Spawn updates This seems to have 2 sized structures on live. One 40 bytes, the other 41. If I adjust the struct one way I get some, if I adjust it the other way I get more. I also consider this critical to being useful.
Objects -- ground spawns etc.

It's slow going but I'm working on it.