Starting Work on SoF Opcodes/Structs

[trevius]

I finally made a little more progress. I am 1 step closer to being able to get in game now lol. I have been trying to figure out the size of the playerprofile struct so that I could at least get the EQ Debug log to show me reaching the next step. After trying many other ways, I finally figured it out by looking at the IDA Assembly code and finding the error I was getting and then checking the HEX for that was causing the error to happen. I checked the HEX and it came out to be 5C18, which is 23576. I checked IDA for Titanium and the same code matched the PlayerProfile size for Titanium (19592). I did another compile with the new structure size and now I am getting this:

[Mon Dec 15 07:06:53 2008]01792:Zone Connect -- 0 -- Received MSG_ZONE_ADDRESS
[Mon Dec 15 07:06:53 2008]01793:Zone addr [192.168.1.102:20253] received...
[Mon Dec 15 07:06:53 2008]01794:ZONING
[Mon Dec 15 07:06:53 2008]01795:Networking: Connection Closed [0] with 0 pending bytes.
[Mon Dec 15 07:06:53 2008]01796:Networking: using port [2629].
[Mon Dec 15 07:06:54 2008]01797:Networking: Connection Established [1]
[Mon Dec 15 07:06:54 2008]01798:Connected to 192.168.1.102:20253...

[Mon Dec 15 07:06:54 2008]01799:Zone Connect -- 2 -- Sending MSG_EQ_ADDPLAYER
[Mon Dec 15 07:06:59 2008]01800:Zone Connect -- 3 -- Received MSG_SEND_PC
[Mon Dec 15 07:06:59 2008]01801:Zone Connect -- 4 -- Received MSG_EQ_ADDPLAYER
[Mon Dec 15 07:06:59 2008]01802:fatal error in main thread Code = c0000005  ADDR=0x004c0aa3

[Mon Dec 15 07:06:59 2008]01803:EAX=0000004b  EBX=0cf3ab0f  ECX=00000088  EDX=f9161180

[Mon Dec 15 07:06:59 2008]01804:ESI=f9161250  EDI=0cf3aa7e  EBP=00000000  ESP=0012a378

[Mon Dec 15 07:06:59 2008]01805:Raw Stack

.....Then a bunch of other error logs

So, there is definitely some issues past that point, but it is a step in the right direction lol. Now to figure out what caused the crash... My first guess is that the problem is related to the items structure, since I know at least the following fields were added at the time of SoF release that weren't in Titanium:
heroic_agi
heroic_cha
heroic_dex
heroic_int
heroic_sta
heroic_str
heroic_wis
I also see these that aren't listed in my Items Table currently, but they may just be newly discovered Uknowns that already exist, but aren't labeled properly:
spelldmg
backstabdmg
clairvoyance
dsmitigation
healamt

I am not sure if that would cause a crash or not, but I guess I will try adding those and see what happens.

Not that it really means much to me, but this seems to be the part where those errors get generated if it isn't equal to whatever check is being done here:

.text:004C0CD3 loc_4C0CD3:                             ; CODE XREF: sub_4C0A50+5Cj
.text:004C0CD3                                         ; sub_4C0A50+78j ...
.text:004C0CD3                 mov     esi, [esp+38h+var_28] ; default
.text:004C0CD7                 mov     eax, [esp+38h+var_18]
.text:004C0CDB                 add     esi, 10h
.text:004C0CDE                 dec     eax
.text:004C0CDF                 mov     [esp+38h+var_28], esi
.text:004C0CE3                 mov     [esp+38h+var_18], eax
.text:004C0CE7                 jnz     loc_4C0AA3
.text:004C0CED                 mov     eax, [esp+38h+var_24]
.text:004C0CF1                 pop     edi
.text:004C0CF2                 pop     esi
.text:004C0CF3                 pop     ebp
.text:004C0CF4                 pop     ebx
.text:004C0CF5                 add     esp, 28h
.text:004C0CF8                 retn

[trevius]

According to the script from 13th floor that dumps items, here:

http://eqitems.13th-floor.org/svn/sc...ump/fields.txt

It looks like the current list of item fields is in this order:

itemclass
name
lore
lorefile
idfile
id
weight
norent
nodrop
size
slots
price
icon
UNK013
UNK014
benefitflag
tradeskills
cr
dr
pr
mr
fr
svcorruption
astr
asta
aagi
adex
acha
aint
awis
hp
mana
endur
ac
classes
races
deity
skillmodvalue
UNK038
skillmodtype
banedmgrace
banedmgbody
banedmgraceamt
banedmgamt
magic
casttime_
reqlevel
reclevel
recskill
bardtype
bardvalue
light
delay
elemdmgtype
elemdmgamt
range
damage
color
itemtype
material
UNK060
UNK061
sellrate
combateffects
shielding
stunresist
strikethrough
extradmgskill
extradmgamt
spellshield
avoidance
accuracy
charmfileid
factionmod1
factionamt1
factionmod2
factionamt2
factionmod3
factionamt3
factionmod4
factionamt4
charmfile
augtype
augrestrict
augdistiller
augslot1type
augslot1visible
augslot1unk2
augslot2type
augslot2visible
augslot2unk2
augslot3type
augslot3visible
augslot3unk2
augslot4type
augslot4visible
augslot4unk2
augslot5type
augslot5visible
augslot5unk2
pointtype
ldontheme
ldonprice
UNK098
ldonsold
bagtype
bagslots
bagsize
bagwr
book
booktype
filename
loregroup
artifactflag
UNK109
favor
guildfavor
fvnodrop
dotshielding
attack
regen
manaregen
enduranceregen
haste
damageshield
UNK120
UNK121
attuneable
nopet
UNK124
potionbelt
potionbeltslots
stacksize
notransfer
UNK129
questitemflag
UNK131
UNK132
clickeffect
clicktype
clicklevel2
clicklevel
maxcharges
casttime
recastdelay
recasttype
clickunk5
clickname
clickunk7
proceffect
proctype
proclevel2
proclevel
procunk1
procunk2
procunk3
procunk4
procrate
procname
procunk7
worneffect
worntype
wornlevel2
wornlevel
wornunk1
wornunk2
wornunk3
wornunk4
wornunk5
wornname
wornunk7
focuseffect
focustype
focuslevel2
focuslevel
focusunk1
focusunk2
focusunk3
focusunk4
focusunk5
focusname
focusunk7
scrolleffect
scrolltype
scrolllevel2
scrolllevel
scrollunk1
scrollunk2
scrollunk3
scrollunk4
scrollunk5
scrollname
scrollunk7
UNK193
purity
dsmitigation
heroic_str
heroic_int
heroic_wis
heroic_agi
heroic_dex
heroic_sta
heroic_cha
healamt
spelldmg
clairvoyance
backstabdmg
evolvinglevel
verified
created

So, I guess all of the extra fields I posted in the previous post were actually new. I know the herioc stuff went in with the SoF release, so I just need to see if I can find in the patch notes when the others after healamt were added. Then, I should just be able to fill in the anniversary (using anniversary files for SoF still) items file either with these new fields, or maybe just set them all to 0 so that the correct number of fields are at least being sent. The heroic stats do sound cool and should add more variety for possible upgraded gear in the emulator. For anyone that doesn't know already, apparently they add more stats that can increase the cap of the particular stat it is adding to, so you can exceed the caps with the stats from them. That shouldn't be hard at all to code if we can get to that point.

[trevius]

I spent most of the day working on this, but this should be the item field list order that is currently used on live according to 13th floor collects:

/* 000 */	//I(ItemClass) Leave this one off on purpose
/* 001 */	S(Name)
/* 002 */	S(Lore)
/* 003 */	S("")//lorefile - Newly Added - Field is Null
/* 003 */	S(IDFile)
/* 004 */	I(ID)
/* 005 */	I(Weight)
/* 006 */	I(NoRent)
/* 007 */	I(NoDrop)
/* 008 */	I(Size)
/* 009 */	I(Slots)
/* 010 */	I(Price)
/* 011 */	I(Icon)
/* 012 */	I(0)
/* 013 */	I(0)
/* 014 */	I(BenefitFlag)
/* 015 */	I(Tradeskills)
/* 016 */	I(CR)
/* 017 */	I(DR)
/* 018 */	I(PR)
/* 019 */	I(MR)
/* 020 */	I(FR)
/* 020 */	I(0)//svcorruption - Newly Added
/* 021 */	I(AStr)
/* 022 */	I(ASta)
/* 023 */	I(AAgi)
/* 024 */	I(ADex)
/* 025 */	I(ACha)
/* 026 */	I(AInt)
/* 027 */	I(AWis)
/* 028 */	I(HP)
/* 029 */	I(Mana)
/* 111 */	I(Endur) //endur - Relocated
/* 030 */	I(AC)
/* 052 */	I(Classes)//classes - Relocated
/* 053 */	I(Races)//races - Relocated
/* 031 */	I(Deity)
/* 032 */	I(SkillModValue)
/* 033 */	I(0)//UNK038 - Default is 0
/* 034 */	I(SkillModType)
/* 035 */	I(BaneDmgRace)
/* 037 */	I(BaneDmgBody)//banedmgbody - Relocated
/* 103 */	I(BaneDmgRaceAmt)//banedmgraceamt - Relocated
/* 036 */	I(BaneDmgAmt)//banedmgamt - Relocated
/* 038 */	I(Magic)
/* 039 */	I(CastTime_)
/* 040 */	I(ReqLevel)
/* 045 */	I(RecLevel)//reclevel - Relocated
/* 046 */	I(RecSkill)//recskill - Relocated
/* 041 */	I(BardType)
/* 042 */	I(BardValue)
/* 043 */	I(Light)
/* 044 */	I(Delay)
/* 047 */	I(ElemDmgType)
/* 048 */	I(ElemDmgAmt)
/* 049 */	I(Range)
/* 050 */	I(Damage)
/* 051 */	I(Color)
/* 056 */	I(ItemType)
/* 057 */	I(Material)
/* 054 */	I(0)//UNK060 - Default is 0
/* 054 */	I(0)//UNK061 - Default is 0
/* 058 */	F(SellRate)
/* 063 */	I(CombatEffects)
/* 064 */	I(Shielding)
/* 065 */	I(StunResist)
/* 066 */	I(StrikeThrough)
/* 067 */	I(ExtraDmgSkill)
/* 068 */	I(ExtraDmgAmt)
/* 069 */	I(SpellShield)
/* 070 */	I(Avoidance)
/* 071 */	I(Accuracy)
/* 072 */	I(CharmFileID)
/* 073 */	I(FactionMod1)//Swapped these so Faction Amt comes after each Faction Mod
/* 077 */	I(FactionAmt1)//Swapped these so Faction Amt comes after each Faction Mod
/* 074 */	I(FactionMod2)//Swapped these so Faction Amt comes after each Faction Mod
/* 078 */	I(FactionAmt2)//Swapped these so Faction Amt comes after each Faction Mod
/* 075 */	I(FactionMod3)//Swapped these so Faction Amt comes after each Faction Mod
/* 079 */	I(FactionAmt3)//Swapped these so Faction Amt comes after each Faction Mod
/* 076 */	I(FactionMod4)//Swapped these so Faction Amt comes after each Faction Mod
/* 080 */	I(FactionAmt4)//Swapped these so Faction Amt comes after each Faction Mod
/* 081 */	S(CharmFile)
/* 082 */	I(AugType)
/* 104 */	I(AugRestrict)//augrestrict - Relocated
/* 122 */	I(AugDistiller)//augdistiller - Relocated
/* 083 */	I(AugSlotType[0])
/* 084 */	I(AugSlotUnk[0])//augslot1visible - Default 1
/* 084 */	I(0)//augslot1unk2 - Newly Added  - Default 0
/* 085 */	I(AugSlotType[1])
/* 086 */	I(AugSlotUnk[1])
/* 084 */	I(0)//augslot2unk2 - Newly Added
/* 087 */	I(AugSlotType[2])
/* 088 */	I(AugSlotUnk[2])
/* 084 */	I(0)//augslot3unk2 - Newly Added
/* 089 */	I(AugSlotType[3])
/* 090 */	I(AugSlotUnk[3])
/* 084 */	I(0)//augslot4unk2 - Newly Added
/* 091 */	I(AugSlotType[4])
/* 092 */	I(AugSlotUnk[4])
/* 084 */	I(0)//augslot5unk2 - Newly Added
/* 128 */	I(PointType)//pointtype - Relocated
/* 093 */	I(LDoNTheme)
/* 094 */	I(LDoNPrice)
/* 084 */	I(70)//UNK098 - Newly Added - Default 70, but some are set to 0
/* 095 */	I(LDoNSold)
/* 096 */	I(BagType)
/* 097 */	I(BagSlots)
/* 098 */	I(BagSize)
/* 099 */	I(BagWR)
/* 100 */	I(Book)
/* 101 */	I(BookType)
/* 102 */	S(Filename)
/* 105 */	I(LoreGroup)
/* 107 */	I(ArtifactFlag)
/* 106 */	I(PendingLoreFlag)//UNK109 - Default 0, but a few are 1
/* 109 */	I(Favor)
/* 121 */	I(GuildFavor)//guildfavor - Relocated
/* 110 */	I(FVNoDrop)
/* 112 */	I(DotShielding)
/* 113 */	I(Attack)
/* 114 */	I(Regen)
/* 115 */	I(ManaRegen)
/* 116 */	I(EnduranceRegen)
/* 117 */	I(Haste)
/* 118 */	I(DamageShield)
/* 123 */	I(-1) //UNK120 - Default is -1
/* 124 */	I(0) //UNK121 - Default is 0
/* 125 */	I(Attuneable)
/* 126 */	I(NoPet)
/* 127 */	I(0) //UNK124 - Default 0, but a few are 1
/* 129 */	I(PotionBelt)
/* 130 */	I(0) //potionbeltslots - Default 0, but a few are 1
/* 131 */	I(StackSize)
/* 132 */	I(NoTransfer)
/* 133 */	I(Stackable)//UNK129 - Default is 0, but some are much higher
/* 133 */	I(0)//questitemflag - Default is 0 (off), flag on = 1
/* 133 */	I(0)//UNK131 - Default is 0, but there is an item set to 1
/* 133 */	I(0)//UNK132 - Default is 0
/* 134 */	I(Click.Effect)
/* 135 */	I(Click.Type)
/* 136 */	I(Click.Level2)
/* 137 */	I(Click.Level)
/* 055 */	I(MaxCharges)//maxcharges - Relocated
/* 060 */	I(CastTime_)//casttime - Relocated - Note Duplicate Entries for CastTime_ and none for CastTime
/* 119 */	I(RecastDelay)//recastdelay - Relocated
/* 120 */	I(RecastType)//recasttype - Relocated
/* 133 */	I(0)//clickunk5 - Newly Added - Default is 0
/* 138 */	S("")//clickname - Newly Added - Default is Null
/* 133 */	I(-1)//clickunk7 - Newly Added - Default is -1, but some set to 0 and some much higher
/* 139 */	I(Proc.Effect)
/* 140 */	I(Proc.Type)
/* 141 */	I(Proc.Level2)
/* 142 */	I(Proc.Level)
/* 133 */	I(0)//procunk1 - Newly Added - Default is 0, but some set to -1 and 1
/* 133 */	I(0)//procunk2 - Newly Added - Default is 0
/* 133 */	I(0)//procunk3 - Newly Added - Default is 0
/* 133 */	I(0)//procunk4 - Newly Added - Default is 0
/* 062 */	I(ProcRate)procrate - Relocated
/* 138 */	S("")//procname - Newly Added - Default is Null
/* 133 */	I(-1)//procunk7 - Newly Added - Default is -1, but some set to 0
/* 144 */	I(Worn.Effect)
/* 145 */	I(Worn.Type)
/* 146 */	I(Worn.Level2)
/* 147 */	I(Worn.Level)
/* 133 */	I(0)//wornunk1 - Newly Added - Default is 0
/* 133 */	I(0)//wornunk2 - Newly Added - Default is 0
/* 133 */	I(0)//wornunk3 - Newly Added - Default is 0
/* 133 */	I(0)//wornunk4 - Newly Added - Default is 0
/* 133 */	I(0)//wornunk5 - Newly Added - Default is 0
/* 138 */	S("")//wornname - Newly Added - Default is Null
/* 133 */	I(-1)//wornunk7 - Newly Added - Default is -1, but some set to 0
/* 149 */	I(Focus.Effect)
/* 150 */	I(Focus.Type)
/* 151 */	I(Focus.Level2)
/* 152 */	I(Focus.Level)
/* 133 */	I(0)//focusunk1 - Newly Added - Default is 0
/* 133 */	I(0)//focusunk2 - Newly Added - Default is 0
/* 133 */	I(0)//focusunk3 - Newly Added - Default is 0
/* 133 */	I(0)//focusunk4 - Newly Added - Default is 0
/* 133 */	I(0)//focusunk5 - Newly Added - Default is 0
/* 138 */	S("")//focusname - Newly Added - Default is Null
/* 133 */	I(-1)//focusunk7 - Newly Added - Default is -1, but some set to 0
/* 154 */	I(Scroll.Effect)
/* 155 */	I(Scroll.Type)
/* 156 */	I(Scroll.Level2)
/* 157 */	I(Scroll.Level)scrollunk1
/* 133 */	I(0)//scrollunk2 - Newly Added - Default is 0
/* 133 */	I(0)//scrollunk3 - Newly Added - Default is 0
/* 133 */	I(0)//scrollunk4 - Newly Added - Default is 0
/* 133 */	I(0)//scrollunk5 - Newly Added - Default is 0
/* 138 */	S("")//scrollname - Newly Added - Default is Null
/* 133 */	I(-1)//scrollunk7 - Newly Added - Default is -1, but some set to 0
/* 158 */	I(0)//UNK193 - Default is 0
/* 133 */	I(0)//purity - Newly Added - Default is 0, but some go up to 75
/* 133 */	I(0)//dsmitigation - Newly Added - Default is 0, but some are up to 2
/* 133 */	I(0)//heroic_str - Newly Added - Default is 0
/* 133 */	I(0)//heroic_int - Newly Added - Default is 0
/* 133 */	I(0)//heroic_wis - Newly Added - Default is 0
/* 133 */	I(0)//heroic_agi - Newly Added - Default is 0
/* 133 */	I(0)//heroic_dex - Newly Added - Default is 0
/* 133 */	I(0)//heroic_sta - Newly Added - Default is 0
/* 133 */	I(0)//heroic_cha - Newly Added - Default is 0
/* 133 */	I(0)//healamt - Newly Added - Default is 0, but some are up to 9
/* 133 */	I(0)//spelldmg - Newly Added - Default is 0, but some are up to 9
/* 133 */	I(0)//clairvoyance - Newly Added - Default is 0, but some are up to 10
/* 133 */	I(0)//backstabdmg - Newly Added - Default is 0, but some are up to 65
/* 133 */	I(0)//evolvinglevel - Newly Added - Default is 0, but some are up to 7

//This doesn't appear to be used /* 102 */	S(verified)//verified
//This doesn't appear to be used /* 102 */	S(serialized)//created
//Unsure where this goes right now (or if it is even used) /* 108 */	I(SummonedFlag)

#undef I
#undef C
#undef S
#undef F

I will try this as soon as I get home tonight. I am guessing that the following fields were added after SoF, so I will comment them out for now and try adding them 1 at a time until it works (hopefully):

/* 133 */	I(0)//healamt - Newly Added - Default is 0, but some are up to 9
/* 133 */	I(0)//spelldmg - Newly Added - Default is 0, but some are up to 9
/* 133 */	I(0)//clairvoyance - Newly Added - Default is 0, but some are up to 10
/* 133 */	I(0)//backstabdmg - Newly Added - Default is 0, but some are up to 65
/* 133 */	I(0)//evolvinglevel - Newly Added - Default is 0, but some are up to 7

Note that the item field numbers are not correct, but once I have the list working, I will renumber it all manually.

Also, the only thing I am not quite sure about here is the use of quotes on ints and nulls. Does anyone know if I(0) is the same as I("0"), and if S("") is the same as NULL, or if there is a better way to put NULL there? Maybe something like S(NULL)?

[MNWatchdog]

Have you checked Macroquest sources from around that time? I would assume most of the structs are in there.

[trevius]

I already looked into it a little, but Macroquest reads directly from memory and doesn't do any kind of packet sniffing as far as I can tell. So, it uses offsets and such, which might possibly be useful for figuring out something, but I don't really know what to do with them personally.

[MNWatchdog]

Just thinking that by comparing MQ Titanium/Emus structs to MQs SoF structs would make it far easier to figure out what new fields need to be added to Emus structs that need to be send to the SoF client.

Im thinking that say a items struct is sent to the client in the same formate its readable from memory. I dont know if this is true, but if it is, looking at MQs SoFs structs should help a lot.

[Rocker8956]

Wow, way above my skill level so all I can do is cheer you on.

Just one thought though, perhaps the correct fields could be found by looking at stats on items that were released with SoF.

I think clairvoyance and healamt were on those items but they could have been added later.

[trevius]

Like I said, MQ reads from memory and is written in a completely different way than our packet structures. They use offsets that line up directly with the assembly code from the eqgame.exe, and so their format is completely different. Possibly if I was very familiar with the MQ source, I might have an idea of how it could be useful. Unfortunately I might as well be trying to read in German (which I don't know how to do), because that is about how different the code is from the emu source.

I figured out last night that all of the fields I had listed in my last post were actually in with the SoF release. The only one I am still not 100% certain about is the evolving items part. According to 13th floor, I saw it mentioned that evolving items need their own separate table and have a separate opcode (I think) if the item is able to evolve. Hopefully setting that field to 0 will just disable evolving so we don't need to write anything for it until we are at a point where we can do it at leisure. As far as the item field list for SoF, I think that I now have it 100% (or very close to it) complete. This is good, because it is 1 more step closer to being done, but the bad part is that it didn't fix my current issue.

I looked into it further last night and checked the debug where I am crashing in SoF vs a debug of a successful Titanium connection. I then compared a few things in the Titanium assembly code to the SoF assembly code. I am pretty sure that I have narrowed the current issue down to the Spawn_Structure. Unfortunately, I think that structure is by far the worst and hardest part left of getting SoF to work with the emulator. Out of all structures, the spawn struct gets completely moved around with almost every patch. I was able to find the packet size of 385 (HEX is 181) for the Titanium spawn struct by using the hex calculator here http://www.squarebox.co.uk/hcalc.html (very useful when messing with the assembly code), and then searching for 0x0181 in the assembly code. I found the same section of code in the SoF assembly code, but I was unable to figure out what the struct size was. Since this technique worked perfectly for finding the struct size for the player profile, I think that means that the new spawn struct for SoF is now a variable length struct. I checked the current SEQ source code and it lists it as variable length, even though the SEQ version from when SoF came out shows it as a fixed size.

I don't know anything about variable structure sizes or what to do with them. The spawn struct also seems to use unions, packet padding, signed and unsigned ints, all of which I know absolutely nothing about... All of the other structures are pretty straight forward and I think most of them should already be ready to start working once I get the spawn struct correct. This is probably going to be the biggest hurdle to getting SoF working.

I am still going to do some further testing on Titanium to see if I can duplicate the exact crash I am having on SoF, which should help narrow down the exact cause of the problem. That technique has already helped me multiple times in figuring out what is causing a problem.

I figured this stuff was over my skill level too, but so far, I have learned alot just reading the structures and forums here and SEQ forums. I have been able to do alot of things that I didn't expect to be able to do. So, even though it might sound above your skill level doesn't mean that you wouldn't be able to figure it out if you put some thought into it