Starting Work on SoF Opcodes/Structs

[KLS]

That's not really what I was talking about, that all really falls under patch files. For example when we wanted to fix inter-zoning we had to actually modify how you zone as 6.2 sent a different series of packets overall than titanium and so required changes to both the patch files and the zone server. When I talk about "core changes" that's things that aren't contained to a certain version and could potentially affect other versions. =p

[Yeormom]

Just do us a flavor and make SoF commits contain only SoF related changes!

[trevius]

As if I had time to make any other changes while I am working on SoF. This thing is like a full time job that I don't get paid for lol.

I will probably be doing an SVN commit for SoF tonight. There is alot more progress since the initial release. Spawns now show up (missing names though), you can actually move around in the zone instead of being stuck in place, doors spawn and ground spawns spawn (but still need work), and a few new opcodes are verified.

Items are a big one that needs work, but those shouldn't be too bad once I know how to do the serialization like a structure so it sends it the way it needs to. I think it is almost to the point where other people might actually be able to start helping out on it soon. I am probably going to try to write a beginner's guide in the wiki for packet structures and maybe some opcode stuff.

[Yeormom]

Originally Posted by trevius

This thing is like a full time job that I don't get paid for lol.

You just described the entire EQemu project!

I am probably going to try to write a beginner's guide in the wiki for packet structures and maybe some opcode stuff.

I do so enjoy documentation~

[trevius]

Back to working on the item serialization again, I broke down items into as close to an exact structure as I can figure out. This is for EQLive, but no new fields have changed from SoF to Live as far as I know, so it should also apply to SoF. This structure may not be 100%, but I think it is at least 95% accurate. This is based off of what I could find from 13th floor and packet collects from EQLive.

struct ItemSerialization_Struct {

/*0000*/	uint32 stacksize;
/*0004*/	uint32 unknown004; // 00 00 00 00
/*0008*/	uint32 price;
/*0012*/	uint32 slot;
/*0016*/	uint32 merchcount;
/*0020*/	uint32 unknown020; // 00 00 00 00
/*0024*/	sint32 serialnumber;
/*0028*/	uint32 instnodrop;
/*0032*/	uint32 typepotion;
/*0036*/	uint32 unknown036; // 00 00 00 00
/*0040*/	uint32 unknown040; // 00 00 00 00
/*0044*/	uint32 unknown044; // 00 00 00 00
/*0048*/	uint32 unknown048; // 00 00 00 00
/*0052*/	uint32 unknown052; // 00 00 00 00
/*0056*/	sint32 unknown056; // 00 00 00 01
/*0060*/	Item_Struct item;
/*0000*/	uint8	unknown00x1;		// 00 Break for separating augments within items?
/*0000*/	Item_Struct augments;		// Bag Slots/Augments within an item
/*0000*/	uint8	unknown00x2;		// 00 Break for separating items?	

}


/*
** Child struct of Item_Struct:
**	Effect data: Click, Proc, Focus, Worn, Scroll
**
*/
struct ItemClickEffect_Struct {
/*0000*/	sint32	Effect;
/*0004*/	uint32	Type;
/*0008*/	uint8	Level2;
/*0009*/	uint8	Level;
/*0010*/	sint32	MaxCharges;		// Max Charges of this effect
			union {
/*0014*/		uint16  Fulfilment;	// Food fulfilment (How long it lasts)
/*0016*/		sint16  CastTime;	// Cast Time for clicky effects, in milliseconds
			};
/*0018*/	uint32	RecastDelay;		// Delay on Recast in seconds
/*0022*/	uint32	RecastType;		// Recast Type from -1 to 18
/*0026*/	uint32	unknown0005;
/*0030*/	char	effectname[1];		//click name - Default is NULL
/*0031*/	uint32	unknown0007;
/*0035*/
};

struct ItemProcEffect_Struct {
/*0000*/	sint32	Effect;
/*0004*/	uint32	Type;
/*0008*/	uint8	Level2;
/*0009*/	uint8	Level;
/*0010*/	uint32	unknown0001;
/*0014*/	uint32	unknown0002;
/*0018*/	uint32	unknown0003;
/*0022*/	uint32	unknown0004;
/*0026*/	sint32	ProcRate;		// Proc Rate - 100 is default
/*0030*/	char	effectname[1];		//proc name - Default is NULL
/*0031*/	uint32	unknown0007;
/*0035*/
};

struct ItemWornFocusScrollEffect_Struct {
/*0000*/	sint32	Effect;
/*0004*/	uint32	Type;
/*0008*/	uint8	Level2;
/*0009*/	uint8	Level;
/*0010*/	uint32	unknown0001;
/*0014*/	uint32	unknown0002;
/*0018*/	uint32	unknown0003;
/*0022*/	uint32	unknown0004;
/*0026*/	uint32	unknown0005;
/*0030*/	char	effectname[1];		//focus, worn, or scroll name - Default is NULL
/*0031*/	uint32	unknown0007;
/*0035*/
};

struct ItemAugmentSlot_Struct {
/*0000*/	uint8	AugSlotType;		// LDoN: Augment Slot 1-5 Type
/*0001*/	sint32	AugSlotVisible;		// LDoN: Augment Slot 1-5 Unknown
/*0005*/	uint8	AugSlotUnk2;		// LDoN: Augment Slot 1-5 Unknown
/*0006*/
};	

#define MAX_AUGMENT_SLOTS 5

struct Item_Struct {
	// Non packet based field
	// uint8	MinStatus; 
	// bool	LoreFlag;		// This will be true if LoreGroup is non-zero
	// bool	SummonedFlag;		// Unused in SoF

	// Packet based fields
	// uint8	ItemClass;		// Item Type: 0=common, 1=container, 2=book
	char	Name[1];		// Variable Length String for Name
	uint8	unknown0001;		// 00 Break between Strings
	char	Lore[1];		// Variable Length String for Item Lore Text
	uint8	unknown0002;		// 00 Break between Strings
	char	IDFile[1];		// Variable Length String for Visible Model Number - IT63
	uint8	unknown0003;		// 00 Break between Strings
	uint32	ID;			// Unique ID (also PK for DB)
	uint8	Weight;			// Item weight * 10
	uint8	NoRent;			// No Rent: 0=norent, 255=not norent
	uint8	NoDrop;			// No Drop: 0=nodrop, 255=not nodrop
	uint8	Size;			// Size: 0=tiny, 1=small, 2=medium, 3=large, 4=giant
	uint8	Slots;			// Bitfield for which slots this item can be used in
	char	Price[8];		// Item cost (?) maybe int64?
	uint32	Icon;			// Icon Number
	uint8	unknown0013;		// UNK013 - 01
	uint32	unknown0014;		// UNK014 - One of these are probably int32	
	bool	BenefitFlag;		// Does this have a benefit flag?
	bool	Tradeskills;		// Is this a tradeskill item?
	sint8	CR;			// Save vs Cold
	sint8	DR;			// Save vs Disease
	sint8	PR;			// Save vs Poison
	sint8	MR;			// Save vs Magic
	sint8	FR;			// Save vs Fire
	sint8	Corruption;		// New Save vs Corruption ****
	sint8	AStr;			// Strength
	sint8	ASta;			// Stamina
	sint8	AAgi;			// Agility
	sint8	ADex;			// Dexterity
	sint8	ACha;			// Charisma
	sint8	AInt;			// Intelligence
	sint8	AWis;			// Wisdom
	sint32	HP;			// HP
	sint32	Mana;			// Mana	
	uint32	Endur;			// Endurance
	sint32	AC;			// AC	
	uint32	unknown0015;		// 
	uint32	unknown0016;		// 
	uint32	unknown0017;		// 
	uint32	Classes;		// Bitfield of classes that can equip item (1 << class#)
	uint32	Races;			// Bitfield of races that can equip item (1 << race#)
	uint32	Deity;			// Bitmask of Deities that can equip this item
	sint32	SkillModValue;		// % Mod to skill specified in SkillModType
	uint32	unknown0038;		// UNK038 - Default is 0
	uint32	SkillModType;		// Type of skill for SkillModValue to apply to
	uint32	BaneDmgRace;		// Bane Damage Race
	uint32	BaneDmgBody;		// Bane Damage Body
	uint32	BaneDmgRaceAmt;		// Bane Damage Race Amount
	sint32	BaneDmgAmt;		// Bane Damage Body Amount
	bool	Magic;			// True=Magic Item, False=not
	sint32	CastTime_;		// Cast Time in Milliseconds
	uint8	ReqLevel;		// Required Level to use item
	uint8	RecLevel;		// Recommended level to use item
	uint8	RecSkill;		// Recommended skill to use item (refers to primary skill of item)
	uint32	BardType;		// Bard Skill Type
	sint32	BardValue;		// Bard Skill Amount
	sint8	Light;			// Light
	uint8	Delay;			// Delay * 10
	uint32	ElemDmgType;		// Elemental Damage Type (1=magic, 2=fire)
	uint32	ElemDmgAmt;		// Elemental Damage
	uint32	Range;			// Range of item
	uint32	Damage;			// Delay between item usage (in 0.1 sec increments)
	uint32	Color;			// RR GG BB 00 <-- as it appears in pc
	uint32	ItemType;		// Item Type/Skill (itemClass* from above)
	uint32	Material;		// Item material type
	uint32	unknown0060;		// UNK060 - Default is 0
	uint8	unknown0061;		// UNK061 - Default is 0
	float	SellRate;		// Sell rate
	sint32	CombatEffects;		// PoP: Combat Effects +
	sint32	Shielding;		// PoP: Shielding %
	sint32	StunResist;		// PoP: Stun Resist %
	sint32	StrikeThrough;		// PoP: Strike Through %
	uint32	ExtraDmgSkill;		// PoP: Extra Damage Skill
	uint32	ExtraDmgAmt;		// PoP: Extra Damage Amount
	sint32	SpellShield;		// PoP: Spell Shield %
	sint32	Avoidance;		// PoP: Avoidance +
	sint32	Accuracy;		// PoP: Accuracy +
	uint32	CharmFileID;		// ID of the Charm File
	sint32	FactionMod1;		// Faction Mod 1
	sint32	FactionAmt1;		// Faction Amt 1
	sint32	FactionMod2;		// Faction Mod 2
	sint32	FactionAmt2;		// Faction Amt 2
	sint32	FactionMod3;		// Faction Mod 3
	sint32	FactionAmt3;		// Faction Amt 3
	sint32	FactionMod4;		// Faction Mod 4
	sint32	FactionAmt4;		// Faction Amt 4
	char	CharmFile[1];		// Name of the Charm File
	uint8	unknown0062;		// 00 Break between Strings
	uint32	AugType;
	uint16	AugRestrict;
	uint16	AugDistiller;
	ItemAugmentSlot_Struct augslots[MAX_AUGMENT_SLOTS]; //Augment Slots
	uint32	PointType;
	uint32	LDoNTheme;
	uint32	LDoNPrice;
	uint32	unknown0098;		// UNK098
	uint32	LDoNSold;
	uint8	BagType;		// 0:Small Bag, 1:Large Bag, 2:Quiver, 3:Belt Pouch ... there are 50 types
	uint8	BagSlots;		// Number of slots: can only be 2, 4, 6, 8, or 10
	uint8	BagSize;		// 0:TINY, 1:SMALL, 2:MEDIUM, 3:LARGE, 4:GIANT 
	uint8	BagWR;			// 0->100
	uint8	Book;			// 0=Not bool, 1=Book
	uint16	BookType;
	uint32	Filename;		// Filename for book data
	uint8	LoreGroup;		// LoreGroup seems to have replaced LoreFlag
	bool	ArtifactFlag;		// Flag as an Artifact
	bool	PendingLoreFlag;	// ???  Maybe just unknown?
	uint32	Favor;			// Individual favor
	uint32	GuildFavor;		// Guild favor
	uint8   FVNoDrop;		// Firiona Vie nodrop flag
	uint8	DotShielding;		// May need to be moved to the PoP stuff ****
	uint32	Attack;
	uint32	Regen;
	uint32	ManaRegen;
	uint32	EnduranceRegen;
	uint8	Haste;
	uint8	DamageShield;
	uint32	unknown0120;		// UNK120 - Default is -1 	
	uint8	unknown0121;		// UNK121 - Default is 0 		
	bool	Attuneable;
	bool	NoPet;
	uint16	unknown0124;		// UNK124 - Default 0
	bool	PotionBelt;
	uint32	PotionBeltSlots;
	uint8	StackSize;
	bool	NoTransfer;		//Swapped Positions with StackSize ****
	// bool	Stackable;		//Not in 13th Floor ???
	bool	QuestItemFlag;
	uint8	unknown0131;		// UNK131 - Default is 0	
	uint8	unknown0132[19];		// UNK132 - Default is 0	
	ItemClickEffect_Struct Click;
	ItemProcEffect_Struct Proc;
	ItemWornFocusScrollEffect_Struct Worn, Focus, Scroll;
	uint32	unknown0193;//UNK193 - Default is 0
	uint8	purity;			//purity - Newly Added - Default is 0, but some go up to 75
	uint8	dsmitigation;		//dsmitigation - Newly Added - Default is 0, but some are up to 2
	uint8	herostr;		//heroic_str - Newly Added - Default is 0
	uint8	heroint;		//heroic_int - Newly Added - Default is 0
	uint8	herowis;		//heroic_wis - Newly Added - Default is 0
	uint8	heroagi;		//heroic_agi - Newly Added - Default is 0
	uint8	herodex;		//heroic_dex - Newly Added - Default is 0
	uint8	herosta;		//heroic_sta - Newly Added - Default is 0
	uint8	herocha;		//heroic_cha - Newly Added - Default is 0
	uint32	heropoison;		//HeroicSvPoison - Newly Added - Default is 0
	uint32	heromagic;		//HeroicSvMagic - Newly Added - Default is 0
	uint32	herofire;		//HeroicSvFire - Newly Added - Default is 0
	uint32	herodisease;		//HeroicSvDisease - Newly Added - Default is 0
	uint32	herocold;		//HeroicSvCold - Newly Added - Default is 0
	uint32	herocorruption;		//HeroicSvCorruption - Newly Added - Default is 0
	uint32	healamt;		//healamt - Newly Added - Default is 0, but some are up to 9
	uint32	spelldmg;		//spelldmg - Newly Added - Default is 0, but some are up to 9
	uint32	clairvoyance;		//clairvoyance - Newly Added - Default is 0, but some are up to 10
	uint32	backstabdmg;		//backstabdmg - Newly Added - Default is 0, but some are up to 65
	uint32	evolvinglevel;		//evolvinglevel - Newly Added - Default is 0, but some are up to 7
	uint32	MaxPower;		//MaxPower - Newly Added
	uint32	Power;			//Power - Newly Added

};

Since items are handled in a different way than any other struct, I am not exactly sure how to implement this properly. Currently, the emulator sends items as a long string. For SoF, they need to be sent like any other normal structure. Right now, I can't figure out how to add these structs in without having to mess with the item serialization for other clients. I will keep working on it, but it is a bit above my skill level right now. For starters, I need to figure out how to change the serialization from using MakeAnyLengthString and instead for it to send the actual data as it is written in the struct.

[trevius]

I was able to get Targeting working finally and also able to get names to show up now too. The only other major thing to get working from the spawn structure is size. Most of the rest isn't really required and could be done later at any point. These are a couple of major things that were holding me back from making much more progress, so having those working will help alot to getting the game playable.

The actual process I used to find those settings is a bit of a pain, but seems to actually work fairly well. Basically, I took all of the unknowns in the spawn structure and have been setting each one to send 1s for all fields in them. So, by doing this for each section of unknowns 1 at a time, I can compile the changes and then start the server and log in to observe if there were any differences and what they are.

Here is the actual spawn encoding that I am using to test:

ENCODE(OP_NewSpawn) {  ENCODE_FORWARD(OP_ZoneSpawns); }
ENCODE(OP_ZoneEntry){  ENCODE_FORWARD(OP_ZoneSpawns); }
ENCODE(OP_ZoneSpawns) {
	//consume the packet
	EQApplicationPacket *in = *p;
	*p = NULL;
	
	//store away the emu struct
	unsigned char *__emu_buffer = in->pBuffer;
	Spawn_Struct *emu = (Spawn_Struct *) __emu_buffer;
	
	//determine and verify length
	int entrycount = in->size / sizeof(Spawn_Struct);
	if(entrycount == 0 || (in->size % sizeof(Spawn_Struct)) != 0) {
		_log(NET__STRUCTS, "Wrong size on outbound %s: Got %d, expected multiple of %d", opcodes->EmuToName(in->GetOpcode()), in->size, sizeof(Spawn_Struct));
		delete in;
		return;
	}
	
	//make the EQ struct.
	in->size = sizeof(structs::Spawn_Struct)*entrycount;
	in->pBuffer = new unsigned char[in->size];
	structs::Spawn_Struct *eq = (structs::Spawn_Struct *) in->pBuffer;
	
	//zero out the packet. We could avoid this memset by setting all fields (including unknowns)
	//in the loop.
	memset(in->pBuffer, 0, in->size);
	
	//do the transform...
	int r;
	int k;
	for(r = 0; r < entrycount; r++, eq++, emu++) {

		eq->deity = emu->deity;
		eq->gender = emu->gender;
		for(k = 0; k < 9; k++) {
			eq->equipment[k].equip0 = emu->equipment[k];
			eq->equipment[k].equip1 = 0;
			eq->equipment[k].itemId = 0;
			eq->colors[k].color = emu->colors[k].color;
		}
		eq->guildID = emu->guildID;
		eq->class_ = emu->class_;
		eq->gm = emu->gm;
		eq->runspeed = emu->runspeed;
		eq->light = emu->light;
		eq->level = emu->level;
		eq->race = emu->race;
		strcpy(eq->suffix, emu->suffix);
		eq->bodytype = emu->bodytype;
		eq->curHp = emu->curHp;
		strcpy(eq->lastName, emu->lastName);
		strcpy(eq->title, emu->title);
		eq->NPC = emu->NPC;
		eq->x = emu->x;
		eq->deltaX = emu->deltaX;
		eq->deltaY = emu->deltaY;
		eq->z = emu->z;
		eq->deltaHeading = emu->deltaHeading;
		eq->y = emu->y;
		eq->deltaZ = emu->deltaZ;
		eq->animation = emu->animation;
		eq->heading = emu->heading;
		eq->spawnId = emu->spawnId;
		strcpy(eq->name, emu->name);
		eq->petOwnerId = emu->petOwnerId;
		eq->anon = emu->anon;
		eq->walkspeed = emu->walkspeed;

		eq->targetable = 1; //New Field - Force NPCs to Targetable for now
		eq->showname = 1; //New Field - Toggles Name Display on or off - 0 = off, 1 = on
		eq->linkdead = 0; //New Field - Toggles LD on or off after name - 0 = off, 1 = on

		//Hack Test for finding more fields in the Struct:
		memset(eq->unknown0001, 0x01, sizeof(eq->unknown0001));
		memset(eq->unknown0005, 0x01, sizeof(eq->unknown0005)); // 15
		//memset(eq->unknown0008, 0x01, sizeof(eq->unknown0008)); // 13
		//memset(eq->unknown0048, 0x01, sizeof(eq->unknown0048)); // 12 - No Visible Change?
		//eq->unknown0820 = 1;	//Stand State - Stand/Sit/Crouch
		//eq->unknown0059 = 1; // 1 Turned off on 6 - west bug?
		//memset(eq->unknown0074, 0x01, sizeof(eq->unknown0074)); // 16 - No Visible Change?
		//memset(eq->unknown0077, 0x01, sizeof(eq->unknown0077));
		//memset(eq->unknown0079, 0x01, sizeof(eq->unknown0079));
		//memset(eq->unknown0106, 0x01, sizeof(eq->unknown0106)); // 11 - No Visible Change?
		//memset(eq->unknown0107, 0x01, sizeof(eq->unknown0107));
		//memset(eq->unknown0110, 0x01, sizeof(eq->unknown0110));
		//eq->unknown0111 = 1; // 1 - No Visible Change?
		//eq->unknown0613 = 0; //was bodytype
		//memset(eq->unknown0154, 0x01, sizeof(eq->unknown0154)); // 2 - freeze in place?
		//memset(eq->unknown0263, 0x01, sizeof(eq->unknown0263)); // 1 - no player character visible?
		//memset(eq->unknown0281, 0x01, sizeof(eq->unknown0281)); // 2 3
		//eq->unknown0307 = 1; // 9 10 11 - No Visible Change?
		//memset(eq->unknown0308, 0x01, sizeof(eq->unknown0308)); //
		//memset(eq->unknown0309, 0x01, sizeof(eq->unknown0309)); // 8 - No Visible Change?
		//memset(eq->unknown442, 0x01, sizeof(eq->unknown442)); // 6 - crash?
		//eq->unknown443 = 1; // 1 turned off on 9 - No Visible Change?
		//memset(eq->unknown0760, 0x01, sizeof(eq->unknown0760)); // 4 avatar height?
		//eq->unknown0779 = 0; // 1 - int32 avatar height?
		//memset(eq->unknown0496, 0x01, sizeof(eq->unknown0496)); // 4 5

		// 1 all set to one shows you at floor level
		// 2 seems to cause invis and freeze in place and west bug
		// 3 causes no player character, but spawns work and has west bug
		// 4 looks just like 2 accept you can move and are at floor level
		// 5 Looks like 3
		// 6 Caused a crash
		// 7 west bug gone 
		// 8 Looks normal accept no player character
		// 9 Targetable mobs!!! Still no player character...
		// 10 Narrowing down bodytype
		// 11 Non-targetable again and narrowing down bodytype
		// 12 Bodytype location identified!!!  Targeting works...  Nother other visible changes
		// 13 Can target self but not spawns.  West bug again
		// 14 Names showing and also LD showing!!!  Still West Bug though...
		// 15 Show Names field now identified.  

	}
	
	
	//kill off the emu structure and send the eq packet.
	delete[] __emu_buffer;
	
	_log(NET__ERROR, "Sending zone spawns");
	_hex(NET__ERROR, in->pBuffer, in->size);
	
	dest->FastQueuePacket(&in, ack_req);
}

The section noted in blue are the new fields I found and am forcing them to send all spawns with a certain setting in that field. The section noted in green are the actual settings to test the unknown fields. I just uncomment the line of the unknown section I want to test. There are some notes next to some of them as to what I noticed when testing them. There are also numbers next to some as well, which correspond to the section in orange below that. Basically, I tried picking the unknowns that were smallest first. I was doing more than 1 at a time to reduce the number of times I would have to recompile. The numbers tell which ones I turned on for each test number I ran. So, the ones that have 1 next to them were the first ones I tried and the notes in orange below are what I saw as a result from that first test. The notes could have been done better, but at least it was something to reference to help keep track of what was being done and being seen.

The reason I am doing a little write-up on how I did this is so others may be able to use the same technique in the future to help refine the structures even further. Other than being able to figure out the structure from reading the IDA output, which is well above my skill level, this guess and test technique is the only other way I could think of to figure out these needed structure fields.

The same technique should be able to work fine for figuring out the rest of the fields. I think it should be fairly easy to figure out stuff like beard and beard color, face, etc with this technique. So, almost anyone could help refine this list of field positions in the structure.

I will most likely be doing a write-up wiki page on what I know about working with packet structures. It really isn't all too complex, but it is important to understand a few key things that took me a while to figure out on my own.

I will probably do an SVN update with the new targeting and name stuff tonight, or very soon. It is finally getting to the point where it is almost playable. I think getting items working is probably one of the biggest things left to do. I have the item structure worked out, but may need help in finding the best way to implement it.

I was finally able to try attacking a mob for the first time in SoF, and of course it caused EQ to crash :P I am pretty sure the crash was due to the fact that items hadn't been loaded though.

For a while, I was a bit concerned that we would never be able to get SoF functioning well enough to actually use as an upgrade to Titanium. Now, I am fairly confident that all possible issues can be resolved and that it will almost certainly be completed and at least as functional as Titanium is now. It is all really just a matter of time and work. I am really hoping that once I have the game somewhat playable and most systems functioning, that others will start assisting in the project. I know most people think this stuff is well beyond their skill level, but if I can learn it, most people with technical skill can as well. And, I think that if I write a couple of wiki pages to help people understand what to do, it will make a huge difference in who can start helping.

[Yeormom]

So much like myself, you too have little room for your other hobbies. :(

[cavedude]

Originally Posted by trevius

Back to working on the item serialization again, I broke down items into as close to an exact structure as I can figure out. This is for EQLive, but no new fields have changed from SoF to Live as far as I know, so it should also apply to SoF. This structure may not be 100%, but I think it is at least 95% accurate. This is based off of what I could find from 13th floor and packet collects from EQLive.

Just wanted to remind you that AugSlotUnk; is now known as AugSlotVisible.

[trevius]

Thanks. Yeah, I noticed that from 13th floor back when I first started working on serialization. I went ahead and changed it in the structure I posted above so it should be accurate now.