SoF Spawn_Struct Hacking

Shendare · #1 05-10-2009, 05:53 PM

I'm new to the PEQ scene, and would like to find ways to contribute.

I thought I might get my feet wet looking into the server code by trying to figure out some of the unknown fields in Spawn_Struct like haircolor, beardcolor, etc.

I found the place in SoF.cpp where such tinkering should go, inside ENCODE(OP_ZoneSpawns) starting around line 757, and knew there needed to be a better way to test things than coding values into random unknown fields and re-compiling for each test.

I put together a little bit of hack code that allows you to push test values into any of the unknown fields of the Spawn_Struct struct at run-time using a test NPC's lastname.

Code:

		//Hack Test for finding more fields in the Struct:
		if (emu->lastName[0] == '*') // Test NPC!
		{
			char code = emu->lastName[1];
			char* sep = (char*)memchr(&emu->lastName[2], '=', 10);
			uint32 ofs;
			uint32 val;
			uint8 rnd = rand() & 0x0F;
			if (sep == NULL)
			{
				ofs = 0;
				if ((emu->lastName[2] < '0') || (emu->lastName[2] > '9'))
				{
					val = rnd;
				}
				else
				{
					val = atoi(&emu->lastName[2]);
				}
			}
			else
			{
				sep[0] = NULL;
				ofs = atoi(&emu->lastName[2]);
				sep[0] = '=';
				if ((sep[1] < '0') || (sep[1] > '9'))
				{
					val = rnd;
				}
				else
				{
					val = atoi(&sep[1]);
				}
			}

			char hex[] = "0123456789ABCDEF";
			size_t len = strlen(emu->lastName);
			
			eq->lastName[len + 0] = ' ';
			eq->lastName[len + 1] = code;
			eq->lastName[len + 2] = '0' + ((ofs / 1000) % 10);
			eq->lastName[len + 3] = '0' + ((ofs / 100) % 10);
			eq->lastName[len + 4] = '0' + ((ofs / 10) % 10);
			eq->lastName[len + 5] = '0' + (ofs % 10);
			eq->lastName[len + 6] = '=';
			eq->lastName[len + 7] = '0' + ((val / 100) % 10);
			eq->lastName[len + 8] = '0' + ((val / 10) % 10);
			eq->lastName[len + 9] = '0' + (val % 10);
			eq->lastName[len + 10] = 0x00;

			switch (code)
			{
				case 'a':
					eq->unknown0001[ofs % 4] = val; break;
				case 'b':
					eq->unknown0006 = val; break;
				case 'c':
					eq->unknown0008 = val; break;
				case 'd':
					eq->unknown0011[ofs % 6] = val; break;
				case 'e':
					eq->unknown0009 = val; break;
				case 'f':
					eq->unknown0010[ofs % 4] = val; break;
				case 'g':
					eq->unknown0048[ofs % 4] = val; break;
				case 'h':
					eq->unknown0059 = val; break;
				case 'i':
					eq->unknown0074[ofs % 24] = val; break;
				case 'j':
					eq->unknown0077[ofs % 7] = val; break;
				case 'k':
					eq->unknown00771[ofs % 184] = val; break;
				case 'l':
					eq->unknown00772[ofs % 10] = val; break;
				case 'm':
					eq->unknown0079[ofs % 123] = val; break;
				case 'n':
					eq->unknown0080[ofs % 10] = val; break;
				case 'o':
					eq->unknown0106[ofs % 4] = val; break;
				case 'p':
					eq->unknown0107[ofs % 5] = val; break;
				case 'q':
					eq->unknown01071[ofs % 5] = val; break;
				case 'r':
					eq->unknown0108[ofs % 6] = val; break;
				case 's':
					eq->unknown01082[ofs % 6] = val; break;
				case 't':
					eq->unknown0110[ofs % 20] = val; break;
				case 'u':
					eq->unknown01101[ofs % 21] = val; break;
				case 'v':
					eq->unknown0154[ofs % 4] = val; break;
				case 'w':
					eq->unknown0155[ofs % 4] = val; break;
				case 'x':
					eq->unknown0156[ofs % 4] = val; break;
				case 'y':
					eq->unknown0157[ofs % 4] = val; break;
				case 'z':
					eq->unknown0158[ofs % 4] = val; break;
				case 'A':
					eq->unknown0159[ofs % 4] = val; break;
				case 'B':
					eq->unknown0160[ofs % 4] = val; break;
				case 'C':
					eq->unknown0161[ofs % 4] = val; break;
				case 'D':
					eq->unknown0162[ofs % 4] = val; break;
				case 'E':
					eq->unknown0163[ofs % 4] = val; break;
				case 'F':
					eq->unknown0263[ofs % 2] = val; break;
				case 'G':
					eq->unknown0281 = val; break;
				case 'H':
					eq->unknown0308[ofs % 4] = val; break;
				case 'I':
					eq->unknown0309[ofs % 11] = val; break;
				case 'J':
					eq->unknown442[ofs % 8] = val; break;
				case 'K':
					eq->unknown0760 = val; break;
				case 'L':
					eq->unknown0761 = val; break;
				case 'M':
					eq->unknown0762 = val; break;
				case 'O':
					eq->unknown0763 = val; break;
				case 'P':
					eq->unknown0764 = val; break;
				case 'Q':
					eq->unknown0496[ofs % 2] = val; break;
				case 'X':
					((uint8*)eq)[ofs % 897] = val; break;
				case 'Z':
					eq->size = (float)val; break; // Test w/ size.
			}

To use the code, simply target any NPC in-game and change its last name to follow the special formula:

*(Code)[Optional-Offset=](Value)

I've been testing on Exterminator Valern just inside Felwithe, so I can target him and type "#npcedit lastname *Z2", then "#repop" (I've hotkeyed that one), and Valern will respawn with a size of 2, instead of his default 6!

You'll also get debug output tacked to the end of the lastname letting you know exactly what the hack-code interpreted from your formula. The debug output is in the format "(Code)(4-digit Offset)=(3-digit Value)". Because the Z code for size doesn't use an offset, the debug output for the previous example will be "Z0000=002".

If you want the code to pick a random value to put in the specified field, use any non-numeric character for the value. I generally use '?'. So, you could type "#npcedit lastname *Z?" and repop and the NPC will be a random size from 0-15 every time you repop.

For the code character, all alpha digits from a-z and A-Q map to specific sets of unknown fields in the Spawn_Struct. If you want to push the value 10 into unknown0110[3], for example, you would make the NPC's lastname "*t3=10" and repop, and there you go!

The real catch-all hack code is 'X'. If you specify X as the code character, you can push any value into any offset of Spawn_Struct. For example, looking in SoF_structs.h you see that gender is at offset 22 in the Spawn_Struct. If you specify "#npcedit lastname *X22=1", the NPC will repop as a female.

Anyway, I thought this might be useful for anyone else who feels like poking around in Spawn_Struct to try to figure out some of these many unknown fields as well!

- Shendare

**trevius** · #2 05-10-2009, 06:57 PM

Woah, thanks! I know Derision has some ways to read through the IDA output from the eqgame.exe that he can use to find many fields. I am not quite that knowledgeable about the ASM code. You have no idea how much of a pain it was to do the field finding stuff I have done for it so far lol. I wish I had this code from the start! I will definitely give it a try.

The SoF spawn struct to this point is pretty much chaotic. The unknown fields are essentially all just spacers at this point. We do not know the exact size of any particular unknown for sure. I had just been breaking them down and testing each section 1 by 1. It is hard to know what fields should be int8, in16 or int32 or maybe char.

I don't know how much of a pain it would be, but possibly if Derision has the time, maybe he could go through and break down the unknown parts of the structure into the proper sized fields. I am pretty sure he can do this just by reading the ASM code. Not that I want to give him more work, or offer him up to do something for us, but I think he is the only one in EQEmu who knows enough to actually do it. I may be wrong, but this step might not be too bad to do by just going down the ASM code.

Then, once the fields are broken down into the proper sizes and places, we can use this testing code to isolate what each field actually does. With the fields all in the right place, this step should be able to be done by just about anyone. I bet we can find new things that we didn't even know existed by doing this lol. Maybe we need to make a list of exactly what we are missing and also new stuff that we might expect to find. That way, we know exactly what to look for when testing this stuff so we do the appropriate test for each change we try.

Without having the proper field sizes and placement, the only other way I can think to do this properly would be to break down each unknown into separate int8s. That would mean a few hundred unknown int8 fields. It would also mean adjusting the case switching on that test code, but it wouldn't be too rough to do if necessary.

Shendare · #3 05-10-2009, 07:15 PM

Well, with just poking random 0-15 values into individual unknown bytes one by one over the course of the last hour, I've so far been able to isolate FlyMode.

It's in unknown00771[1], byte offset 174. So, if you use the lastname code *k1=2 or *X174=2 the NPC will have levitation FlyMode on.

I'm still poking away. I'm mostly through unknown0079[] so far, which probably puts me 2/3 through the unknowns!

Shendare · #4 05-10-2009, 07:37 PM

Woohoo!

hairstyle = unknown01082[4], Offset 533
haircolor = unknown01082[5], Offset 534
flymode = unknown00771[1], Offset 174

Shendare · #5 05-10-2009, 08:25 PM

Fixing hairstyle and haircolor in the struct definition and adding the eq->hairstyle = emu->hairstyle entries for the two into the .cpp code fixes NPCs, but turns my PC bald. Incorrect field location in the player profile?

**trevius** · #6 05-10-2009, 09:39 PM

It may depend on what exactly you are filling into those unknowns. Probably best to convert each one into separate int8s and see what happens. It could be that they are int32 and setting more than 1 of the bytes is making a number that is out of the range of the hair options, setting them to default of being bald. Purely guess at this point until I can do testing later.

Also, for your testing, I didn't think about it before, but it would have been best if you were using a Drakkin (race 522) for finding spawn struct stuff. I am sure they have a decent piece of the struct just for them and the new heritage settings they have.

Shendare · #7 05-11-2009, 01:17 AM

IsStatue = unknown0006, Offset 6 (Some models have a "Statue" animation that makes the NPC larger and striking a pose.)

DrakkinHeritage = unknown0011[3], Offset 14 (Body coloring, available colors at character creation dependent on player class)

DrakkinWhite = unknown0011[4], Offset 15 (Nonzero yields a white haired, white tattooed, white face-spotted Drakkin)

DrakkinWhite2 = unknown0011[5], Offset 16 (Acts like DrakkinWhite for all values 0-15 except 0, 2, 8, and 10)

NOTE: It's possible DrakkinWhite and DrakkinWhite2 somehow interact with DrakkinHeritage, and it's possible there's some 16-bit workings going on in the field causing the weirdness. Dunno. I can only test push 8-bit values at the moment.

DrakkinTattoo = unknown0079[0], Offset 369

Targetable11 = unknown0156[3], Offset 625 (If this field is set to 11, the NPC is untargetable. Weird, I know. Totally found it by accident!)

I'm still looking for beard color and Drakkin facial dots.

Also, I found the bug with my bald player character. This block of code in mob.cpp at line # 724 was messing things up. It assumes a zero value for the appearance fields is invalid, and sets such zero fields to 0xFF instead.

Code:

	ns->spawn.haircolor = haircolor ? haircolor : 0xFF;
	ns->spawn.beardcolor = beardcolor ? beardcolor : 0xFF;
	ns->spawn.eyecolor1 = eyecolor1 ? eyecolor1 : 0xFF;
	ns->spawn.eyecolor2 = eyecolor2 ? eyecolor2 : 0xFF;
	ns->spawn.hairstyle = hairstyle ? hairstyle : 0xFF;
	ns->spawn.face = luclinface;
	ns->spawn.beard = beard ? beard : 0xFF;

The problem is, zero is perfectly valid. It's generally the first available appearance option for each field! Zero is the matted-hair for high elves, the dark brown hair color for races that can use it, the moustache face for humans, etc. Setting them to 255 actually SET them to an invalid value, making anyone with a zero value for the fields bald or incorrectly colored.

I simply commented out the 0xFF stuff, and things started working normally again.

Code:

	ns->spawn.haircolor = haircolor; // ? haircolor : 0xFF;
	ns->spawn.beardcolor = beardcolor; // ? beardcolor : 0xFF;
	ns->spawn.eyecolor1 = eyecolor1; // ? eyecolor1 : 0xFF;
	ns->spawn.eyecolor2 = eyecolor2; // ? eyecolor2 : 0xFF;
	ns->spawn.hairstyle = hairstyle; // ? hairstyle : 0xFF;
	ns->spawn.face = luclinface;
	ns->spawn.beard = beard; // ? beard : 0xFF;

Maybe they were put in place to correct a render problem in a pre-SoF client. I tried messing around with it for a couple of minutes with the Titanium client, but I couldn't even get beards to load with Titanium to test with. Hair color and hair style seemed to work properly with the 0 values, so the 0xFF stuff might have been implemented for a pre-Titanium client, though that would mean this bug would have been around for a long time.

**trevius** · #8 05-11-2009, 03:39 AM

Quote:

Originally Posted by Shendare

Targetable11 = unknown0156[3], Offset 625 (If this field is set to 11, the NPC is untargetable. Weird, I know. Totally found it by accident!)

That is interesting, because body type 11 is untargetable. So, my guess is that our current field for bodytype is probably incorrect. I got most of the fields originally from SEQ and they really only care about a few key parts of that structure. This probably explains why certain NPCs show up on SoF that shouldn't. Or at least partially explains that possibly.

Quote:

Originally Posted by Shendare

Also, I found the bug with my bald player character. This block of code in mob.cpp at line # 724 was messing things up. It assumes a zero value for the appearance fields is invalid, and sets such zero fields to 0xFF instead.

Code:

	ns->spawn.haircolor = haircolor ? haircolor : 0xFF;
	ns->spawn.beardcolor = beardcolor ? beardcolor : 0xFF;
	ns->spawn.eyecolor1 = eyecolor1 ? eyecolor1 : 0xFF;
	ns->spawn.eyecolor2 = eyecolor2 ? eyecolor2 : 0xFF;
	ns->spawn.hairstyle = hairstyle ? hairstyle : 0xFF;
	ns->spawn.face = luclinface;
	ns->spawn.beard = beard ? beard : 0xFF;

The problem is, zero is perfectly valid. It's generally the first available appearance option for each field! Zero is the matted-hair for high elves, the dark brown hair color for races that can use it, the moustache face for humans, etc. Setting them to 255 actually SET them to an invalid value, making anyone with a zero value for the fields bald or incorrectly colored.

I simply commented out the 0xFF stuff, and things started working normally again.

Code:

	ns->spawn.haircolor = haircolor; // ? haircolor : 0xFF;
	ns->spawn.beardcolor = beardcolor; // ? beardcolor : 0xFF;
	ns->spawn.eyecolor1 = eyecolor1; // ? eyecolor1 : 0xFF;
	ns->spawn.eyecolor2 = eyecolor2; // ? eyecolor2 : 0xFF;
	ns->spawn.hairstyle = hairstyle; // ? hairstyle : 0xFF;
	ns->spawn.face = luclinface;
	ns->spawn.beard = beard; // ? beard : 0xFF;

Maybe they were put in place to correct a render problem in a pre-SoF client. I tried messing around with it for a couple of minutes with the Titanium client, but I couldn't even get beards to load with Titanium to test with. Hair color and hair style seemed to work properly with the 0 values, so the 0xFF stuff might have been implemented for a pre-Titanium client, though that would mean this bug would have been around for a long time.

This is an interesting find and might be a great breakthrough for handling face settings. Definitely worth looking into.

I am going to have to give this code a try. It beats the crap out of changing whole chunks of the struct to a certain value, compiling, starting the server, logging in and hopefully seeing a difference, and then starting the whole process over again lol.

**trevius** · #9 05-11-2009, 05:52 AM

LOL, I got it setup and playing with it now. Just found some weird effect with "i5" as the last name. It creates a rain cloud above the spawn and rains on them for a few seconds. Also, other ones like i10, i11, i2, and more have really cool effects as well. I am guessing you need spell effects on to see these, but they are things I have never seen before :P Not sure exactly what they are for other than maybe having invisible NPCs use them to create visual effects maybe. Definitely loving the new way of handling this testing.

Also, after trying X613=11, I verified that the current setting for bodytype works, but I wonder why they have 2 fields for it, since X625=11 makes them untargetable as well.

Quote:

DrakkinHeritage = unknown0011[3], Offset 14 (Body coloring, available colors at character creation dependent on player class)

DrakkinWhite = unknown0011[4], Offset 15 (Nonzero yields a white haired, white tattooed, white face-spotted Drakkin)

DrakkinWhite2 = unknown0011[5], Offset 16 (Acts like DrakkinWhite for all values 0-15 except 0, 2, 8, and 10)

Yeah, 14, and 15 definitely seem to be an int16 for heritage. Not sure why 16 effects it too, but maybe it is an int32 and only accept certain ranges.

Found Drakkin face spikes are 882 and they range from 0 to 7. Also note that Heritage ranges from 0 to 6.

770 appears to be an int32 that if you set anything in (other than 0), the NPC will not be visible. It will still be targetable with /target, but that seems to be the only way.

740 puts an exclamation point in front of the name of the NPC.

656 is invisible/sneaking

Shendare · #10 05-11-2009, 12:12 PM

Yeah, I was messing with 770 a little and found that my client crashed if I attacked the NPC while it was invisible.

Good find on the special i-codes. I totally forgot I'd turned off spell effects for something or other. I totally missed them because of that!

The only thing I haven't found that I've been looking for is beard color, and I'm kicking myself for not having the foresight to make sure my test NPCs had beards while I was looking for hairstyle and haircolor. Bonehead.

I also figured out that some races (high elves for one) get their facial hair not from the beard field, but from a tens-digit offset to the face field. This also appears to be the way Barbarians get their woads.

Checks for these should probably be put into the OP_ZoneSpawns area there to check the race and (if necessary) dynamically adjust the eq->face field by (beard * 10) for beard-face races. I'm guessing we'll have to add some fields to npc_types to allow for face decorations like woads (if race == Barb, face += face_decoration*10) and Drakkin spikes, as well as Drakkin tattoos and heritage.

Shendare · #11 05-11-2009, 12:24 PM

Quote:

Originally Posted by trevius

...
740 puts an exclamation point in front of the name of the NPC.
...

Oh my god, I think you found Doug! I remember from years and years ago after a patch, people were finding unresponsive naked human male models in random places in the game with only an exclamation point as their name. The EQ team responded to the community's questions with the response that these were test characters that aren't normally visible, and they'd figure out what broke in the patch that allowed people to see them. The team revealed the little tidbit that they had taken to calling these default human models "Doug", so whenever someone in the game found an unresponsive naked human male NPC that probably shouldn't be there (or should look different), they reported a "Doug".

I wonder if offset 740 marks the character as an EQ dev test character, and the original Doug didn't actually have "!" as its name, but had a blank name with the equivalent of Flag 740 set. That just blows my mind, and totally brings back some old EQ nostalgia. LOL.

- Shendare

**trevius** · #12 05-11-2009, 06:00 PM

Yeah, woads being set by factor of 10 on the face setting is known and in the wiki I believe. Shouldn't be too hard to find beard color and such once test is done with an NPC that has a beard set on them since beard is already identified.

I was thinking about the storm cloud that spawns from i5 and I actually recall something in either BoT or PoStorms about storm clouds spawning over your head and you have to move or something or something bad happens. Not too clear on the details, but it occurred to me that this is probably a case of when those effects are used.

I bet we could apply your same code to figure out more structure related stuff. Like, we could modify it for the player profile by using something that is easily changeable like currency. Just have it so that if the character has 1337 copper on their character, it will go into test mode. Then, use the plat and gold on that character to set the offset (number of plat) and what to set that offset to (number of gold). It would require zoning each time to test the changes, but it is at least a possible option to consider if we need it. Most of the player profile should be pretty good already, but if we still need to find anything in it, this would probably work.

One thing we need to get working better cosmetically is the character select screen. Unfortunately I think the client limits names during creation to only include letters. So either we would need to use letters and convert them or maybe use stats, but stats would probably be even trickier. Maybe for testing, the first 3 letters could be Z (Zzz) to designate that it is a test, and the 4th letter could be the field in the struct and the 5th would be the value to set it at. This might get a bit confusing, but the char select struct isn't overly large so it shouldn't be too much work to get it all figured out. Can't really set offsets since it is a variable length structure. I think the struct is already pretty close, so maybe finding what is throwing it off would make an easy fix to align everything back up again. There are only a few unknown fields in that struct, so I am sure those are Drakkin related, or at least most of them probably are.

Maybe for char select, we could use 1337 speak lettering in place of numbers to make it easier to set.

0 = o (O)
1 = i (I)
2 = z (Z)
3 = e (E)
4 = a (A)
5 = s (S)
6 = g (G)
7 = t (T)
8 = b (B)
9 = q (Q)

So, if you wanted to set field "a" (level) to 25, you would name the character "Zzzazs". Or if you wanted to set field "b" (haircolor) to 10, you would name the character Zzzbio.

This isn't quite as critical as the spawn struct, but everything that gets fixed makes SoF that much closer to being fully completed

Shendare · #13 05-11-2009, 06:56 PM

Sounds interesting! I'm all for getting SoF working better.

I did notice I left a quickie hack in the code that should be cleaned up.

This line...

Code:

  size_t len = strlen(emu->lastName);

...should be moved up to this block, which should be changed to refer to it...

Code:

  char code = emu->lastName[1];

+ size_t len = strlen(emu->lastName);
- char* sep = (char*)memchr(&emu->lastName[2], '=', 10);
+ char* sep = (char*)memchr(&emu->lastName[2], '=', len - 2);

  uint32 ofs;

This prevents reading beyond the end of lastName in case it's fewer than 10 characters including the null.

**trevius** · #14 05-12-2009, 07:45 AM

I finally identified beard color as offset 373. Got all of the new fields in and adjusted the struct to have the proper offset labels as well as correctly renamed all of the unknowns. Seems to work perfectly for NPCs now. I also added in your change to mob.cpp for facial features, but that might require further testing to verify if it is a good change or not (though it makes sense to me). And, I added in your testing code in place of the sloppy hack I had commented out before lol. It is commented out, but I figure it might be useful to others doing struct work, so it may be a good idea to keep it in there as commented out. I put this up on the SVN on R501 and gave you (shendare) credit for it since you did most of it.

I think that pretty much finalizes the spawn struct, or at least very close now. Still need to figure out a few things for players, but that shouldn't be too hard I think. I put temp hacks in for the drakkin extra features so they get set in the encode to whatever face is set to. At least this will let some variety into drakkin NPCs (and PCs for now as well).

Shendare · #15 05-12-2009, 09:25 AM

I'm glad to see that BeardColor gets applied to the face-based facial hair as well as the beard field mesh facial hair.

I dinked around a little with player appearance as well, but found that I had to modify some things. SoF's FaceChange_Struct is 24 bytes now instead of 7, assumedly to account for Tattoo, DrakkinFaceSpikes, and possibly DrakkinHeritage (though that's read-only in face changes). The rest must be reserved space.

I also noticed a block in Client::Handle_OP_FaceChange (or whatever it was called) that did something similar to the 0xFF problem above, except this time it's changing zero values to 99. That'll have to be taken out to properly handle face changes as well.

Then it's just a matter of making sure things are applied in PlayerProfile properly.