16 bit checksums

[fathernitwit]

Hello,

im new around here, just found out about eqemu about 2 days ago, but I am very excited about it... anyways... I am a decently skilled network programmer, I have about 5 years of protocol development experience.

I have been reading about the login server problems, and am trying to analyze the protocol... One thing I am running into is once the login process gets started, they start appending 16 bit checksums to the end of the packets.

I am unable to figure out how this checksum is computed... The only 16 bit checksum I am framiliar with is crc16, and beings that they use crc32 for their normal packets, I gave it a try... but it isnt giving me anything... I am using CalcCRC16Bytes (from http://www.indigosystems.com/CServices/crc16_c.html). I have tried it both including the trailing 2 bytes as 0's as well as excluding the trailing 2 bytes...

Is anybody framiliar with any other 16 bit checksums, a different implementation of crc16 for bytes, or might offer other insight as to how they might be computing this checksum?

two example packets, in hex:
00 15 00 01 a9 bf
00 15 00 02 f8 05

I belive that the last 2 bytes of each line are the checksum.. I dont know what else it could be.

[RangerDown]

While I might not have any idea what I'm talking about, I'll throw out a possibility:

The bytes as you look at them might be in network byte order or host byte order. Using htons() or ntohs() functions, try to look at them the other way around. Maybe the CRC will calculate differently.

[Facet42]

Are you sure it's a checksum? Could be generated by some kind of hashing function, similar to MD5, etc.

[fathernitwit]

yes, I agree with that.. It could easily be a hash, i basically consider them to be the same thing... but I dont know any hashing functions which produce 16 bit results... md5 is much longer than that (128?), and sha is even longer...

im not framiliar with any that produce this short of a checksum/hash... they could have easily devised their own checksum too, which would be very very effective at stopping people like me (:


Another possibility is it could be using some exchanged key as part of its computation... there are two 4 byte components of the first packet exchange (the only two packets without these checksums) that seems to be data of some sort... I figured it was some encryption key or something, and it could be easily used to compilcate my life.

[kathgar]

<pre>

unsigned long IntArray[]={
0x00000000,
0x77073096,
0xEE0E612C,
0x990951BA,
0x076DC419,
0x706AF48F,
0xE963A535,
0x9E6495A3,
0x0EDB8832,
0x79DCB8A4,
0xE0D5E91E,
0x97D2D988,
0x09B64C2B,
0x7EB17CBD,
0xE7B82D07,
0x90BF1D91,
0x1DB71064,
0x6AB020F2,
0xF3B97148,
0x84BE41DE,
0x1ADAD47D,
0x6DDDE4EB,
0xF4D4B551,
0x83D385C7,
0x136C9856,
0x646BA8C0,
0xFD62F97A,
0x8A65C9EC,
0x14015C4F,
0x63066CD9,
0xFA0F3D63,
0x8D080DF5,
0x3B6E20C8,
0x4C69105E,
0xD56041E4,
0xA2677172,
0x3C03E4D1,
0x4B04D447,
0xD20D85FD,
0xA50AB56B,
0x35B5A8FA,
0x42B2986C,
0xDBBBC9D6,
0xACBCF940,
0x32D86CE3,
0x45DF5C75,
0xDCD60DCF,
0xABD13D59,
0x26D930AC,
0x51DE003A,
0xC8D75180,
0xBFD06116,
0x21B4F4B5,
0x56B3C423,
0xCFBA9599,
0xB8BDA50F,
0x2802B89E,
0x5F058808,
0xC60CD9B2,
0xB10BE924,
0x2F6F7C87,
0x58684C11,
0xC1611DAB,
0xB6662D3D,
0x76DC4190,
0x01DB7106,
0x98D220BC,
0xEFD5102A,
0x71B18589,
0x06B6B51F,
0x9FBFE4A5,
0xE8B8D433,
0x7807C9A2,
0x0F00F934,
0x9609A88E,
0xE10E9818,
0x7F6A0DBB,
0x086D3D2D,
0x91646C97,
0xE6635C01,
0x6B6B51F4,
0x1C6C6162,
0x856530D8,
0xF262004E,
0x6C0695ED,
0x1B01A57B,
0x8208F4C1,
0xF50FC457,
0x65B0D9C6,
0x12B7E950,
0x8BBEB8EA,
0xFCB9887C,
0x62DD1DDF,
0x15DA2D49,
0x8CD37CF3,
0xFBD44C65,
0x4DB26158,
0x3AB551CE,
0xA3BC0074,
0xD4BB30E2,
0x4ADFA541,
0x3DD895D7,
0xA4D1C46D,
0xD3D6F4FB,
0x4369E96A,
0x346ED9FC,
0xAD678846,
0xDA60B8D0,
0x44042D73,
0x33031DE5,
0xAA0A4C5F,
0xDD0D7CC9,
0x5005713C,
0x270241AA,
0xBE0B1010,
0xC90C2086,
0x5768B525,
0x206F85B3,
0xB966D409,
0xCE61E49F,
0x5EDEF90E,
0x29D9C998,
0xB0D09822,
0xC7D7A8B4,
0x59B33D17,
0x2EB40D81,
0xB7BD5C3B,
0xC0BA6CAD,
0xEDB88320,
0x9ABFB3B6,
0x03B6E20C,
0x74B1D29A,
0xEAD54739,
0x9DD277AF,
0x04DB2615,
0x73DC1683,
0xE3630B12,
0x94643B84,
0x0D6D6A3E,
0x7A6A5AA8,
0xE40ECF0B,
0x9309FF9D,
0x0A00AE27,
0x7D079EB1,
0xF00F9344,
0x8708A3D2,
0x1E01F268,
0x6906C2FE,
0xF762575D,
0x806567CB,
0x196C3671,
0x6E6B06E7,
0xFED41B76,
0x89D32BE0,
0x10DA7A5A,
0x67DD4ACC,
0xF9B9DF6F,
0x8EBEEFF9,
0x17B7BE43,
0x60B08ED5,
0xD6D6A3E8,
0xA1D1937E,
0x38D8C2C4,
0x4FDFF252,
0xD1BB67F1,
0xA6BC5767,
0x3FB506DD,
0x48B2364B,
0xD80D2BDA,
0xAF0A1B4C,
0x36034AF6,
0x41047A60,
0xDF60EFC3,
0xA867DF55,
0x316E8EEF,
0x4669BE79,
0xCB61B38C,
0xBC66831A,
0x256FD2A0,
0x5268E236,
0xCC0C7795,
0xBB0B4703,
0x220216B9,
0x5505262F,
0xC5BA3BBE,
0xB2BD0B28,
0x2BB45A92,
0x5CB36A04,
0xC2D7FFA7,
0xB5D0CF31,
0x2CD99E8B,
0x5BDEAE1D,
0x9B64C2B0,
0xEC63F226,
0x756AA39C,
0x026D930A,
0x9C0906A9,
0xEB0E363F,
0x72076785,
0x05005713,
0x95BF4A82,
0xE2B87A14,
0x7BB12BAE,
0x0CB61B38,
0x92D28E9B,
0xE5D5BE0D,
0x7CDCEFB7,
0x0BDBDF21,
0x86D3D2D4,
0xF1D4E242,
0x68DDB3F8,
0x1FDA836E,
0x81BE16CD,
0xF6B9265B,
0x6FB077E1,
0x18B74777,
0x88085AE6,
0xFF0F6A70,
0x66063BCA,
0x11010B5C,
0x8F659EFF,
0xF862AE69,
0x616BFFD3,
0x166CCF45,
0xA00AE278,
0xD70DD2EE,
0x4E048354,
0x3903B3C2,
0xA7672661,
0xD06016F7,
0x4969474D,
0x3E6E77DB,
0xAED16A4A,
0xD9D65ADC,
0x40DF0B66,
0x37D83BF0,
0xA9BCAE53,
0xDEBB9EC5,
0x47B2CF7F,
0x30B5FFE9,
0xBDBDF21C,
0xCABAC28A,
0x53B39330,
0x24B4A3A6,
0xBAD03605,
0xCDD70693,
0x54DE5729,
0x23D967BF,
0xB3667A2E,
0xC4614AB8,
0x5D681B02,
0x2A6F2B94,
0xB40BBE37,
0xC30C8EA1,
0x5A05DF1B,
0x2D02EF8D,
};

unsigned long myCRC(unsigned char *buf, int size, int key)
{
	
/*
sub_0_10020760  proc near               ; CODE XREF: sub_0_10008620+AEp
                                        ; sub_0_10022A90+14Fp ...

arg_0           = dword ptr  4
arg_4           = dword ptr  8
arg_8           = dword ptr  0Ch
*/

	//int *pecx = buf;
	int ecx = key;				//mov     ecx, [esp+arg_8]
	int eax = ecx;				//mov     eax, ecx
	eax~=eax;					//not     eax
    eax&=0xFF;					//and     eax, 0FFh
    eax=IntArray[eax];			//mov     eax, dword_0_10115D38[eax*4] IntArray
    eax ^= 0x00FFFFFF;			//xor     eax, 0FFFFFFh
  	int edx = ecx;				//mov     edx, ecx
	edx = edx >> 8;				//sar     edx, 8
    edx = edx ^ eax;			//xor     edx, eax
    eax = eax >> 8;				//sar     eax, 8
    edx &= 0xFF;				//and     edx, 0FFh
    eax &= 0x00FFFFFF;			//and     eax, 0FFFFFFh
								//push    esi
    eax ^= IntArray[edx];		//xor     eax, dword_0_10115D38[edx*4]
    edx = ecx;					//mov     edx, ecx
    edx = edx >> 0x10;			//sar     edx, 10h
    edx ^= eax;					//xor     edx, eax
    eax = eax >> 8;				//sar     eax, 8
    edx &= 0xFF;				//and     edx, 0FFh
    int esi = IntArray[edx];	//mov     esi, dword_0_10115D38[edx*4]
    edx = size;				//mov     edx, [esp+4+arg_4]
    eax &= 0x00FFFFFF;			//and     eax, 0FFFFFFh
    eax ^= esi;					//xor     eax, esi
    ecx = ecx >> 0x18;			//sar     ecx, 18h
    ecx ^= eax;					//xor     ecx, eax
    ecx &= 0xFF;				//and     ecx, 0FFh
    esi = IntArray[ecx];		//mov     esi, dword_0_10115D38[ecx*4]
   // ecx = buf;				//mov     ecx, [esp+4+arg_0]
    eax = eax >> 8;				//sar     eax, 8
    eax &= 0x00FFFFFF;			//and     eax, 0FFFFFFh
    eax ^= esi;					//xor     eax, esi
    /* int* esi = ecx+edx //??*///lea     esi, [ecx+edx]
    for(int x = 0; x < size; x++)
	{							//eax is the crc, ecx is the current part of the buffer
		int edx = 0;			//xor     edx, edx
		edx = buf[x] & 0x00FF;	//mov     dl, [ecx]

	/*if(pos > size)			//cmp     ecx, esi
		return ~eax;			//jnb     short loc_0_10020803
	*/
								//push    edi

//loc_0_100207E0:                         ; CODE XREF: sub_0_10020760+A0j //LOOP
        edx ^= eax;				//xor     edx, eax
        eax = eax >> 8;			//sar     eax, 8
        edx &= 0xFF;			//and     edx, 0FFh
        edi = IntArray[edx];	//mov     edi, dword_0_10115D38[edx*4]
        eax &= 0x00FFFFFF;		//and     eax, 0FFFFFFh
        eax ^= edi;				//xor     eax, edi
        					//inc     ecx
								//cmp     ecx, esi
								// jb      short loc_0_100207E0
								//pop     edi
	}

	return ~eax;
}
/*loc_0_10020803:                         ; CODE XREF: sub_0_10020760+7Dj
                not     eax
                pop     esi
                retn
sub_0_10020760  endp

*/

</pre>

I never did finish that before my accident.. I'm not sure it works but that IS the function that is called. I had completely forgot about it until now. Also the forums apparently screwed up my indentation.

Edit: Yes, this will generate a 32 bit number, it is either ah or al of it that is used depending.

[Windcatcher]

Whatever it is, it's not IEEE 802.3 Ethernet. For comparison, here's the standard Ethernet algorithm, which the .S3D files use:

Unit Ethernet;
// ----------------------------------------------------------------------------------
// This calculates a string's 32-bit CRC value usng the IEEE 802.3 Ethernet standard.
//
// This algorithm can be found at:
//
// http://cell-relay.indiana.edu/cell-relay/publications/software/CRC/32bitCRC.c.html
// ----------------------------------------------------------------------------------

Interface

Type BytePtr = ^Byte;

Function Update_CRC(CRC_Accum: LongWord; P: BytePtr; Size: LongWord): LongWord;
Function GetCRCOfString(St: String): LongWord;

Implementation

Const Polynomial = $04C11DB7;

Var CRC_Table: Array [0..255] Of LongWord;

Procedure Gen_CRC_Table;
Var I,J,CRC_Accum: LongWord;
Begin
  For I := 0 To 255 Do
  Begin
    CRC_Accum := I Shl 24;
    For J := 0 To 7 Do
    Begin
      If (CRC_Accum And $80000000) <> 0
       Then CRC_Accum := (CRC_Accum Shl 1) Xor Polynomial
       Else CRC_Accum := CRC_Accum Shl 1;
    End; // For J
    CRC_Table[I] := CRC_Accum;
  End; // For I
End; // Gen_CRC_Table

Function Update_CRC(CRC_Accum: LongWord; P: BytePtr; Size: LongWord): LongWord;
Var I: LongWord;
Begin
  While Size > 0 Do
  Begin
    I := ((CRC_Accum Shr 24) Xor P^) And $FF;
    Inc(LongInt(P));
    CRC_Accum := (CRC_Accum Shl 8) Xor CRC_Table[I];
    Dec(Size);
  End; // For J
  Result := CRC_Accum;
End; // Update_CRC

Function GetCRCOfString(St: String): LongWord;
Var St1: BytePtr;
Begin
  If St <> '' Then
  Begin
    GetMem(St1,Length(St) + 1);
    FillChar(St1^,Length(St) + 1,0);
    Move(St[1],St1^,Length(St));
    Result := Update_CRC(0,St1,Length(St) + 1);
    FreeMem(St1,Length(St) + 1);
  End
  Else Result := 0;
End; // GetCRCOfString

Initialization
  Gen_CRC_Table;
End.

[fathernitwit]

Originally Posted by kathgar

I never did finish that before my accident.. I'm not sure it works but that IS the function that is called. I had completely forgot about it until now. Also the forums apparently screwed up my indentation.

Do you have any idea what that key argument is?

do you think it is the 4 byte thing from the initial packet exchange?
Im just going to try it however I can, but maybe you can share some insight.... messing with checksums is such a pain, since they are practically impossible to debug unless you really understand them.

also there is a reference to arg_0, i am not framiliar with that assembly notation, so im going to assume it is the first argument...

[kathgar]

I believe it is the initial key, and then the checksum from the last packet is the new key. I never finished looking into it. There also might be a different key for both client and server. So far i've only looked at the crc function and not much on how it interacted with everything else too much.

[fathernitwit]

NICE! Thanks for myCRC, thats the ticket.

Finally figured it out...

the first two packets in the login process are:

//client sends to server
struct {
	unsigned short type;
	unsigned long num1;
	unsigned long key1;
	unsigned long num2;
};

//server's reply
struct {
	unsigned short type;
	unsigned long key1;
	unsigned long key2;
	unsigned long num1;
	unsigned short num2;
	unsigned long num3;
};

I dont know if you can put anything in key2, my key2 is stolen from a real packet stream (havent tried anything else)... the client's key1 seems to actually be a function of how many times you have tried to log in, and seems to be always the same on the first login. I know that the key2 used by real servers changes each time even if key1 is the same.

once you send key2 back to the client, that is your keyargument, for all packets, in either direction... I got tied up because I forgot to ntohl the key, damnit.

   //the len-2 assumes that the checksum bytes are allready appended to the buffer, be it send or receive
unsigned long crc;
	crc = myCRC((unsigned char *) buffer, len-2, ntohl(reply.key2));
	unsigned short cc = htons(crc & 0xFFFF);
// now set the trailing two bytes...

There is one problem, hell if I know what it is... where a specific 2-byte packets is not checksuming properly (00 06)... must be an initilization thing or something... this 2 byte dude also seems to break the rules as far as the 'same checksum in either direction'... they both reply with a strange checksum, but i dont care because #1, i can hard-code that checksum, and #2, they seems to be just 'ping' type packets, no information.



so now with some DNS rewrites, I can mimic the login server and connect to my world (nothing intelligent yet)... but it is now broken when i log into a zone? I THINK this is a known thing.. that the newest client is broken/incompatible with eqemu... is this a true statement? Is there any technical discussion about this problem anywhere?

[RangerDown]

EQEmu login server always sends key2 with a value of 0. The result of that is the username/pass are sent in the clear, so we don't have to worry about how the user/pass are encrypted. At least that's how pre-June22 works. Something about that might have changed in later patches.

I think key1 is *supposed* to be random, but it looks like somebody forgot to put in a line that seeds the random. That's quality SOE programming for ya :lol:

And as far as entering a zone goes, yeah every few patches it seems the opcodes of the packets change big time. I think the Emu coders have all been pretty much hanging at the pre-Jun22 patch until the login code is figured out, so it's almost certain that Emu zoneserver is not gonna be talking to a newly-patched client correctly.

[kathgar]

The first few patches of the new login server were fine once we got it done, but this time they also changed more parts the basic eq protocol that we have to go over. Then there is also the NEW NEW login that is on test. Needless to say there is not a whole lot of work being put into the login that is about to die.

[fathernitwit]

so there is a new login server in testing by eqlive that is going to replace what is there now, as far as the client is going to patch? Or are you referencing the login server -> world server stuff?

AKA.. i am wasting my time? that would be good to know

im not framiliar with test EQ... is that as simple as running the test EQ client?

[RangerDown]

In coming months they will be implementing a SWG-style launchpad for the login process. While I may be over the top in my doom-and-gloom prediction, I suspect that we won't be using that launchpad client for quite some time after it goes live. I'd bet it makes the month we've had our current client locked look like a daytime nap.

If you think that it'd be a great deal of effort to proceed with the current login protocol, then Kathgar might be right in saying don't waste your time. From what you've posted though, it looks like you've got the CRC figured out. And did you try to set Key2 to 0 all the time, does that make the user/pass send in the clear? If so, then your work is 90% done, and if the Emu login server is allowed to use your code, our users might be able to patch up to at least this month's version... to hold them over for what might be months of locking down a client version after that launchpad goes live.

[kathgar]

I did not say it was a waste of time, just that there is no sense of urgancy among the developers. The old new login was only on test for a short time before it went live, and I see no reason why they would not do the same with this launchpad. We only have a few people that can/will work on these kinds of updates anyways. At the moment I am devoting more of my time to other efforts, yet still working on the current situation. Even with a new login done however, there is still the change that broke our packetcollector that as far as I know has not been resolved.

[fathernitwit]

ok, with this information, let me state where I am sitting, and maybe you can tell me if my efforts can get me to my goal.

Basically, my goal is to set up a purely local EQ server, no internet required... once this is up and running, I see no reason that I will bother to patch EQ. I also have no resources to get ahold of older versions of EQ.

So assuming that I get this login server coded up and working fine such that I can connect to my world server... is it realistic to think that eqemu will get updated to a state which is compatible with the EQlive client one could install today?
I know it all depends on what really changed recently, so i guess my question is really: are people working on the current probelm, or are most devs just sort of waiting to see what happens before they put a lot of effort into it.

Insight is appriciated.