EQEmulator Forums - 16 bit checksums

EQEmulator Forums (https://www.eqemulator.org/forums/index.php)

- Archive::Development (https://www.eqemulator.org/forums/forumdisplay.php?f=621)

- - 16 bit checksums (https://www.eqemulator.org/forums/showthread.php?t=15048)

16 bit checksums

Hello,

im new around here, just found out about eqemu about 2 days ago, but I am very excited about it... anyways... I am a decently skilled network programmer, I have about 5 years of protocol development experience.

I have been reading about the login server problems, and am trying to analyze the protocol... One thing I am running into is once the login process gets started, they start appending 16 bit checksums to the end of the packets.

I am unable to figure out how this checksum is computed... The only 16 bit checksum I am framiliar with is crc16, and beings that they use crc32 for their normal packets, I gave it a try... but it isnt giving me anything... I am using CalcCRC16Bytes (from http://www.indigosystems.com/CServices/crc16_c.html). I have tried it both including the trailing 2 bytes as 0's as well as excluding the trailing 2 bytes...

Is anybody framiliar with any other 16 bit checksums, a different implementation of crc16 for bytes, or might offer other insight as to how they might be computing this checksum?

two example packets, in hex:
00 15 00 01 a9 bf
00 15 00 02 f8 05

I belive that the last 2 bytes of each line are the checksum.. I dont know what else it could be.

While I might not have any idea what I'm talking about, I'll throw out a possibility:

The bytes as you look at them might be in network byte order or host byte order. Using htons() or ntohs() functions, try to look at them the other way around. Maybe the CRC will calculate differently.

Are you sure it's a checksum? Could be generated by some kind of hashing function, similar to MD5, etc.

yes, I agree with that.. It could easily be a hash, i basically consider them to be the same thing... but I dont know any hashing functions which produce 16 bit results... md5 is much longer than that (128?), and sha is even longer...

im not framiliar with any that produce this short of a checksum/hash... they could have easily devised their own checksum too, which would be very very effective at stopping people like me (:

Another possibility is it could be using some exchanged key as part of its computation... there are two 4 byte components of the first packet exchange (the only two packets without these checksums) that seems to be data of some sort... I figured it was some encryption key or something, and it could be easily used to compilcate my life.

<pre>

Code:

unsigned long IntArray[]={

0x00000000,

0x77073096,

0xEE0E612C,

0x990951BA,

0x076DC419,

0x706AF48F,

0xE963A535,

0x9E6495A3,

0x0EDB8832,

0x79DCB8A4,

0xE0D5E91E,

0x97D2D988,

0x09B64C2B,

0x7EB17CBD,

0xE7B82D07,

0x90BF1D91,

0x1DB71064,

0x6AB020F2,

0xF3B97148,

0x84BE41DE,

0x1ADAD47D,

0x6DDDE4EB,

0xF4D4B551,

0x83D385C7,

0x136C9856,

0x646BA8C0,

0xFD62F97A,

0x8A65C9EC,

0x14015C4F,

0x63066CD9,

0xFA0F3D63,

0x8D080DF5,

0x3B6E20C8,

0x4C69105E,

0xD56041E4,

0xA2677172,

0x3C03E4D1,

0x4B04D447,

0xD20D85FD,

0xA50AB56B,

0x35B5A8FA,

0x42B2986C,

0xDBBBC9D6,

0xACBCF940,

0x32D86CE3,

0x45DF5C75,

0xDCD60DCF,

0xABD13D59,

0x26D930AC,

0x51DE003A,

0xC8D75180,

0xBFD06116,

0x21B4F4B5,

0x56B3C423,

0xCFBA9599,

0xB8BDA50F,

0x2802B89E,

0x5F058808,

0xC60CD9B2,

0xB10BE924,

0x2F6F7C87,

0x58684C11,

0xC1611DAB,

0xB6662D3D,

0x76DC4190,

0x01DB7106,

0x98D220BC,

0xEFD5102A,

0x71B18589,

0x06B6B51F,

0x9FBFE4A5,

0xE8B8D433,

0x7807C9A2,

0x0F00F934,

0x9609A88E,

0xE10E9818,

0x7F6A0DBB,

0x086D3D2D,

0x91646C97,

0xE6635C01,

0x6B6B51F4,

0x1C6C6162,

0x856530D8,

0xF262004E,

0x6C0695ED,

0x1B01A57B,

0x8208F4C1,

0xF50FC457,

0x65B0D9C6,

0x12B7E950,

0x8BBEB8EA,

0xFCB9887C,

0x62DD1DDF,

0x15DA2D49,

0x8CD37CF3,

0xFBD44C65,

0x4DB26158,

0x3AB551CE,

0xA3BC0074,

0xD4BB30E2,

0x4ADFA541,

0x3DD895D7,

0xA4D1C46D,

0xD3D6F4FB,

0x4369E96A,

0x346ED9FC,

0xAD678846,

0xDA60B8D0,

0x44042D73,

0x33031DE5,

0xAA0A4C5F,

0xDD0D7CC9,

0x5005713C,

0x270241AA,

0xBE0B1010,

0xC90C2086,

0x5768B525,

0x206F85B3,

0xB966D409,

0xCE61E49F,

0x5EDEF90E,

0x29D9C998,

0xB0D09822,

0xC7D7A8B4,

0x59B33D17,

0x2EB40D81,

0xB7BD5C3B,

0xC0BA6CAD,

0xEDB88320,

0x9ABFB3B6,

0x03B6E20C,

0x74B1D29A,

0xEAD54739,

0x9DD277AF,

0x04DB2615,

0x73DC1683,

0xE3630B12,

0x94643B84,

0x0D6D6A3E,

0x7A6A5AA8,

0xE40ECF0B,

0x9309FF9D,

0x0A00AE27,

0x7D079EB1,

0xF00F9344,

0x8708A3D2,

0x1E01F268,

0x6906C2FE,

0xF762575D,

0x806567CB,

0x196C3671,

0x6E6B06E7,

0xFED41B76,

0x89D32BE0,

0x10DA7A5A,

0x67DD4ACC,

0xF9B9DF6F,

0x8EBEEFF9,

0x17B7BE43,

0x60B08ED5,

0xD6D6A3E8,

0xA1D1937E,

0x38D8C2C4,

0x4FDFF252,

0xD1BB67F1,

0xA6BC5767,

0x3FB506DD,

0x48B2364B,

0xD80D2BDA,

0xAF0A1B4C,

0x36034AF6,

0x41047A60,

0xDF60EFC3,

0xA867DF55,

0x316E8EEF,

0x4669BE79,

0xCB61B38C,

0xBC66831A,

0x256FD2A0,

0x5268E236,

0xCC0C7795,

0xBB0B4703,

0x220216B9,

0x5505262F,

0xC5BA3BBE,

0xB2BD0B28,

0x2BB45A92,

0x5CB36A04,

0xC2D7FFA7,

0xB5D0CF31,

0x2CD99E8B,

0x5BDEAE1D,

0x9B64C2B0,

0xEC63F226,

0x756AA39C,

0x026D930A,

0x9C0906A9,

0xEB0E363F,

0x72076785,

0x05005713,

0x95BF4A82,

0xE2B87A14,

0x7BB12BAE,

0x0CB61B38,

0x92D28E9B,

0xE5D5BE0D,

0x7CDCEFB7,

0x0BDBDF21,

0x86D3D2D4,

0xF1D4E242,

0x68DDB3F8,

0x1FDA836E,

0x81BE16CD,

0xF6B9265B,

0x6FB077E1,

0x18B74777,

0x88085AE6,

0xFF0F6A70,

0x66063BCA,

0x11010B5C,

0x8F659EFF,

0xF862AE69,

0x616BFFD3,

0x166CCF45,

0xA00AE278,

0xD70DD2EE,

0x4E048354,

0x3903B3C2,

0xA7672661,

0xD06016F7,

0x4969474D,

0x3E6E77DB,

0xAED16A4A,

0xD9D65ADC,

0x40DF0B66,

0x37D83BF0,

0xA9BCAE53,

0xDEBB9EC5,

0x47B2CF7F,

0x30B5FFE9,

0xBDBDF21C,

0xCABAC28A,

0x53B39330,

0x24B4A3A6,

0xBAD03605,

0xCDD70693,

0x54DE5729,

0x23D967BF,

0xB3667A2E,

0xC4614AB8,

0x5D681B02,

0x2A6F2B94,

0xB40BBE37,

0xC30C8EA1,

0x5A05DF1B,

0x2D02EF8D,

};



unsigned long myCRC(unsigned char *buf, int size, int key)

{

        

/*

sub_0_10020760  proc near               ; CODE XREF: sub_0_10008620+AEp

                                        ; sub_0_10022A90+14Fp ...



arg_0           = dword ptr  4

arg_4           = dword ptr  8

arg_8           = dword ptr  0Ch

*/



        //int *pecx = buf;

        int ecx = key;                                //mov     ecx, [esp+arg_8]

        int eax = ecx;                                //mov     eax, ecx

        eax~=eax;                                        //not     eax

    eax&amp;=0xFF;                                        //and     eax, 0FFh

    eax=IntArray[eax];                        //mov     eax, dword_0_10115D38[eax*4] IntArray

    eax ^= 0x00FFFFFF;                        //xor     eax, 0FFFFFFh

          int edx = ecx;                                //mov     edx, ecx

        edx = edx >> 8;                                //sar     edx, 8

    edx = edx ^ eax;                        //xor     edx, eax

    eax = eax >> 8;                                //sar     eax, 8

    edx &amp;= 0xFF;                                //and     edx, 0FFh

    eax &amp;= 0x00FFFFFF;                        //and     eax, 0FFFFFFh

                                                                //push    esi

    eax ^= IntArray[edx];                //xor     eax, dword_0_10115D38[edx*4]

    edx = ecx;                                        //mov     edx, ecx

    edx = edx >> 0x10;                        //sar     edx, 10h

    edx ^= eax;                                        //xor     edx, eax

    eax = eax >> 8;                                //sar     eax, 8

    edx &amp;= 0xFF;                                //and     edx, 0FFh

    int esi = IntArray[edx];        //mov     esi, dword_0_10115D38[edx*4]

    edx = size;                                //mov     edx, [esp+4+arg_4]

    eax &amp;= 0x00FFFFFF;                        //and     eax, 0FFFFFFh

    eax ^= esi;                                        //xor     eax, esi

    ecx = ecx >> 0x18;                        //sar     ecx, 18h

    ecx ^= eax;                                        //xor     ecx, eax

    ecx &amp;= 0xFF;                                //and     ecx, 0FFh

    esi = IntArray[ecx];                //mov     esi, dword_0_10115D38[ecx*4]

   // ecx = buf;                                //mov     ecx, [esp+4+arg_0]

    eax = eax >> 8;                                //sar     eax, 8

    eax &amp;= 0x00FFFFFF;                        //and     eax, 0FFFFFFh

    eax ^= esi;                                        //xor     eax, esi

    /* int* esi = ecx+edx //??*///lea     esi, [ecx+edx]

    for(int x = 0; x &lt; size; x++)

        {                                                        //eax is the crc, ecx is the current part of the buffer

                int edx = 0;                        //xor     edx, edx

                edx = buf[x] &amp; 0x00FF;        //mov     dl, [ecx]



        /*if(pos > size)                        //cmp     ecx, esi

                return ~eax;                        //jnb     short loc_0_10020803

        */

                                                                //push    edi



//loc_0_100207E0:                         ; CODE XREF: sub_0_10020760+A0j //LOOP

        edx ^= eax;                                //xor     edx, eax

        eax = eax >> 8;                        //sar     eax, 8

        edx &amp;= 0xFF;                        //and     edx, 0FFh

        edi = IntArray[edx];        //mov     edi, dword_0_10115D38[edx*4]

        eax &amp;= 0x00FFFFFF;                //and     eax, 0FFFFFFh

        eax ^= edi;                                //xor     eax, edi

                                                //inc     ecx

                                                                //cmp     ecx, esi

                                                                // jb      short loc_0_100207E0

                                                                //pop     edi

        }



        return ~eax;

}

/*loc_0_10020803:                         ; CODE XREF: sub_0_10020760+7Dj

                not     eax

                pop     esi

                retn

sub_0_10020760  endp



*/

</pre>

I never did finish that before my accident.. I'm not sure it works but that IS the function that is called. I had completely forgot about it until now. Also the forums apparently screwed up my indentation.

Edit: Yes, this will generate a 32 bit number, it is either ah or al of it that is used depending.

Whatever it is, it's not IEEE 802.3 Ethernet. For comparison, here's the standard Ethernet algorithm, which the .S3D files use:

Code:

Unit Ethernet;

// ----------------------------------------------------------------------------------

// This calculates a string's 32-bit CRC value usng the IEEE 802.3 Ethernet standard.

//

// This algorithm can be found at:

//

// http://cell-relay.indiana.edu/cell-relay/publications/software/CRC/32bitCRC.c.html

// ----------------------------------------------------------------------------------



Interface



Type BytePtr = ^Byte;



Function Update_CRC(CRC_Accum: LongWord; P: BytePtr; Size: LongWord): LongWord;

Function GetCRCOfString(St: String): LongWord;



Implementation



Const Polynomial = $04C11DB7;



Var CRC_Table: Array [0..255] Of LongWord;



Procedure Gen_CRC_Table;

Var I,J,CRC_Accum: LongWord;

Begin

  For I := 0 To 255 Do

  Begin

    CRC_Accum := I Shl 24;

    For J := 0 To 7 Do

    Begin

      If (CRC_Accum And $80000000) &lt;> 0

       Then CRC_Accum := (CRC_Accum Shl 1) Xor Polynomial

       Else CRC_Accum := CRC_Accum Shl 1;

    End; // For J

    CRC_Table[I] := CRC_Accum;

  End; // For I

End; // Gen_CRC_Table



Function Update_CRC(CRC_Accum: LongWord; P: BytePtr; Size: LongWord): LongWord;

Var I: LongWord;

Begin

  While Size > 0 Do

  Begin

    I := ((CRC_Accum Shr 24) Xor P^) And $FF;

    Inc(LongInt(P));

    CRC_Accum := (CRC_Accum Shl 8) Xor CRC_Table[I];

    Dec(Size);

  End; // For J

  Result := CRC_Accum;

End; // Update_CRC



Function GetCRCOfString(St: String): LongWord;

Var St1: BytePtr;

Begin

  If St &lt;> '' Then

  Begin

    GetMem(St1,Length(St) + 1);

    FillChar(St1^,Length(St) + 1,0);

    Move(St[1],St1^,Length(St));

    Result := Update_CRC(0,St1,Length(St) + 1);

    FreeMem(St1,Length(St) + 1);

  End

  Else Result := 0;

End; // GetCRCOfString



Initialization

  Gen_CRC_Table;

End.

Quote:

Originally Posted by kathgar

I never did finish that before my accident.. I'm not sure it works but that IS the function that is called. I had completely forgot about it until now. Also the forums apparently screwed up my indentation.

Do you have any idea what that key argument is?

do you think it is the 4 byte thing from the initial packet exchange?
Im just going to try it however I can, but maybe you can share some insight.... messing with checksums is such a pain, since they are practically impossible to debug unless you really understand them.

also there is a reference to arg_0, i am not framiliar with that assembly notation, so im going to assume it is the first argument...

I believe it is the initial key, and then the checksum from the last packet is the new key. I never finished looking into it. There also might be a different key for both client and server. So far i've only looked at the crc function and not much on how it interacted with everything else too much.

NICE! Thanks for myCRC, thats the ticket.

Finally figured it out...

the first two packets in the login process are:

Code:

//client sends to server

struct {

        unsigned short type;

        unsigned long num1;

        unsigned long key1;

        unsigned long num2;

};



//server's reply

struct {

        unsigned short type;

        unsigned long key1;

        unsigned long key2;

        unsigned long num1;

        unsigned short num2;

        unsigned long num3;

};

I dont know if you can put anything in key2, my key2 is stolen from a real packet stream (havent tried anything else)... the client's key1 seems to actually be a function of how many times you have tried to log in, and seems to be always the same on the first login. I know that the key2 used by real servers changes each time even if key1 is the same.

once you send key2 back to the client, that is your keyargument, for all packets, in either direction... I got tied up because I forgot to ntohl the key, damnit.

Code:

   //the len-2 assumes that the checksum bytes are allready appended to the buffer, be it send or receive

unsigned long crc;

        crc = myCRC((unsigned char *) buffer, len-2, ntohl(reply.key2));

        unsigned short cc = htons(crc &amp; 0xFFFF);

// now set the trailing two bytes...

There is one problem, hell if I know what it is... where a specific 2-byte packets is not checksuming properly (00 06)... must be an initilization thing or something... this 2 byte dude also seems to break the rules as far as the 'same checksum in either direction'... they both reply with a strange checksum, but i dont care because #1, i can hard-code that checksum, and #2, they seems to be just 'ping' type packets, no information.

so now with some DNS rewrites, I can mimic the login server and connect to my world (nothing intelligent yet)... but it is now broken when i log into a zone? I THINK this is a known thing.. that the newest client is broken/incompatible with eqemu... is this a true statement? Is there any technical discussion about this problem anywhere?

EQEmu login server always sends key2 with a value of 0. The result of that is the username/pass are sent in the clear, so we don't have to worry about how the user/pass are encrypted. At least that's how pre-June22 works. Something about that might have changed in later patches.

I think key1 is *supposed* to be random, but it looks like somebody forgot to put in a line that seeds the random. That's quality SOE programming for ya :lol:

And as far as entering a zone goes, yeah every few patches it seems the opcodes of the packets change big time. I think the Emu coders have all been pretty much hanging at the pre-Jun22 patch until the login code is figured out, so it's almost certain that Emu zoneserver is not gonna be talking to a newly-patched client correctly.

The first few patches of the new login server were fine once we got it done, but this time they also changed more parts the basic eq protocol that we have to go over. Then there is also the NEW NEW login that is on test. Needless to say there is not a whole lot of work being put into the login that is about to die.

so there is a new login server in testing by eqlive that is going to replace what is there now, as far as the client is going to patch? Or are you referencing the login server -> world server stuff?

AKA.. i am wasting my time? that would be good to know :)

im not framiliar with test EQ... is that as simple as running the test EQ client?

In coming months they will be implementing a SWG-style launchpad for the login process. While I may be over the top in my doom-and-gloom prediction, I suspect that we won't be using that launchpad client for quite some time after it goes live. I'd bet it makes the month we've had our current client locked look like a daytime nap.

If you think that it'd be a great deal of effort to proceed with the current login protocol, then Kathgar might be right in saying don't waste your time. From what you've posted though, it looks like you've got the CRC figured out. And did you try to set Key2 to 0 all the time, does that make the user/pass send in the clear? If so, then your work is 90% done, and if the Emu login server is allowed to use your code, our users might be able to patch up to at least this month's version... to hold them over for what might be months of locking down a client version after that launchpad goes live.

I did not say it was a waste of time, just that there is no sense of urgancy among the developers. The old new login was only on test for a short time before it went live, and I see no reason why they would not do the same with this launchpad. We only have a few people that can/will work on these kinds of updates anyways. At the moment I am devoting more of my time to other efforts, yet still working on the current situation. Even with a new login done however, there is still the change that broke our packetcollector that as far as I know has not been resolved.

ok, with this information, let me state where I am sitting, and maybe you can tell me if my efforts can get me to my goal.

Basically, my goal is to set up a purely local EQ server, no internet required... once this is up and running, I see no reason that I will bother to patch EQ. I also have no resources to get ahold of older versions of EQ.

So assuming that I get this login server coded up and working fine such that I can connect to my world server... is it realistic to think that eqemu will get updated to a state which is compatible with the EQlive client one could install today?
I know it all depends on what really changed recently, so i guess my question is really: are people working on the current probelm, or are most devs just sort of waiting to see what happens before they put a lot of effort into it.

Insight is appriciated.