How to generate the CRC used by .s3d files?

[Ropethunder]

I am working on a C++ class to be used for modifying .s3d files using the Pascal source code included in S3DSpy as a reference. The only problem I've encountered is trying to decipher how to generate the CRC used by the format.

Here is an example file:

http://3dfolio.com/files/bafch0002.bmp

The CRC provided by the file is: 3222067071.

I have not been able to duplicate this CRC. It "should" be IEEE 802.3 CRC-32, which is the same CRC used by the crc32.cpp included in the eqemu source.

I can confirm that the CRC generated by crc32.cpp is accurate. But it is not the same as what is provided by the .s3d file.

According to s3d.pas, the CRC is being generated from the file name??

Procedure SetFileName(Index: Integer; St: String);
Begin
  If (Index >= 0) And (Index <= High(FFileNames)) Then
  Begin
    St                    := Trim(St);
    FFileNames[Index]     := St;
    FDirectory[Index].CRC := GetCRCOfString(LowerCase(St));
  End;
End; // TS3DFile.SetFileName

This makes no sense. And the CRC generated this way also doesn't match.

One other reference I found suggests that the CRC used by EQ is "IEEE 802.3 Ethernet CRC-32", which I think is the same thing. Can anyone confirm this?

I can't seem to figure out how to generate this CRC. Does any one know?

[lerxst2112]

This seems to cover it.

http://www.eqemulator.org/forums/showthread.php?t=7942

[Derision]

This python code will build an .eqg (same structure as an .s3d):

packeqg.py

import struct, posixfile, socket, zlib, pdb, sys, socket, os, operator, string


CRCTable = []
def CompareString(is1, is2):

        s1 = is1[0]
        s2 = is2[0]

        print s1, s2

        l1 = len(s1)
        l2 = len(s2)

        l = l1
        if(l2 < l):
                l = l2

        for a in range(0, l):

                if((s1[a] in string.ascii_letters) and (s2[a] == '_')):
                        print "%s is less than %s" % (s1, s2)
                        return -1

                if((s2[a] in string.ascii_letters) and (s1[a] == '_')):
                        print "%s is less than %s" % (s2, s1)
                        return 1

                if(ord(s1[a]) < ord(s2[a])):
                        return -1

                if(ord(s1[a]) > ord(s2[a])):
                        return 1

        if l1 < l2:
                return -1

        if l2 > l1:
                return 1

        return 0

def GenCRCTable():
        Polynomial = 0x04C11DB7

        for i in range(0, 256):
                CRC_Accum = i << 24
                for j in range(0, 8):
                        if((CRC_Accum & 0x80000000) !=0):
                                CRC_Accum = (CRC_Accum << 1) ^ Polynomial
                        else:
                                CRC_Accum = (CRC_Accum << 1)

                #print "%10X" % CRC_Accum
                CRCTable.append(CRC_Accum)

def CalcCRC(s):
        size = len(s)

        CRC_Accum = 0
        j = 0

        while size > 0:
                i = ((CRC_Accum >> 24) ^ ord(s[j])) & 0xff
                j = j + 1
                CRC_Accum = (CRC_Accum << 8) ^ CRCTable[i]
                CRC_Accum = CRC_Accum & 0xffffffff
                size = size - 1

        return CRC_Accum

class PFSFileEntry:

    def __init__(self, file, offset):
        self.file = file
        self.offset = offset

FILTER=''.join([(len(repr(chr(x)))==3) and chr(x) or '.' for x in range(256)])


def dump(src, offset, length=8):
    N=offset; result=''
    while src:
        s,src = src[:length],src[length:]
        hexa = ' '.join(["%02X"%ord(x) for x in s])
        s = s.translate(FILTER)
        result += "%04X   %-*s   %s\n" % (N, length*3, hexa, s)
        N+=length
        return result

def pname(buffer, offset):
    strlen = 0
    while buffer[offset+strlen] != chr(0):
        strlen = strlen + 1
    fmt = str(strlen) + 's'
    strvar = struct.unpack(fmt,buffer[offset:offset+strlen])
    return strvar[0]

filenames = []
files = []
modelname = []

GenCRCTable()

directory = sys.argv[2]

fnames = os.listdir(directory)

for f in fnames:
        filedetails = [f, CalcCRC(f + chr(0)), 0, 0]
        files.append(filedetails)

#for f in files:
#       print "File: %-40s, CRC: %8X" % (f[0], f[1])

#sortedfiles = sorted(files, key=operator.itemgetter(1))

sortedfiles = sorted(files, cmp=CompareString)
outputfile = open(sys.argv[1] + ".eqg", "wb")
#ps = struct.pack("L", 10)
#outputfile.write(ps)
#outputfile.close()
outputfile.seek(12, 0)


for f in sortedfiles:
        print "File: %-40s, CRC: %8X, Offset: %8X" % (f[0], f[1], outputfile.tell())
        f[2] = outputfile.tell()

        inf = open(directory + "/" + f[0], "rb")
        uncompressed = inf.read()
        inf.close()
        print "Uncompressed length is %10i" % len(uncompressed)

        f[3] = len(uncompressed)

        NumBlocks = len(uncompressed) / 8192 + 1
        print "Need %i 8K Blocks" % NumBlocks

        print "Compressing:"

        TotalCompressedLength = 0

        for i in range(0, len(uncompressed), 8192):
                BlockStart = i
                BlockEnd = i + 8192
                if(BlockEnd > len(uncompressed)):
                        BlockEnd = len(uncompressed)

                CompressedBlock = zlib.compress(uncompressed[BlockStart:BlockEnd])

                print "  Uncompressed: %8i Compressed Block is %8i bytes" % (BlockEnd - BlockStart, len(CompressedBlock))
                TotalCompressedLength = TotalCompressedLength + len(CompressedBlock)

                ps = struct.pack("LL", len(CompressedBlock), BlockEnd - BlockStart)
                outputfile.write(ps)
                outputfile.write(CompressedBlock)

        print "Compressed size is %8i" % TotalCompressedLength

print "File pointer is now at %8X" % outputfile.tell()

uncompressed = ""
# Build Filename file
print "Number of files is %8X" % len(files)
ps = struct.pack("L", len(files))

uncompressed = uncompressed + ps

for f in sortedfiles:
        ps = struct.pack("L", len(f[0]) + 1)
        uncompressed = uncompressed + ps + f[0] + chr(0)

print "Uncompressed filename file is %8i" % len(uncompressed)

for a in range(0, len(uncompressed), 32):
        print dump(uncompressed[a:a+32], a, 32).strip('\n')

NumBlocks = len(uncompressed) / 8192 + 1
print "Need %i 8K Blocks" % NumBlocks

files.append(["", 0x61580AC9, outputfile.tell(), len(uncompressed)])

print "Compressing:"

TotalCompressedLength = 0

for i in range(0, len(uncompressed), 8192):
        BlockStart = i
        BlockEnd = i + 8192
        if(BlockEnd > len(uncompressed)):
                BlockEnd = len(uncompressed)

        CompressedBlock = zlib.compress(uncompressed[BlockStart:BlockEnd])

        print "  Uncompressed: %8i Compressed Block is %8i bytes" % (BlockEnd - BlockStart, len(CompressedBlock))
        TotalCompressedLength = TotalCompressedLength + len(CompressedBlock)

        ps = struct.pack("LL", len(CompressedBlock), BlockEnd - BlockStart)
        outputfile.write(ps)
        outputfile.write(CompressedBlock)

        print "Compressed size is %8i" % TotalCompressedLength

print "File pointer is now at %8X" % outputfile.tell()

FileTablePos = outputfile.tell()

sortedfiles = sorted(files, key=operator.itemgetter(1))
ps = struct.pack("L", len(sortedfiles))
outputfile.write(ps)

for f in sortedfiles:
        ps = struct.pack("LLL", f[1], f[2], f[3])
        print "Writing entry for file %-40s, CRC: %8X, Offset: %8X, Size %10i" % (f[0], f[1], f[2], f[3])
        outputfile.write(ps)

ps = struct.pack("LccccL", FileTablePos, 'P', 'F', 'S', ' ', 0x00020000)

print "Header size is %i bytes" % len(ps)
print dump(ps, 0, 12)
outputfile.seek(0,0)
outputfile.write(ps)

outputfile.close()

Usage is: python packeqg.py myeqg unpacked

Where myeqg is the name of the eqg you are creating (the .eqg extension will be added automatically) and unpacked is a directory containing all the files you want to put in the .eqg

I downloaded the .bmp you linked and ran my python script as so:

python packeqg.py myeqg unpacked|grep "File: bafch0002.bmp"

And the output was:

File: bafch0002.bmp                           , CRC: C00CD77F, Offset:    75366

Hex C00CD77F is decimal 3222067071, so the CRC code in there is correct. Can't remember where I got it from, but most likely Windcatcher's Openzone code.

I know that Python code is good for generating .eqg files as I successfully loaded files generated by it in-game while working on the EQGv4 format.

[Ropethunder]

Yes, the CRC algorithm is indeed different. I've got it working now based on the information from Windcatcher's post.

I did try to do a search but "CRC" was too short and I didn't come up with the correct term to locate his post. It would have saved me a lot of grief!

Thanks a lot for your help.

[Ropethunder]

Was a use for the s3d file constant 131072, preceding 4 bytes of the directory list, or the 4 unknown bytes associated with a file from the directory list ever found?