|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Development::Development Forum for development topics and for those interested in EQEMu development. (Not a support forum) |
|
|
|
03-16-2011, 06:25 PM
|
Fire Beetle
|
|
Join Date: Apr 2007
Posts: 16
|
|
How to generate the CRC used by .s3d files?
I am working on a C++ class to be used for modifying .s3d files using the Pascal source code included in S3DSpy as a reference. The only problem I've encountered is trying to decipher how to generate the CRC used by the format.
Here is an example file:
http://3dfolio.com/files/bafch0002.bmp
The CRC provided by the file is: 3222067071.
I have not been able to duplicate this CRC. It "should" be IEEE 802.3 CRC-32, which is the same CRC used by the crc32.cpp included in the eqemu source.
I can confirm that the CRC generated by crc32.cpp is accurate. But it is not the same as what is provided by the .s3d file.
According to s3d.pas, the CRC is being generated from the file name??
Code:
Procedure SetFileName(Index: Integer; St: String);
Begin
If (Index >= 0) And (Index <= High(FFileNames)) Then
Begin
St := Trim(St);
FFileNames[Index] := St;
FDirectory[Index].CRC := GetCRCOfString(LowerCase(St));
End;
End; // TS3DFile.SetFileName
This makes no sense. And the CRC generated this way also doesn't match.
One other reference I found suggests that the CRC used by EQ is "IEEE 802.3 Ethernet CRC-32", which I think is the same thing. Can anyone confirm this?
I can't seem to figure out how to generate this CRC. Does any one know?
|
|
|
|
03-16-2011, 07:04 PM
|
Demi-God
|
|
Join Date: Aug 2010
Posts: 1,743
|
|
|
|
|
|
03-16-2011, 07:06 PM
|
Developer
|
|
Join Date: Feb 2004
Location: UK
Posts: 1,540
|
|
This python code will build an .eqg (same structure as an .s3d):
packeqg.py
Code:
import struct, posixfile, socket, zlib, pdb, sys, socket, os, operator, string
CRCTable = []
def CompareString(is1, is2):
s1 = is1[0]
s2 = is2[0]
print s1, s2
l1 = len(s1)
l2 = len(s2)
l = l1
if(l2 < l):
l = l2
for a in range(0, l):
if((s1[a] in string.ascii_letters) and (s2[a] == '_')):
print "%s is less than %s" % (s1, s2)
return -1
if((s2[a] in string.ascii_letters) and (s1[a] == '_')):
print "%s is less than %s" % (s2, s1)
return 1
if(ord(s1[a]) < ord(s2[a])):
return -1
if(ord(s1[a]) > ord(s2[a])):
return 1
if l1 < l2:
return -1
if l2 > l1:
return 1
return 0
def GenCRCTable():
Polynomial = 0x04C11DB7
for i in range(0, 256):
CRC_Accum = i << 24
for j in range(0, 8):
if((CRC_Accum & 0x80000000) !=0):
CRC_Accum = (CRC_Accum << 1) ^ Polynomial
else:
CRC_Accum = (CRC_Accum << 1)
#print "%10X" % CRC_Accum
CRCTable.append(CRC_Accum)
def CalcCRC(s):
size = len(s)
CRC_Accum = 0
j = 0
while size > 0:
i = ((CRC_Accum >> 24) ^ ord(s[j])) & 0xff
j = j + 1
CRC_Accum = (CRC_Accum << 8) ^ CRCTable[i]
CRC_Accum = CRC_Accum & 0xffffffff
size = size - 1
return CRC_Accum
class PFSFileEntry:
def __init__(self, file, offset):
self.file = file
self.offset = offset
FILTER=''.join([(len(repr(chr(x)))==3) and chr(x) or '.' for x in range(256)])
def dump(src, offset, length=8):
N=offset; result=''
while src:
s,src = src[:length],src[length:]
hexa = ' '.join(["%02X"%ord(x) for x in s])
s = s.translate(FILTER)
result += "%04X %-*s %s\n" % (N, length*3, hexa, s)
N+=length
return result
def pname(buffer, offset):
strlen = 0
while buffer[offset+strlen] != chr(0):
strlen = strlen + 1
fmt = str(strlen) + 's'
strvar = struct.unpack(fmt,buffer[offset:offset+strlen])
return strvar[0]
filenames = []
files = []
modelname = []
GenCRCTable()
directory = sys.argv[2]
fnames = os.listdir(directory)
for f in fnames:
filedetails = [f, CalcCRC(f + chr(0)), 0, 0]
files.append(filedetails)
#for f in files:
# print "File: %-40s, CRC: %8X" % (f[0], f[1])
#sortedfiles = sorted(files, key=operator.itemgetter(1))
sortedfiles = sorted(files, cmp=CompareString)
outputfile = open(sys.argv[1] + ".eqg", "wb")
#ps = struct.pack("L", 10)
#outputfile.write(ps)
#outputfile.close()
outputfile.seek(12, 0)
for f in sortedfiles:
print "File: %-40s, CRC: %8X, Offset: %8X" % (f[0], f[1], outputfile.tell())
f[2] = outputfile.tell()
inf = open(directory + "/" + f[0], "rb")
uncompressed = inf.read()
inf.close()
print "Uncompressed length is %10i" % len(uncompressed)
f[3] = len(uncompressed)
NumBlocks = len(uncompressed) / 8192 + 1
print "Need %i 8K Blocks" % NumBlocks
print "Compressing:"
TotalCompressedLength = 0
for i in range(0, len(uncompressed), 8192):
BlockStart = i
BlockEnd = i + 8192
if(BlockEnd > len(uncompressed)):
BlockEnd = len(uncompressed)
CompressedBlock = zlib.compress(uncompressed[BlockStart:BlockEnd])
print " Uncompressed: %8i Compressed Block is %8i bytes" % (BlockEnd - BlockStart, len(CompressedBlock))
TotalCompressedLength = TotalCompressedLength + len(CompressedBlock)
ps = struct.pack("LL", len(CompressedBlock), BlockEnd - BlockStart)
outputfile.write(ps)
outputfile.write(CompressedBlock)
print "Compressed size is %8i" % TotalCompressedLength
print "File pointer is now at %8X" % outputfile.tell()
uncompressed = ""
# Build Filename file
print "Number of files is %8X" % len(files)
ps = struct.pack("L", len(files))
uncompressed = uncompressed + ps
for f in sortedfiles:
ps = struct.pack("L", len(f[0]) + 1)
uncompressed = uncompressed + ps + f[0] + chr(0)
print "Uncompressed filename file is %8i" % len(uncompressed)
for a in range(0, len(uncompressed), 32):
print dump(uncompressed[a:a+32], a, 32).strip('\n')
NumBlocks = len(uncompressed) / 8192 + 1
print "Need %i 8K Blocks" % NumBlocks
files.append(["", 0x61580AC9, outputfile.tell(), len(uncompressed)])
print "Compressing:"
TotalCompressedLength = 0
for i in range(0, len(uncompressed), 8192):
BlockStart = i
BlockEnd = i + 8192
if(BlockEnd > len(uncompressed)):
BlockEnd = len(uncompressed)
CompressedBlock = zlib.compress(uncompressed[BlockStart:BlockEnd])
print " Uncompressed: %8i Compressed Block is %8i bytes" % (BlockEnd - BlockStart, len(CompressedBlock))
TotalCompressedLength = TotalCompressedLength + len(CompressedBlock)
ps = struct.pack("LL", len(CompressedBlock), BlockEnd - BlockStart)
outputfile.write(ps)
outputfile.write(CompressedBlock)
print "Compressed size is %8i" % TotalCompressedLength
print "File pointer is now at %8X" % outputfile.tell()
FileTablePos = outputfile.tell()
sortedfiles = sorted(files, key=operator.itemgetter(1))
ps = struct.pack("L", len(sortedfiles))
outputfile.write(ps)
for f in sortedfiles:
ps = struct.pack("LLL", f[1], f[2], f[3])
print "Writing entry for file %-40s, CRC: %8X, Offset: %8X, Size %10i" % (f[0], f[1], f[2], f[3])
outputfile.write(ps)
ps = struct.pack("LccccL", FileTablePos, 'P', 'F', 'S', ' ', 0x00020000)
print "Header size is %i bytes" % len(ps)
print dump(ps, 0, 12)
outputfile.seek(0,0)
outputfile.write(ps)
outputfile.close()
Usage is: python packeqg.py myeqg unpacked
Where myeqg is the name of the eqg you are creating (the .eqg extension will be added automatically) and unpacked is a directory containing all the files you want to put in the .eqg
I downloaded the .bmp you linked and ran my python script as so:
Code:
python packeqg.py myeqg unpacked|grep "File: bafch0002.bmp"
And the output was:
Code:
File: bafch0002.bmp , CRC: C00CD77F, Offset: 75366
Hex C00CD77F is decimal 3222067071, so the CRC code in there is correct. Can't remember where I got it from, but most likely Windcatcher's Openzone code.
I know that Python code is good for generating .eqg files as I successfully loaded files generated by it in-game while working on the EQGv4 format.
|
|
|
|
03-16-2011, 08:27 PM
|
Fire Beetle
|
|
Join Date: Apr 2007
Posts: 16
|
|
Yes, the CRC algorithm is indeed different. I've got it working now based on the information from Windcatcher's post.
I did try to do a search but "CRC" was too short and I didn't come up with the correct term to locate his post. It would have saved me a lot of grief!
Thanks a lot for your help.
|
04-04-2011, 03:23 AM
|
Fire Beetle
|
|
Join Date: Apr 2007
Posts: 16
|
|
Was a use for the s3d file constant 131072, preceding 4 bytes of the directory list, or the 4 unknown bytes associated with a file from the directory list ever found?
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
All times are GMT -4. The time now is 05:54 PM.
|
|
|
|
|
|
|
|
|
|
|
|
|