Windcatcher
06-12-2003, 09:35 AM
I'm so excited I hardly know where to begin.
For seemingly forever now we've wanted to make our own zones, but one major restriction has been that any file you put into an .S3D archive has to have EXACTLY the same name, because no one knew how to set the CRC value to something valid. We've known for ages that a file's CRC, however it's calculated, depends only on its actual name, not on its contents. This is because of two things:
- The same file in different archives (e.g. w1.bmp) has the same CRC, ruling out the archive as a factor
- The fact that we can replace a file's contents and it works rules out its contents and size
Last night I asked myself why, if whoever created this format couldn't trouble himself to use the CRC for it's actual INTENDED PURPOSE--that is, to make sure the data IN the file isn't corrupted, WHY would such a person go to all the trouble of rolling his own CRC algorithm??? I sure wouldn't do that, and I couldn't see any manager willing to have someone spend time and money doing such a thing. As I've stated before, I don't believe in reinventing the wheel.
This morning I resolved to scour the net for every CRC algorithm I could find when I got home, no matter how obscure. CRC algorithms are typically trivial to code, and I could test filenames against known-good CRC values in no time.
The VERY first web search I did pointed me at the Ethernet CRC-32 algorithm. The standard, run-of-the-mill IEEE 802.3 algorithm. It's been around since 1983--TWENTY DAMNED YEARS. It seemed like everything was pointing to it--the search results were so pervasive that I decided to immediately try that one first, since I could code it in no time and it would jump out at anyone looking for one to use.
I broke out the trusty DOS compiler and had a DOS program up and running in a few minutes. I tried a filename...NO GO. But, this was without including the string's null terminator...
I used length + 1 to include the null terminator......EXACT MATCH!!!!!
I tried more filenames......EXACT MATCH!!!!!
Every filename I could find.....MATCHES!
If you look inside .S3D files, you'll quickly see that the names of the files they contain are all lowercase. I've been concerned about case sensitivity, and I hunted for one with uppercase characters, but I couldn't find any. The next best thing was to take a .WLD file, change some letters to uppercase, and see what would happen if I left the CRC the same. I figured that if the client couldn't find the .WLD, it sure as heck wouldn't load.
When I did this, the zone still loaded. This tells me unequivocally that the CRC is based on the filename AFTER it's been converted to lowercase.
So there you have it. It's nothing proprietary, no top-secret SOE code, or anything of the kind. The coder did what I probably would have done in his shoes; he pulled what was probably the first CRC algorithm he saw, implemented it, and moved on. It's killing me to realize that this thing has been standing in our way for so long and the whole time not only was it sitting right under our noses, it's nothing special at all. If the coders at SOE are reading this they must be laughing their butts off.
You can find a sample of the algorithm here:
http://cell-relay.indiana.edu/cell-relay/publications/software/CRC/32bitCRC.c.html
Basically you convert the filename to lowercase and get the CRC value on the string, including the null terminator. The Ethernet standard specifies to use zero as the initial CRC value, and this works. They followed the standard to the letter.
The final test was to take one of my sample OpenZone scenes, switch something to one of my own textures (white_green_marble.bmp) and tell the scene to use that texture instead of one already in the .S3D file (I did this by putting my texture in the textureset folder and switching a scripted object to it). I exported to .WLD and put it in the .S3D file as usual. I then added the .BMP file, calculated the CRC using the standard Ethernet algorithm, and used EQInside to manually type in the CRC. I DIDN"T EVEN BOTHER TO CHANGE THE TEXTURE TO .DDS FORMAT, BUT LEFT IT AS 24-BIT .BMP.
IT LOADED! My texture showed up, too!
Like I said, I'm excited :D :P :lol:
Now that it's just an IEEE standard CRC, I think we need to prevail on the EQInside author to release a version that automaticaly provides the correct CRC. Then, the sky's the limit regarding custom zones and textures.
Wind
For seemingly forever now we've wanted to make our own zones, but one major restriction has been that any file you put into an .S3D archive has to have EXACTLY the same name, because no one knew how to set the CRC value to something valid. We've known for ages that a file's CRC, however it's calculated, depends only on its actual name, not on its contents. This is because of two things:
- The same file in different archives (e.g. w1.bmp) has the same CRC, ruling out the archive as a factor
- The fact that we can replace a file's contents and it works rules out its contents and size
Last night I asked myself why, if whoever created this format couldn't trouble himself to use the CRC for it's actual INTENDED PURPOSE--that is, to make sure the data IN the file isn't corrupted, WHY would such a person go to all the trouble of rolling his own CRC algorithm??? I sure wouldn't do that, and I couldn't see any manager willing to have someone spend time and money doing such a thing. As I've stated before, I don't believe in reinventing the wheel.
This morning I resolved to scour the net for every CRC algorithm I could find when I got home, no matter how obscure. CRC algorithms are typically trivial to code, and I could test filenames against known-good CRC values in no time.
The VERY first web search I did pointed me at the Ethernet CRC-32 algorithm. The standard, run-of-the-mill IEEE 802.3 algorithm. It's been around since 1983--TWENTY DAMNED YEARS. It seemed like everything was pointing to it--the search results were so pervasive that I decided to immediately try that one first, since I could code it in no time and it would jump out at anyone looking for one to use.
I broke out the trusty DOS compiler and had a DOS program up and running in a few minutes. I tried a filename...NO GO. But, this was without including the string's null terminator...
I used length + 1 to include the null terminator......EXACT MATCH!!!!!
I tried more filenames......EXACT MATCH!!!!!
Every filename I could find.....MATCHES!
If you look inside .S3D files, you'll quickly see that the names of the files they contain are all lowercase. I've been concerned about case sensitivity, and I hunted for one with uppercase characters, but I couldn't find any. The next best thing was to take a .WLD file, change some letters to uppercase, and see what would happen if I left the CRC the same. I figured that if the client couldn't find the .WLD, it sure as heck wouldn't load.
When I did this, the zone still loaded. This tells me unequivocally that the CRC is based on the filename AFTER it's been converted to lowercase.
So there you have it. It's nothing proprietary, no top-secret SOE code, or anything of the kind. The coder did what I probably would have done in his shoes; he pulled what was probably the first CRC algorithm he saw, implemented it, and moved on. It's killing me to realize that this thing has been standing in our way for so long and the whole time not only was it sitting right under our noses, it's nothing special at all. If the coders at SOE are reading this they must be laughing their butts off.
You can find a sample of the algorithm here:
http://cell-relay.indiana.edu/cell-relay/publications/software/CRC/32bitCRC.c.html
Basically you convert the filename to lowercase and get the CRC value on the string, including the null terminator. The Ethernet standard specifies to use zero as the initial CRC value, and this works. They followed the standard to the letter.
The final test was to take one of my sample OpenZone scenes, switch something to one of my own textures (white_green_marble.bmp) and tell the scene to use that texture instead of one already in the .S3D file (I did this by putting my texture in the textureset folder and switching a scripted object to it). I exported to .WLD and put it in the .S3D file as usual. I then added the .BMP file, calculated the CRC using the standard Ethernet algorithm, and used EQInside to manually type in the CRC. I DIDN"T EVEN BOTHER TO CHANGE THE TEXTURE TO .DDS FORMAT, BUT LEFT IT AS 24-BIT .BMP.
IT LOADED! My texture showed up, too!
Like I said, I'm excited :D :P :lol:
Now that it's just an IEEE standard CRC, I think we need to prevail on the EQInside author to release a version that automaticaly provides the correct CRC. Then, the sky's the limit regarding custom zones and textures.
Wind