.S3D CRC algorithm...FOUND

[Windcatcher]

I'm so excited I hardly know where to begin.



For seemingly forever now we've wanted to make our own zones, but one major restriction has been that any file you put into an .S3D archive has to have EXACTLY the same name, because no one knew how to set the CRC value to something valid. We've known for ages that a file's CRC, however it's calculated, depends only on its actual name, not on its contents. This is because of two things:

- The same file in different archives (e.g. w1.bmp) has the same CRC, ruling out the archive as a factor
- The fact that we can replace a file's contents and it works rules out its contents and size

Last night I asked myself why, if whoever created this format couldn't trouble himself to use the CRC for it's actual INTENDED PURPOSE--that is, to make sure the data IN the file isn't corrupted, WHY would such a person go to all the trouble of rolling his own CRC algorithm??? I sure wouldn't do that, and I couldn't see any manager willing to have someone spend time and money doing such a thing. As I've stated before, I don't believe in reinventing the wheel.

This morning I resolved to scour the net for every CRC algorithm I could find when I got home, no matter how obscure. CRC algorithms are typically trivial to code, and I could test filenames against known-good CRC values in no time.

The VERY first web search I did pointed me at the Ethernet CRC-32 algorithm. The standard, run-of-the-mill IEEE 802.3 algorithm. It's been around since 1983--TWENTY DAMNED YEARS. It seemed like everything was pointing to it--the search results were so pervasive that I decided to immediately try that one first, since I could code it in no time and it would jump out at anyone looking for one to use.

I broke out the trusty DOS compiler and had a DOS program up and running in a few minutes. I tried a filename...NO GO. But, this was without including the string's null terminator...

I used length + 1 to include the null terminator......EXACT MATCH!!!!!
I tried more filenames......EXACT MATCH!!!!!
Every filename I could find.....MATCHES!

If you look inside .S3D files, you'll quickly see that the names of the files they contain are all lowercase. I've been concerned about case sensitivity, and I hunted for one with uppercase characters, but I couldn't find any. The next best thing was to take a .WLD file, change some letters to uppercase, and see what would happen if I left the CRC the same. I figured that if the client couldn't find the .WLD, it sure as heck wouldn't load.

When I did this, the zone still loaded. This tells me unequivocally that the CRC is based on the filename AFTER it's been converted to lowercase.

So there you have it. It's nothing proprietary, no top-secret SOE code, or anything of the kind. The coder did what I probably would have done in his shoes; he pulled what was probably the first CRC algorithm he saw, implemented it, and moved on. It's killing me to realize that this thing has been standing in our way for so long and the whole time not only was it sitting right under our noses, it's nothing special at all. If the coders at SOE are reading this they must be laughing their butts off.

You can find a sample of the algorithm here:

http://cell-relay.indiana.edu/cell-r...2bitCRC.c.html

Basically you convert the filename to lowercase and get the CRC value on the string, including the null terminator. The Ethernet standard specifies to use zero as the initial CRC value, and this works. They followed the standard to the letter.

The final test was to take one of my sample OpenZone scenes, switch something to one of my own textures (white_green_marble.bmp) and tell the scene to use that texture instead of one already in the .S3D file (I did this by putting my texture in the textureset folder and switching a scripted object to it). I exported to .WLD and put it in the .S3D file as usual. I then added the .BMP file, calculated the CRC using the standard Ethernet algorithm, and used EQInside to manually type in the CRC. I DIDN"T EVEN BOTHER TO CHANGE THE TEXTURE TO .DDS FORMAT, BUT LEFT IT AS 24-BIT .BMP.

IT LOADED! My texture showed up, too!

Like I said, I'm excited :P :lol:

Now that it's just an IEEE standard CRC, I think we need to prevail on the EQInside author to release a version that automaticaly provides the correct CRC. Then, the sky's the limit regarding custom zones and textures.

Wind

[Merth]

Great job! This is really cool to hear, and should open up a lot of interesting possibilities!

[Jay]

Excellent work Wind. Thanks to you it's becoming easier and easier to create custom zones.

[Bardboy]

Great work, Windcatcher! /cheer

[kellexor]

It is posts like these that can destroy all my weekend plans heh, looks like I will be a zone texturing fool for hours and hours....

[a_Guest03]

I'm going to piss myself, WC! Huzzah! New zones!

[DeletedUser]

fuck dude, EverQuest 2fort is gonna come out soon. roofles

Isn't EQInside opensource? So fixing it up to the crc shouldn't be that hard?

[a_Guest03]

hehe EQ 2fort - I loved that map for TFC. That would be great to have a war like that. Wizards and mages in the back with the archers, with warriors, clerics, druids storming the gates.

[DeletedUser]

Can't release the zone files unless you want to get fucked majorly :P

I'm working on a new EQInside.. its called.. EQInside 2!

There should be a site dedicated to EQ Zone creation, can post all the different tools.. maybe make a tool that allows helpless people easily add the .wld files and textures into a .s3d so they can use the zone too.

Windcatcher, you should make the Openzone program like worldcraft, entities and stuff. :P

[a_Guest03]

How would homemade zone files be illegal? If we made them, they're ours. Just because they work with Everquest doesn't mean we broke the law by stealing their copyrights. Make a few from scratch and see what they say.

On a negative note, they may have patented the format (why the hell would they, but people patent everything). If it's patented, it wouldn't be okay to distribute because we would violate the patent.

I think distributing zone files of a homebrew style would be completely legal, depending on patents filed.

[DeletedUser]

you can't distribute .s3d files unless you made all the files in it yourself.
If you make all of the textures and the .wld then it'd be fine.

Otherwise all the shit made by verant = copyrighted.

[Jay]

You have to replace lights.wld and objects.wld too as well as all textures involved in the zone. I believe Windcatcher has already figured out how to do this, so I do not presume it to be a problem.

[a_Guest03]

I meant to say COMPLETELY homebrewed zones, top to bottom, should be legal to distribute unless Verant or Sony patented the format. If they did, you could still distribute it as a different format (wld) and distribute a tool to convert it (to .s3d, Verant style).

[Windcatcher]

Let me preface this with some information: files like ecommons.s3d, lavastorm.s3d, etc. (that is, the <zone>.s3d files, which are the ones OpenZone deals with) contain three .WLD files:

<zone>.wld (e.g. ecommons.wld)
lights.wld
objects.wld

Everything else is a texture. You can use .DDS-format files if you need irregular shapes (that is, transparent portions) or I've found that standard 24-bit .BMP files work if you just want standard textures.

Last night I had a successful test using totally empty objects.wld and lights.wld files in my ecommons.s3d file (by totally empty I mean they only contain their names and an empty "intro" packet, which all .WLD files must have). OpenZone 1.3 will have menu entries availble that make these files for you. Since they have the same name in every zone, you'll only need to make them once. This also has the benefit of getting rid of any leftover placed objects from the zone you're overwriting (trees, torches, braziers, and such). The downside is that you can't place any such objects, since all I know about objects.wld is how to make an empty one--I otherwise don't understand the format.

I'm working on a full-blown class for working with .S3D files. When it's done OpenZone will be able to take all the textures you're using, create the three .WLD files from scratch, and build a single .S3D file for you. It will also warn you that you CANNOT distribute any .S3D files you make unless you have permission to distribute ALL of the textures within. This might be a good time to go buy TextureMaker. :P

Each zone uses three .S3D files: <zone>.S3D, <zone>_chr.s3d, and <zone>_obj.s3d. OpenZone only deals with the first one, <zone>.s3d, such as ecommons.s3d. If you use all your own textures you'll be able to make that file and distribute it as-is. You will NOT, however, be able to distribute the other two files, <zone>_obj.s3d and <zone>_chr.s3d. Remember.

Since we can now put any file we like into an .S3D archive, I'm going to relax the textureset requirements a bit. OpenZone will still require and support texturesets (that is, textures in specific zone folders), but it will also support "common" textures in the \textures folder (above the textureset folders). You'll have to choose a textureset before exporting to .S3D, but you won't have to have anything in your textureset folders if you don't want to. For now I'm considering placing created .S3D files in the \zones subfolder.

WC

P.S. You CANNOT patent a file format. You can't copyright it either. Period, end of story. I'm sure the folks at Microsoft would have LOVED to patent the .DOC format, but the law won't allow it.

[mwmdragon]

I have started the web site to help all aspects of world building for EQEmu. Please let me know if you guys would like to use my site for all things World Building and I'll set it all up ready for the new custom zones and such.