Opensource EQInside

[daeken_bb]

I'm currently working on an opensource version of eqinside. I have it reading all the files (easy) and am about to add the CRC algorithm stuff that WC posted.

Thanks to WC we will have an easy way to modify existing s3d files :)

Release as soon as i figure out the illegal operation that is plaguing me at the moment... perhaps sooner if i can't figure it out and need help :)

[daeken_bb]

Well, i've fixed the illegal operation... now it's just not writing the files to the correct filename... hopefully someone will help me out with this.

Damn attachment BS... having a friend host the files for me instead :P

http://www.slatkisnet.net/cody/OSeqinside.zip

thanks to the slatkisnet admins for hosting this for me :)

[Windcatcher]

You should take a look at my S3D.PAS file in OpenZone 1.3. There are some issues with writing the files:

- In the directory entries near the bottom of the archive, I found that all "canned" files (those that come with EQ) are sorted in increasing CRC order. The CRC for the filename list itself is always fixed at a magic value.

- As far as the data blocks goes, you can write the files in any order you wish, but bear in mind: in every .S3D file I've ever seen they're sorted by lowercase filename, and whatever order you use they have to match the filename list. That's why I have two sort routines in my source...first by filename, and then by CRC.

With my S3D class I could probably whip up a quick EQInside replacement in a couple of days, since the hard work is all done...but it would be in Delphi (i.e., Object Pascal). Is that useful?

Wind

P.S. When writing the data blocks, you *have* to break up each file into 8k blocks before compressing, or the client will refuse to load them.

[daeken_bb]

I have already taken a look at your pascal code for the s3d files, which was _very_ useful although it was the first time i'd ever seen pascal code and tried to understand it ;)

Perhaps we may work together on this? i'd like to be able to convert to zip and back which may be very very useful.


Before i found that openzone was delphi, i was going to add character model importing... but since i can't do pascal/delphi at all, i can't do that in openzone... unless of course you'd like some help in porting openzone to c/c++ :)

[Windcatcher]

I look at C++ like using a chainsaw with the guards taken off...it's powerful, but really easy to cut your arm off :)

I was thinking about some simple steps for you and here's what I came up with:

Before you begin, go to the HackersQuest forums and search for S3D. This is where I found all my information on S3D files. There is a post there that does a terrific job of documenting the format.

My steps for writing an S3D file:
------------------------------------------------------

1. Make an index list of your files, sorted by lowercase filename. I used a numbered list, which should work well for this purpose.

2. For each of your files, break it into 8k blocks and compress each block, but *don't* write them to the archive yet (for each file you should now have a list of compressed blocks).

3. Create another block that consists of the filenames in the archive, sorted by lowercase filename (see step 1). The first long in the block is the number of filenames, then for each filename there is a long containing its length (*including* null terminator) and the filename data itself.

4. Just as you did for the files, break up the filename data block into 8k chunks and compress the chunks (usually it's just one block, but do it right and assume it can be more).

5. Now it's time to make the directory entries. Stepping through your filename data in order of lowercase filename, create a directory entry for each file and assign its filename position, starting at sizeof(header) and counting up. The reason for all of this is because the order of the filenames in the filename data block *must* match the order in which the filename data is actually written to disk. In every .S3D file I've seen, they're written ordered by lowercase filename. The order of directory entries at the end of the archive doesn't have to be the same since they contain the actual file positions of the file data.

6. Now create one more directory entry for the filename data block. The filename data blocks *always* come after the other file data blocks. The directory entry has a fixed CRC value. You can find it in S3D.PAS.

7. Now it's time to write to disk:

7a. Write the header.

7b. Stepping through your files in order of lowercase filename, write the data blocks for each file. Remember that each block has to have a block header with it.

7c. Write the data blocks for the filename data. 99 times out of 100 it's just one block, but you should do it right and handle it just like any other file.

7d. Write a long that says how many directory entries there are.

7e. Write the directory entries, in order of increasing CRC value. Remember that there is a directory entry for the filename data block(s).

7f. Write the footer. Note that, if you're writing an .S3D that was previously read, that not all .S3D files have a footer. A reader shouldn't fail if it doesn't find it and in that case it shouldn't write one, since it doesn't know what to put in the date field (this is something I have to do in the next version of OpenZone). If you're creating one from scratch, I find that putting anything in there works just fine, but otherwise you need to preserve what was already there. The docs on Hackersquest describe it as a possible CRC, but I've convinced myself that is has absolutely nothing to do with the S3D name or its contents. I believe it's a pseudo-datestamp that the patcher uses to quickly determine a file's version.

[daeken_bb]

I have most of the s3d creation stuff done now (yay!)

however, the crc stuff is really getting on my nerves :P

would you be able to post functional C/c++ code for it?

thanks,
Dae

[Windcatcher]

Here's the example I used...

http://cell-relay.indiana.edu/cell-relay/publications/software/CRC/32bitCRC.c.html

Be careful of his formatting with respect to the curly braces...IMHO whoever decided that tabs were legal whitespace should be shot!

Wind

[daeken_bb]

Thanks again!

I think i have the CRC stuff right now... i was doing sizeof(filenames[i]) ... which i think was causing it to not pick up the correct info... so i have to go test now :)

wish me luck!

[daeken_bb]

Damn, no i didn't... i already changed that before...


data->crc = update_crc(0, filename, strlen(filename) + 1);


See anything wrong with that? filename is a const char*.

[daeken_bb]

Well, i finally got the crc stuff right (yay!)

however, i figured out now that i have the filenames not matching up with the correct files...

Aren't the directory entries in the order given in the filenames chunk?

Thanks in advance for the help again ;)

[Windcatcher]

No. First you need to make the file data blocks, then order those and the filename list by lowercase filename. Then form the directory entries and put in the file locations of the data blocks. Once you do this the order of the directory entries doesn't have to be the same as for the data blocks, since the absolute file position is known. The directory entries should be written ordered by CRC instead.

WC

[daeken_bb]

Thanks again... i think i understand now :)