New Character Model File Formats

[daeken_bb]

Well, I've begun to reverse-engineer the new (packaged in .eqg files) character models. I've not done anything concrete yet, but I pulled the files out of bat.eqg and started poking around, and I mostly understand what I'm seeing so far.

So far my findings are as such:

.MOD files are just like with placeable objects, except there's a bone index (named, which makes things a ton easier) after the material list.
.ANI files have a bone index then what appears to be a series of rotations, translations, etc.
.PTS and .PRT seem to tell the points where things like spells occur (for the particle engine to know where to center on), as well as where damage should be taken (not quite sure on this, but it says it in plaintext lol)

That seems to be all the new file formats (well, .MOD isn't new, but it's different than placeable objects).

I'll work on getting these rendering tomorrow. I'll probably whip up a quick python viewer for them, and hopefully even get animations working :)

I go back to school on wednesday, so I need to get a lot done tomorrow hehe

Happy Hacking,
Lord Daeken M. BlackBlade
(Cody Brocious)

[KhaN]

ANI are animation reference used to animate model (MOD).

[jbb]

work on getting these rendering tomorrow. I'll probably whip up a quick python viewer for them, and hopefully even get animations working
Nice :)
I'd like to see that

[daeken_bb]

Ok, time to post an update.

WindCatcher and I have been working to figure out the bone formats for models, and I just found the last portion of it :)

We now have all of the .mod file format done, and we're ready to move onto the .ani format :)

[daeken_bb]

Actually, small correction. There's a section we don't know what it is yet (I presume skinning data) but it looks to be useless to us right now. I'll take a crack at reverse-engineering it, but I think I'll have to wait for WC to get back.

[Windcatcher]

Get back? I've had my irc client open for an hour waiting for you to return :P

[daeken_bb]

Some screenshots :)

http://home.archshadow.com/~daeken/skeleton_wings.jpg
http://home.archshadow.com/~daeken/skeleton_wings2.jpg

The entire .mod format (except for the section right before vertices that's in all 3 variations of the .ter format (ter, mod for placeable objects, and mod for characters)) is now fully known, so I (with WC's assistance hopefully :) ) will start on .ani after school tomorrow.

Now it's time for bed... g'night all :)

[jbb]

nice :)

[jbb]

I don't really undertand a great deal about how animated models would be stored (in general, not in EQ specifically) and "bones" etc.

Presumably there is a single mesh for the model but it can be "deformed" in various ways to move?

[Mongrel]

Each bone has a set of (triangles, points, whatever) assigned to it. The animation "frames" only describe the movement and rotation of the bone.
Bones are also usually set up in a hierarchical way. That means, if you rotate the upper arm bone, the rest of the arm and the hand get rotated as well (easily doable in OpenGL with a matrix stack (PushMatrix / PopMatrix), DX probably has similar mechanics).

[daeken_bb]

Ok, well, I've just finished the bone loader for character models finally.

Here's the problem. I have a list of polygons that has 3 indexes into a list of vertices, corresponding to the 3 points of the triangles that are being rendered. Well, bones are linked to vertices rather than triangles, so I can't figure out how I should be matching polygons to bones so I can use GL's matrix functions to do all of the animation math. I'm not sure if there are any polygons that don't have all of their vertices connected to a single bone, but it wouldn't suprise me. I'll do some testing in a bit I guess.

Anyway, the bone tree structure is finished, and you can very obviously see the structure. Here's a dump of the bone structure for a bat. The first name is the name of the current bone. It's followed by up to 2 bone names, 1 with a - in front of it, 1 with a +. This allows you to see the bone connections.

ROOT_BONE + PELV
PELV + PELV_TAIL01
CHEST_CHEST01 + LEGR_THIGH
CHEST_CHEST02 + CHEST_CHEST03
CHEST_CHEST03 + CHEST03_TORCH01
HEAD_NECK + ARMR_CLAV
HEAD_HEAD + HEAD_CAM02
HEAD_BROWL01
HEAD_BROWR01 - HEAD_BROWL01
HEAD_EARL01 - HEAD_BROWR01 + HEAD_EARL02
HEAD_EARL02 + HEAD_EARL03
HEAD_EARL03
HEAD_EARR01 - HEAD_EARL01 + HEAD_EARR02
HEAD_EARR02 + HEAD_EARR03
HEAD_EARR03
HEAD_JAW - HEAD_EARR01
HEAD_LIDLL - HEAD_JAW
HEAD_LIDLU - HEAD_LIDLL
HEAD_LIDRL - HEAD_LIDLU
HEAD_LIDRU - HEAD_LIDRL
HEAD_NAME - HEAD_LIDRU
HEAD_CAM01 - HEAD_NAME
HEAD_CAM02 - HEAD_CAM01
ARML_CLAV - HEAD_HEAD + ARML_BCEP
ARML_BCEP + ARML_FARM
ARML_FARM + ARML_SHLD
ARML_HAND + ARML_WEAP
ARML_INDEX01 + ARML_INDEX02
ARML_INDEX02 + ARML_INDEX03
ARML_INDEX03
ARML_PINK01 - ARML_INDEX01 + ARML_PINK02
ARML_PINK02 + ARML_PINK03
ARML_PINK03
ARML_POINT01 - ARML_PINK01 + ARML_POINT02
ARML_POINT02 + ARML_POINT03
ARML_POINT03
ARML_RING01 - ARML_POINT01 + ARML_RING02
ARML_RING02 + ARML_RING03
ARML_RING03
ARML_THMB01 - ARML_RING01 + ARML_THMB02
ARML_THMB02 + ARML_THMB03
ARML_THMB03
ARML_WEAP - ARML_THMB01
ARML_WING01 - ARML_HAND
ARML_SHLD - ARML_WING01
ARMR_CLAV - ARML_CLAV + ARMR_BCEP
ARMR_BCEP + ARMR_FARM
ARMR_FARM + ARMR_WING01
ARMR_HAND + ARMR_WEAP
ARMR_INDEX01 + ARMR_INDEX02
ARMR_INDEX02 + ARMR_INDEX03
ARMR_INDEX03
ARMR_PINK01 - ARMR_INDEX01 + ARMR_PINK02
ARMR_PINK02 + ARMR_PINK03
ARMR_PINK03
ARMR_POINT01 - ARMR_PINK01 + ARMR_POINT02
ARMR_POINT02 + ARMR_POINT03
ARMR_POINT03
ARMR_RING01 - ARMR_POINT01 + ARMR_RING02
ARMR_RING02 + ARMR_RING03
ARMR_RING03
ARMR_THMB01 - ARMR_RING01 + ARMR_THMB02
ARMR_THMB02 + ARMR_THMB03
ARMR_THMB03
ARMR_WEAP - ARMR_THMB01
ARMR_WING01 - ARMR_HAND
CHEST03_TORCH01 - HEAD_NECK
LEGL_THIGH - CHEST_CHEST02 + LEGL_CALF
LEGL_CALF + LEGL_FOOT
LEGL_FOOT + LEGL_THUMB01
LEGL_RING01 + LEGL_RING02
LEGL_RING02 + LEGL_RING03
LEGL_RING03
LEGL_POINT01 - LEGL_RING01 + LEGL_POINT02
LEGL_POINT02 + LEGL_POINT03
LEGL_POINT03
LEGL_THUMB01 - LEGL_POINT01 + LEGL_THUMB02
LEGL_THUMB02 + LEGL_THUMB03
LEGL_THUMB03
LEGR_THIGH - LEGL_THIGH + LEGR_CALF
LEGR_CALF + LEGR_FOOT
LEGR_FOOT + LEGR_THUMB01
LEGR_RING01 + LEGR_RING02
LEGR_RING02 + LEGR_RING03
LEGR_RING03
LEGR_POINT01 - LEGR_RING01 + LEGR_POINT02
LEGR_POINT02 + LEGR_POINT03
LEGR_POINT03
LEGR_THUMB01 - LEGR_POINT01 + LEGR_THUMB02
LEGR_THUMB02 + LEGR_THUMB03
LEGR_THUMB03
PELV_TAIL01 - CHEST_CHEST01 + PELV_TAIL02
PELV_TAIL02


So yea, we're getting somewhere now :)

[daeken_bb]

Ok, another update.

I was right in my assumption that a given polygon could have vertices attached to more than 1 bone heh

http://home.archshadow.com/~daeken/bat_tree.txt
That has the full bat bone tree, as well as the matchups of the polygon vertices and the bones.

M'be I'm getting closer :P

[Windcatcher]

It's the same way in WLD. I had the same problem until I realized that I had to transform all of the vertices for the model before rendering it. The way to do it is to load the list of vertex-bone indices in advance. When you are ready to render the model, iterate through all of the vertices and go from un-transformed vertex to transformed vertex (basically keep them in two different arrays, so the original ones are never lost). Then just render all of the polygons, using the list of transformed vertices. It means that you can't get away with using the matrix stack to go from parent bone to child bone, but it will render the models properly (besides, OpenGL has pretty small limits on some stack depths).

[Mongrel]

besides, OpenGL has pretty small limits on some stack depths

True, but not that small. My Geforce 4 has a modelview stack depth of 32. I doubt that there's a model that would need more than that.

The main advantage of this is, that you're doing all the transformations in hardware and hardware processing = goooooood. Of course, you could also write a vertex shader for this. Isn't very hard with GLslang and works with Geforce 3+.

[jbb]

Nice work guys :)
I found I had a book already which described a lot of this for the directx skinned mesh stuff at least and this sounds very similar.

[wyndam]

Heyas -- new to the boards.

I've spent some time working on the EQ .wld format files attempting to decode them, and it sounds like the entire structure of the file format has already been mapped out. Any chance there's a reference to this information somewhere? Source code is also fine.

My goal is to be able to import the zone and character models into Blender so I can render some images for tactical maps and the like. However, there's two problems.

One is that it is my understanding WLD armatures define the X-axis to travel down the length of the bone. Blender defines this as the Y-axis instead. I had lots of fun dealing with the Quaternions, but I got the meshes to deform properly by brute force. However, this won't work at all for the skeleton animations because of this difference in definiton.

Second is that I have NO idea how to even unpack the EQG files into useable files, much less attempt to decode the format just yet. If someone could point me in the direction of how I can unpack the files it would be grand. Beyond that, I'd love to get cracking on the file formats for the new zones and models.

[Windcatcher]

The newest reference to .WLD files can be found here:

http://prdownloads.sourceforge.net/eqemu/wlddoc.pdf?download

Good luck...

[wyndam]

Hey, thanks a ton for the reference. Looks exactly like what I've been wanting for the WLD stuff. Still no info on the new EQG format just yet? I'd like to at least be able to unpack the files in the archive.

Another question:

In the WLD Reference, under Fragment 0x12 -- it lists the 8 elements of each Data1 Item to be:

RotateDenominator
RotateXNumerator
RotateYNumerator
RotateZNumerator

ShiftXNumerator
ShiftYNumerator
ShiftZNumerator
ShiftDenominator

I interpreted the first four to be a Quaternion defining the rotation. Using Blender, I deformed the Mesh Vertices using this method (converting the Quats to Matrices), and here's what it looks like:

http://www.pancua.net/wyndam/final_destiny/images/regrua.jpg

Or the Wireframe View Here:

http://www.pancua.net/wyndam/final_destiny/images/regrua-wire.jpg

I'm assuming you got the same or similar results using the method outlined in the WLD documentation?

[Windcatcher]

Did you download the newest version? I thought I revised that section. The first four values are actually Euler values that describe a rotation about an arbitrary axis (what I guess you mean by a quaternion). The results you show are exactly what I get when I treat them as such...you're on the right track.

[wyndam]

Hrmm ... Euler Values? Interesting.

Quaternions can be defined in two different ways. One is axis/angle. The other is similar to a 4d vector (x,y,z,w). I treated these as the 2nd, and got the pictures you see.

I *believe* I've downloaded the latest WLD reference. In case it hasn't been updated (or I'm a moron and didn't get the latest), can you briefly describe how to put these together? From the Documentation I've read (admittedly most of it from Blender), Euler Rotations only require 3 components. Am I off my rocker?

The reason I'm asking so many questions is because while the deformed meshes look correct, the animations don't look correct. While this may in fact be a bug in Blender, I'm also entertaining the possibility I interpreted these fragments incorrectly. Prehaps a SS of the Skeletons might help clarify a bit more, and see if we're on the same page.

Also, it appears SOE defines the vertex groups such that the X axis travels down the length of the bone. Am I correct in this assumption?

You'll notice on the pictures below, the resulting deformed mesh has the Left Arm of the model on Positive-Y, the Top (head) of the model on Positive-Z, and the front of the model on Positive-X. Did you see similar results?

[wyndam]

Here's some quick screens of the Skeleton for the Regrua Above. See if it looks anything in the zip code of what you've got.

http://www.pancua.net/wyndam/final_destiny/images/regrua-skel.jpg
http://www.pancua.net/wyndam/final_destiny/images/regrua-skel2.jpg

[wyndam]

Aha! Now I see we actually are talking about the same thing. :-)

http://mathworld.wolfram.com/EulerParameters.html

Euler Parameters are four components. These are also called Quaternions, or so it appears.

[wyndam]

So, for those of you scoring at home, I finally went back and re-wrote my code to import the Armature Fragments. After a few hours, I managed to finally crack the import, and I now have the animations working in blender for the older format WLD files.

I'd still REALLY like to know what information has been acquired from the EQG format files thus far, as my guild is fighting mobs in these new zones and I'd like to be able to make some renderings of them.

[sysadmin]

Excelent work!

[wyndam]

Well, I downloaded the OpenEq Sources from the Subversion Repository, and I looked them over. Using them, I was able to decode the entirety of the MOD and TER format files. It appears, from the ter.hpp/cpp and mod.hpp/cpp files, the loaders do not fully understand the material segment. I manage to decode it.

I wrote a new version of the Loader for the TER format files based upon the interfaces and data structures already defined in the OpenEQ Header Files. This Loader at least compiles, thought I cannot test it currently as I don't have SDL on my machine. I compiled it with g++ on cygwin. The Header and Implementation files should be found at the bottom of this message (so long as I don't forget to attach them, hehe).

For those of you not familiar with C++ code, here's how the Material Segment Works:

The Material Segment looks like this:
[ long mat0_index ]
[ long mat0_name0_offset ]
[ long mat0_name1_offset ]
[ long mat0_parameter_count ]
[ long param0_name_offset ] [ long param0_value_type ] [ (long|float) param0_value ]
[ long param1_name_offset ] [ long param1_value_type ] [ (long|float) param1_value ]
...
[ long paramN_name_offset ] [ long paramN_value_type ] [ (long|float) paramN_value ]

[ long mat1_index ]
[ long mat1_name0_offset ]
[ long mat1_name1_offset ]
[ long mat1_parameter_count ]
[ long param0_name_offset ] [ long param0_value_type ] [ (long|float) param0_value ]
[ long param1_name_offset ] [ long param1_value_type ] [ (long|float) param1_value ]
...
[ long paramN_name_offset ] [ long paramN_value_type ] [ (long|float) paramN_value ]
...


long index : 32-bit integer material index.
long name_offset0 : 32-bit integer offset. Reading from this location in the string table gives the material name.
long name_offset1 : 32-bit integer offset. Reading from this location in the string table gives another name string, which I'm not really sure what this second string is for.
long parameter_count : 32-bit integer number of 12-byte parameters belonging to this material.
long param0_name_offset : 32-bit integer offset. Reading from this location in the string table gives the parameter name. (Example: "e_TextureDiffuse0")
long param0_value_type : 32-bit enumeration of what type the param0_value will be. (These values are always the same size of 4 bytes, just interpreted differently).
(long | float) param0_value : If param0_value_type is '0', this value is a 32-bit float. If param0_value_type is '2', this value is a 32-bit integer offset into the string table.

EQG TER Format and EQG MOD Format use the same structure for their material segments. The Material records specify a number of different parameters, including texture names, shinyness values, and normal map textures. The current structures inside the openeq sources written didn't provide locations to store them, so I just stored the same information the original loader did with respect to the textures.

I hope you guys find this information helpful. :-)