eXtensible World File

[daeken_bb]

This thread will chronicle the creation of the XWF (eXtensible World File -- good name?) file format.

I believe we've pretty much agreed that the proper way to do this is an atomic structure where every atom either contains data directly or is a container for other atoms, or both. So, the first thing that needs to be decided is the structure of the atom header.

I propose the following:

struct Atom {
  char type[4];
  unsigned long children;
  unsigned long size;
};

Where `type' is a FourCC (4 character code) representing the type of atom it is, e.g. vertex data, texture, placeable object, etc. `children' is the number of child atoms the current atom contains. `size' is the size in bytes of pure data that follows.
This would then be followed by `size' bytes of pure data and then `children' number of atoms.

I also propose we store all strings in this format: a 2-byte unsigned integer in network order followed by that number of bytes.

Now we need to figure out the FourCCs we will need for the initial version.

Please post your input here.

Happy Hacking,
Lord Daeken M. BlackBlade
(Cody W. Brocious)

Edit #1: Changed children to long from char as per WC's suggestion.
Edit #2: Changed longs to unsigned longs.

[Windcatcher]

I think children should be a long instead of a char. It eliminates the possibility of running out of headroom and modern PC's like grabbing entire long's at a time anyway. Lately I've been a fan of using 32-bit fields (longs and floats) by default unless there is a reason not to.

Edit #1: Well, some of the things we definitely need:
- texture
- vertex
- color (RGBA only?) (mapped to vertex)
- triangle
- texture coordinate (mapped to vertex)
- model
- bounding box (corner points)
- bounding sphere (center and radius, or maybe just radius if the center is implied)

[daeken_bb]

Ok, changed that

Now we need to figure out what FourCCs we need. We should use a standard naming convention... all lower case would be simplest. Also, when listing them, replace spaces with dashes ('-') but don't use that in the files themselves.

First we'll need textures. I propose that we have a FourCC called 'tex-' which stores the properties and filenames of a given texture. My current thought is that we simply have an unsigned long indicating the texture ID, then properties/filenames in the format 'string name, string value' for however many entries there are. This will keep some degree of simplicity.

Now the other three main things are vertices, polygons, and octree data.

Vertices, I think, should be stored in an atom with the FourCC 'vert'. The format should simply be an unsigned long indicating its vertex group id, then an unsigned long indicating the number of vertices, then an array of the following structs:

struct Vertex {
  double x, y, z; // Position
  double i, j, k; // Normal
  double u, v; // Texture coords
};

The vertex group ID allows multiple 'vert' atoms in the same file to be addressed seperately.

Polygons should be in an atom with the FourCC 'poly'. To allow for addressing of vertex groups, the 'poly' atom should start with an unsigned long indicating the vertex group it will be dealing with, then an unsigned long indicating the number of polygons, followed by an array of the following structs:

struct Poly {
  unsigned long v1, v2, v3; // Vertex indices in the indicated vertex group
  unsigned long tex; // Texture id.
  unsigned long flags; // If bit 1 is set, it can be walked through, if bit 2 is set, it is transparent.
};

Any input?

Edit #1: Added flags field to Poly
Edit #2: Added i, j, and k fields to vertices.
Edit #3: Switched the order of texcoords and normals.

[daeken_bb]

Moving on, we should have two different ways of creating meshes. A 'wld-' atom and an 'obj-' atom.

An 'obj-' atom should have the following things:

* A string in its data field containing its name
* A bunch of 'tex-' atoms describing all of the materials used in the object
* A 'vert' atom containing all of the vertices for the object
* A 'poly' atom containing all of the polygons for the object


A 'wld-' atom should contain the following:
* 'tex-' atoms containing material information
* A 'vert' atom containing all the vertices for the zone
* An 'octp' atom containing the octree for the zone

An octree should consist of two types of atoms, 'octp' and 'octe'. 'octp' would be an octree node with children, where an 'octe' atom would be an octree end node.

An 'octp' atom should contain the following in its data field:

struct OctP {
  double cen[3];
  double size[3];
};

Where cen is the center of the octree node cube, and size is the distance from the center to the extreme x, y, and z... in other words, half of the total length, width, and depth of the cube. It should have 8 children atoms, all of which being with 'octp' or 'octe' atoms.


And 'octe' atom should have the same data field as an 'octp' atom. The difference is that an 'octe' atom has a 'poly' atom as its only child. This poly atom contains all of the polygons wihin this octree end node.

That should almost do it for an initial file format... anything I'm missing?

[Windcatcher]

Any proposition for vertex color/shading (resists going into a tirade on EQ's use of emissive coloring).

Also, we need to tag certain polys as "solid" or "non-solid". For instance, the player can walk through the leaves of a tree.

Alpha transparency. Part of color?

As a side note, Quake files have shaders as separate and you assign them to vertices...

Edit #1: for frustum culling, you'll need bounding spheres in addition to bounding boxes (sphere culling should be the first step as it's way faster)

[jbb]

One question -
what is the advantage of using "atoms"?
It seems to make it hard to get to the data you want without reading the whole file?

Why not store named "files" in a zip file format?

[daeken_bb]

Originally Posted by Windcatcher

Any proposition for vertex color/shading (resists going into a tirade on EQ's use of emissive coloring).

Also, we need to tag certain polys as "solid" or "non-solid". For instance, the player can walk through the leaves of a tree.

Alpha transparency. Part of color?

As a side note, Quake files have shaders as separate and you assign them to vertices...

How would you suggest the vertex color/shading be done? Likewise, how would you suggest the shaders be done? I'm not quite sure how it should be done

[daeken_bb]

Originally Posted by jbb

One question -
what is the advantage of using "atoms"?
It seems to make it hard to get to the data you want without reading the whole file?

Why not store named "files" in a zip file format?

The primary advantage of atoms is you have a completely open-ended file format. You can change it majorly without breaking compatibility. It's also _very_ easy to read and write.

[Windcatcher]

Well, in OpenGL, we have ambient, diffuse, and emissive shading, and maybe some others, where most take an rgba value (emissive ignores the alpha component -- so to do transparency you have to set the ambient alpha value). I guess you could define a shader that says which ones to use and the values, and then the vertex group would have a shader list atom that had n shader atoms (1 per vertex). We should probably think it through before diving into it. What you have is probably fine as a first cut. The format is extensible, as you say.

Transparency is part of ambient alpha shading, though, so it will be an issue as soon as we have zone bounding polys (that keep you from walking off the world). Also, being able to tag polys as solid or not is a must, but maybe that should go into another atom?

[daeken_bb]

Ok, so do you think that the file format described above is good enough for an initial version? If so, I'll start work immediatly on getting it in place in OpenEQ so we can start using it and testing stuff

It shouldn't take long for you to make OZ work with it, nor should it take me long to get OpenEQ to work with it, I don't think

[Windcatcher]

Let's try it and see what we get. I already have OZ open and I've been working on it. I just changed ambient light to time-of-day and put in code to better simulate sky color, ambient light, and it even shows stars that wheel in the sky (one of the features I added to the engine was a star field). The bird's-eye view is still broken but everything else works...I can either try and fix that while you make a viewer or I can forget it for now and start on the exporter...what do you think?

[daeken_bb]

Originally Posted by Windcatcher

Let's try it and see what we get. I already have OZ open and I'v been working on it. The bird's- eye view is still broken but everything else works...I can either try and fix that while you make a viewer or I can forget it for now and start on the exporter...what do you think?

The sooner the exporter works, the sooner we have a viewer available to test stuff with hehe

[Windcatcher]

Okay, I'll get on it. One question: you mentioned bytes in network order. Is that big-endian? I think there's a standard Windows API call that will put them in network order for me, but I'm not sure.

[daeken_bb]

Originally Posted by Windcatcher

Okay, I'll get on it. One question: you mentioned bytes in network order. Is that big-endian? I think there's a standard Windows API call that will put them in network order for me, but I'm not sure.

Yea, I'd like to have everything in network order (big-endian) as it'll help with cross-architecture compatibility.

Simply use the function htonl() (or its Delphi equivalent) to convert to/from network order.

[jbb]

struct Vertex {
  double x, y, z; // Position
  double u, v; // Texture coords
};

You might want to add vertex normals to that.