ZoneProxy-friendly changes to 0.5.3DR2

[Windcatcher]

I've updated my copy of EQEmu 0.5.1 to make it work with ZoneProxy (and hence OpenZone zones), and it is tested and working (except for the GM #goto command in third-party zones; using it forces the client to reload the zone--I don't know why). I then made the same changes to 0.5.3DR2. It compiles, but I'm not equipped to test it on my setup (the code is exactly the same where the changes were made and I'm highly confident that it will work).

I've built a small ZIP file with the changed 0.5.3DR2 source code files, but I'm reluctant to put this in the Admin Tool download area. I'd really like for someone to get this into the server CVS repository so the server will be ZoneProxy compatible from here on out. It is *not* necessary that clients connecting to such a server have ZoneProxy. If a client doesn't have ZoneProxy running, the server tells the client that the third-party zone is unavailable. If a client wants to use third-party zones, they must have ZoneProxy running *before* starting the EQ client (see the ZoneProxy instructions for a full explanation).

ZoneProxy compatibility requires a small change to the database so the server knows which zones are third-party and which aren't:

alter table zone add thirdparty int(2) unsigned default '0';

Basically, set the thirdparty field to zero (0) for all zones that come from SOE and one (1) for all zones created with OpenZone. Regarding the zone ID, I recommend starting with numbers that definitely won't interfere with any potential SOE zone IDs (I started with 800). DO NOT USE 999.

My changes to the server code use zone ID 999 under certain circumstances to force a zone reload. At some point this might have to be changed to 9999 to make absolutely sure there is no conflict, but I seriously doubt it will ever be necessary.

[Lurker_005]

And he quietly keeps churning out the good stuff!

where is this wonderfull change ? is it on sourceforge?

[Windcatcher]

I can put it there if people wish...the question is, would a dev please try this out and put it in CVS if it works? I'm not interested in having to do this for every EQEmu release that comes out in the future...

WC

P.S. OpenZone 2.6 is IMMINENT. Here's the $64,000 question: release it now, or try to make animated meshes working first (flags, trees, etc. -- not creatures). I know how animated meshes work in .WLD files, but OpenZone doesn't have that capability yet.

[Trumpcard]

WC, I'll take a look at it as soon as we can. It will probably be a couple of weeks though.

[Windcatcher]

Okay, I'll upload the EQEmu changes soon (tonight). Regarding animated meshes in OpenZone, I started laying the groundwork and realized that it'll take a lot of work and experimentation to get it completed, so I'll save that for the next version. In the meantime I'm taking a stab at dungeon creation with some dungeon-specific scripts.

[daeken_bb]

WC, I was wondering if you'd be able to take the time to document WLD a bit, or come on IRC so we can discuss it and I'll write it, my nick on IRC is Daeken if you want to get in touch with me via that method :)

Cheers,
Daeken

[Windcatcher]

I just uploaded the my source files to 0.5.3DR2 that have to be changed to make it compatible with ZoneProxy. You can get it at:

http://prdownloads.sourceforge.net/eqemu/0.5.3DR2_zoneproxy_friendly_source.zip?download

I've been considering documenting what I know about .WLD files for a long time. It's definitely an overdue project, but I simply haven't had the time for it yet.

There are two major issues when dealing with .WLD files: the file format itself (that is, the overall file organization, including dealing with string locations, etc.) and fragment definitions. Documenting the fragment types will consume 99% of the document and is the most important part.

A .WLD file contains what are called "fragments" (that's what the original ZoneConverter source calls them--I have no idea who wrote it). A fragment is basically just a data structure, and there are two overall types:

- standalone fragments
- fragments that have a built-in reference to another fragment (such a fragment is a superset of a standalone fragment)

Fragments that reference other fragments are something like nodes in a linked list: they contain information that's related to another fragment, such that the client will want to be able to walk from one fragment to another. An example is animated meshes: there is a fragment that details basic mesh information, but if that mesh is animated it will contain a reference to another fragment that contains the additional animation stuff (actually there is a "glue" fragment that sits in the middle, kind of like an animation reference fragment--these "glue" fragments let data that takes a lot of memory be reused). Fragments can contain multiple such references, but the FragmentReference flavor always gives you one for free. Any others are specific to a fragment type.

Every fragment, whether standalone or otherwise, contains an ID value that tells the client its type. The highest ID I've seen to date is 0x37. That's not to say I've encountered every ID between 0 and 0x37, but there are a lot of different kinds of fragments. I've figured out most of the ones I've seen, though some are still a mystery. Others are only partially understood.

[Lurker_005]

Windcatcher, as I recall you don't do any C programming do you? :/ I was hoping your knowledge could fix kaiyodo's model viewer to display character models corectly. I havn't seen kaiyodo around for quite a long time, but the web page with source and exe's is still up. I figure a proper model viewer could seriously help the GenRaTex project figure out what models/textures are avaliable in each zone.

[Windcatcher]

I continuously upgrade DZoneConverter to help decipher .WLD files, and I took a stab at chatacter models too. So far I haven't been able to figure it out. There's something about the Data6 and Data9 areas in the Data36 fragment that relates to character models, but I don't understand it yet.

[Scorpious2k]

Speaking of chracter models, it occured to me the other day that there are names for all the models in the s3d files... if that's true, and if the names are contained in some kind of index that is easy to find, and if we could get a correlation between model names and race/model numbers....

Wouldn't it be possible to get the available race/models for NPCs by reading the s3d files for a zone, matching model names to race/model numbers (possibly with a table) and therefore identifying all races/models available for a given zone?

Just a thought.

[daeken_bb]

I started porting zoneconvertor's eqwlddata and fragment code to php, commenting, and documenting the wld format... i'll probably whip up some c structs for everything and release it in increments today :)

Cheers,
Daeken

[Windcatcher]

The original ZoneConverter source had some bugs in that code. I suggest you look at my Delphi port of it in the OpenZone source as an update.

[daeken_bb]

I understand the fragment structures and such now, but I don't actually understand the purpose of some... mind clueing me in on the use of the different types?

Cheers,
Daeken

[Windcatcher]

That's kind of a broad question. Which ones don't you understand?

[daeken_bb]

Most of them, unfortunately. I haven't found much of a description of what each fragment is used for, e.g. which fragment is for animated meshes, which is for static objects, etc.

Perhaps you could point me to a file in your source?

Cheers,
Daeken

[Windcatcher]

EQWldData.pas in the OpenZone source is a direct translation of the corresponding ZoneConverter file, but also has bug fixes and improvements.

Here's a rundown of the fragment types as I understand them and some basic documentation below:

0x03 - Texture Bitmap Name(s)
0x04 - Texture Bitmap Info
0x05 - Texture Bitmap Info Reference
0x08 - Camera
0x09 - Camera Reference
0x10 - Track Set
0x11 - Track Set Reference
0x12 - Mob Track/Object Track
0x13 - Mob Track Reference/Object Track Reference
0x14 - Placeable Reference/Player Info
0x15 - Object Location
0x16 - Zone Unknown
0x17 - Polygon Animation?
0x18 - Polygon Animation Reference?
0x1B - Light Source
0x1C - Light Source Reference
0x21 - BSP Tree
0x22 - BSP Region
0x28 - Light Info
0x29 - Region Flag
0x2A - Ambient Light
0x2D - Mesh Reference
0x2F - Mesh Animated Vertices Reference
0x30 - Texture
0x31 - Texture List
0x32 - Vertex Color
0x33 - Vertex Color Reference
0x35 - First Fragment
0x36 - Mesh
0x37 - Mesh Animated Vertices

There are other fragment types in the ZoneConverter source, but I haven't encountered them. Here I talk about all of the ones I've seen.



Misc
----

0x35 - First Fragment - For some reason every .WLD file I've ever encountered begins with one of these. I have no idea why or whether this is necessary.

Textures
--------

0x03 - Texture Bitmap Name(s) - Contains the names of one or more bitmaps used in a particular texture. When there is more than one bitmap in the fragment it means that the texture is animated (e.g. water).

0x04 - Texture Bitmap Info - Refers to a 0x03 fragment. Also contains flags to tell the client information about this particular texture (normal or animated).

0x05 - Texture Bitmap Info Reference - Refers to a 0x04 fragment so 0x04 fragments can be reused.

0x30 - Texture - Refers to a 0x05 fragment. Contains flags to tell the client information about this texture (normal, semitransparent, transparent, masked, etc.) Having this fragment separated from the 0x05 fragment means that the zone can have multiple flavors of the same texture bitmap (e.g. one that is normal, one that is semitransparent, etc.)

0x31 - Texture List - Contains references to all of the 0x30 textures used in the zone (or, in the case of placeable objects, in that particular object).


Meshes
------

0x36 - Mesh - Contains vertex, normal, color, and polygon information for a mesh. In the case of zones, the mesh is a subdivision of the zone. In the case of placeable objects, the mesh contains the entire information for the object.

Notes on 0x36 fragment:

1. Fragment1 refers to a 0x31 fragment to tell the client what textures are used.
2. Polygons are sorted by texture index. That is, all polygons in the Data5 area that use a particular texture are grouped together.
3. Fragment2 optionally refers to a 0x2F fragment if the mesh is animated (e.g. trees or flags that sway in the breeze).
4. Fragment4 always refers to the first 0x03 fragment in the file (I have no idea why).
5. I don't fully understand this fragment. The Data6 and Data9 areas have something to do with mob models, but I don't know how they work yet.
6. There are new-format and old-format .WLD files. They have different values in the .WLD header and the main difference is in the 0x36 fragment. In new-format files, the texture coordinate values are signed 32-bit values; in old-format files they're signed 16-bit values. At this time OpenZone only exports old-format files but it would be no great effort to switch it to new-format files.

0x37 - Mesh Animated Vertices - For a given 0x36 fragment, this fragment contains a number of animation "frames". For each frame it contains a complete vertex list that supercedes the vertex list in the 0x36 fragment. For instance, if there are three frames and 15 vertices, there will be three sets of 15 vertex values in the 0x37 fragment and they will be used in lieu of the 15 vertex values in the 0x36 fragment.

0x2F - Mesh Animated Vertices Reference - Refers to a 0x37 fragment so it can be reused (e.g. for flags that all have the same shape but have different textures).


Zones
-----

0x21 - BSP Tree - BSP stands for "binary space partition". Basically the zone is broken up into regions so the client can quickly find polygons that are near the player or mobs for purposes of collision avoidance. Normally this is done along a simple grid, but the zone also has to be broken up so that "special" regions (water, lava, PvP, ambient lighting, and others) are distinct.

0x22 - BSP Region - For each node in the BSP tree there is a 0x22 fragment that describes the region. It contains information on the region's bounds and an optional reference to a 0x36 fragment if there are polygons in that region. Also contains an RLE-encoded array of data that tells the client which regions are "nearby". That way, if a player enters one of the "nearby" regions, the client knows that objects in the region are visible to the player. The client does this to know when it has to make sure mobs fall to the ground instead of stay at the spawn points, which might be in midair.

0x1B - Light Source - I suspect this defines the ambient light level in a zone (see "Light sources" below for info).

0x08 - Camera - I don't know what the parameters mean yet.

0x1C - Light Source Reference - See "Light sources" below for info.

0x2A - Ambient Light - Refers to a 0x1C fragment. Contains a list of numbers that refer to the 0x22 fragments in the zone (e.g. if there are 100 0x22 fragments then the numbers will be in the range 0-99). This fragment tells the client which regions have the particular light setting. I suspect that you could conceivably have some regions with one ambient light setting, other regions with another ambient light setting, etc. by having multiple 0x1B-0x1C-0x2A sets.

0x09 - Camera Reference - Refers to a 0x08 fragment. I don't know its purpose.

0x14 - Player Info - I don't know its purpose. Its Fragment1 value seems to use a "magic" string: "FLYCAMCALLBACK".

0x16 - Zone Unknown - It's used in zone files for some reason...

0x15 - Object Location - In zone files, this might contain the safe point?

0x29 - Region Flag - This is similar to the 0x2A fragment in that it contains a list of numbers, where each number refers to a 0x22 region. It tells the client that those regions are "special". The name of the fragment is "magic" in that it determines how the regions are flagged:

"WT_ZONE" .................... Regions are underwater

"LA_ZONE" ................... Regions are lava

"DRP_ZONE" ................... Regions are PvP areas

"DRNTP##########_ZONE" ....... e.g. DRNTP00025-02698-645.6-00020999_ZONE. This seems to tell the client that these regions constitute a zoneline. If the player enters one of these regions the client knows the player is zoning and knows the destination. I don't know if the client makes use of this since I don't think every zone has this at all zone points, but it looks interesting. I don't understand the format of the numbered part of the name.


Mob models
----------

0x12 - Mob Track - I don't know what this does, but I suspect it has to do with mob animation.

0x13 - Mob Track Reference - Refers to a 0x12 fragment.

0x2D - Mesh Reference - Refers to a 0x36 fragment. I don't know its purpose.


Light sources
-------------
0x1B - Light Source - Contains information on light color. I don't know what the other parameters in it mean.

0x1C - Light Source Reference - Refers to a 0x1B fragment.

0x28 - Light Info - Refers to a 0x1C fragment. Contains light position and radius. I don't know what the other parameters mean.


Placeable objects
-----------------
0x32 - Vertex Color - For each object that has been placed, there is one of these (put 100 trees in your zone and there are 100 of these fragments). It contains vertex shading information for each object. For example, if you have a torch near some trees, those trees should have their polygons shaded based on the light color, angle of incidence, distance, and any intervening polygons. The EQ client does *not* dyamically shade polygons in a zone; all polygons must be shaded in this way (including 0x36 fragments in the main zone file).

0x33 - Vertex Color Reference - Refers to a 0x32 fragment.

0x15 - Object Location - Contains object position, rotation, and size. Refers to a 0x33 fragment. When used in the main zone file, this fragment contains information for the whole zone (see frmMainUnit.pas in the OpenZone source).

0x17 - Polygon Animation? - I don't know what this does. I suspect it has something to do with polygon animation.

0x18 - Polygon Animation Reference? - Refers to a 0x17 fragment. I don't know its purpose.

0x2D - Mesh Reference - Refers to the 0x36 fragment.

0x12 - Object Track - I don't know what this does, but I suspect it has to do with object animation.

0x13 - Object Track Reference - Refers to a 0x12 fragment.

0x10 - Track Set - This seems to be optional for placeable objects. My guess is that it allows a placeable object to have more than one animation track. For each track that is used there is a reference to a 0x13 fragment. There might (though not necessarily) also be a reference to a 0x2D fragment for each track.

0x11 - Track Set Reference - Refers to a 0x10 fragment.

0x14 - Placeable Reference - Fragment1 contains a "magic" string: "SPRITECALLBACK". Data2 contains at least one fragment reference that refers to either a 0x2D or 0x11 fragment. I don't understand what all the parameters in this fragment mean.




There is much more to .WLD files than this, and this document doesn't contain anything on the file header and it's overall organization. Also, many fragments contain flags that have to be set properly for the zone to work (e.g. whether a 0x22 fragment refers to a 0x36 fragment). See frmMainUnit.pas in the OpenZone source to see how I create fragments when exporting zone files. My fragment variables typically start with "D"; that is, D36 for a 0x36 fragment, D21 for a 0x21 fragment, etc. By searching for something like "D36." you can see how I set the fields in a 0x36 fragment, for example. Over time we can possibly grow this to become a more comprehensive reference.

[daeken_bb]

Wow. Thanks a ton, perhaps I can do some fun stuff with WLD files now ;)

[Windcatcher]

I found a problem in my description of the 0x18 fragment. Camera is actually 0x08, and 0x09 refers to that. I've since updated my notes above.

[Lurker_005]

I remember some referances to some zones "Knowing" where to zone to without EQEmu telling it... I'm sure the "DRNTP##########_ZONE" data includes zoneline_id zone_id, X,Y,Z destinations, and rotation A 999 means to keep the current zone out value. With a little cross referancing to the zoneline table, and experimenting in game that little tidbit should be deciferable.

The rest is greek to me :shock:

[Windcatcher]

I'm in the process of building a comprehensive reference. I just finished documenting all of the fragments described above, including fields, flag values, etc. So far the reference is a 23-page Word document... :?

WC

[Kroeg]

Windcatcher... I'm addicted to openzone.. I love that tool :D
My question, though, in order to test the third party zones (zoneproxy is required).. but so is the additional code. I'm using one of the various dr2 builds that is relatively recent.. what build was this code addition for? Would it be too much of a pain in the ass to post a possible snippet for later versions?

I'm assuming I MUST use zoneproxy to test a third party zone, in asking this :)

[Windcatcher]

I can't find the thread that had the source .ZIP file anymore. Perhaps it got wiped. My 0.5.3DR2 changes are identical to changes I had to make to the 0.5.1 source and I suspect that any build you use should be similar. All of my changes are bracketed by the following comments:

// ZONEPROXY BEGIN
.
.
.
// ZONEPROXY END

There aren't all that many of them; you shouldn't have too much trouble adding them to just about any build...

WC

P.S. Please let me know if you get it working...

[Kroeg]

Thanks, I'm working on it. So far, I've created a sample zone with one building in it and a minimal heightmap. I then exported it as "infernal1", and it saved all 3 files for my zone. I then copy these over to the zoneproxy folder, set the zonealias for "tutorial" ... load eq, attempt to #zone tutorial and 1 of 2 things happen: 1) eq client crashes 2) I zone to the normal tutorial zone (not infernal1)

Perhaps I'm just an idiot and am not taking my time to get it working right, but is there something specific im doing wrong? :)

[Trumpcard]

WC. I'll try to merge this in to the baseline this week so it doesnt get lost. I may put it in an ifdef section if you dont mind.

[Windcatcher]

Don't zone to "tutorial". Here's what you need to do:

1. In your zone table in the DB, create an entry for infernal1.

2. Add an integer column to the zone table called "thirdparty":

alter table zone add thirdparty int(2) unsigned default '0';

3. Set thirdparty to 1 for your new zone.

4. Log into the client and #zone to infernal1. The server will tell ZP to set up "infernal1". ZP will copy and re-code the files so they look like tutorial to the client. ZP will then tell the server when it's done and the server will tell the client to zone to "tutorial". It's all transparent.

WC