EQEmulator Forums

EQEmulator Forums (http://www.eqemulator.org/forums/index.php)
-   General::News (http://www.eqemulator.org/forums/forumdisplay.php?f=594)
-   -   Zone MMF Files (http://www.eqemulator.org/forums/showthread.php?t=40805)

Uleat 07-28-2016 10:57 PM

Zone MMF Files
 
A memory-mapped file feature has been added to zone maps.


This feature should alleviate some wasted memory in the map loading process and quicken zone boot-up times by a percentage.
(Actual benefit will vary depending hardware, zone size, etc...)


Please consider using this option as experimental for the time being.
[continued...]

Uleat 07-28-2016 10:57 PM

[continuation...]

This feature essentially saves the post-load state of the zone map..

..meaning, that once a zone map file has been loaded and the complex transformation has occurred, the in-memory object is saved
to a memory-mapped file for future loading.


This is not typical of a 'standard' server..but, in testing, I realized a huge difference in map loading time for my test configuration (in debug build.)

- pre-mmf conversion map load time of moors: ~85 seconds
- post-mmf conversion map load time of moors: ~0.2 seconds (uncompressed format)

Yes, that is not a typo. that's a decrease of ~84.2 seconds.

This is an exaggerated example and is not typical (or even close) to the realized performance increase.
(Obviously, I found a bottleneck in my system..but, its alleviation should help some with most peoples' servers.)


I decided to implement compression into the file storage because I noticed that some of the files were extremely large (77+ megs for moors,)
and this is where, I suspect, most people will encounter problems...

Using compression, the additional file storage appears to be approximately 3.19 gigs total for all files in the maps directory (up 2.67 gigs.)

Non-compressed (raw) files took that requirement up to 8.69 gigs of storage (up 5.5 gigs over compressed MMF usage.)

I think the slight zone boot-up overhead of compressing the MMF is well worth its use.


Problems with the zlib1.dll can summarized to this: "It will crash!"

The server code is not the problem..but, rather the way that the zlib dll is configured when it is built.


These particular zone map conversions were consistent crashes with my system: abysmal, acrylia and moors

There is no guarantee that testing only these three maps will ensure safety with other conversions..but, it's a start.

To test, open a command prompt window and change to your server directory.

Then, use these three commands:
Code:

zone convert_map abysmal.map
zone convert_map acrylia.map
zone convert_map moors.map

These commands will start zone.exe in a special 'map conversion' mode and will produce a mmf file of the zone map - if successful.

If you get a 'zone.exe has stopped' type pop-window, hit continue and the stack should dump to the console window. (Linux systems will probably
complain as well..but, I'm not sure how it responds to issues like this.)

Look just above the 'SaveMMF' call for a zlib crc message.

If the call is to 'crc32_combine64' and you have a 32-bit server build, this is your problem. You will need to compile your own version of zlib
with source code from here: http://www.zlib.net/ (download the source code, not the compiled DLL.)

This solution should solve the problem with the conversion crash. If you have other problems not described in this thread, please post them in a new one.


Please see the changelog for more information.


EDIT:

The batch conversion script will process all map files in the 'maps' sub-directory.

It took over two hours on my system..but, performance will vary according to your system's specs.


I have not tested running the conversion script on a running server..but, it may be possible since there are checks in-place
to work around 'partial' files.

I'm thinking that either the conversion's zone instance will fail conversion or the booted zone's save process will fail..but, a crash
shouldn't ensue nor should a mmf not be created by the 'winning' process.

Again, untested theory.


EDIT2:

I should iterate that this feature is really designed for low-spec hardware/dynamic zone servers.

Due to the nature of those criteria, this feature should really help client load times on those servers..

..but, that's not to say that higher-end servers wouldn't benefit from this feature.

It is a forced recompile option..so, its use is left up to the individual admins to enable it on their servers.

DanCanDo 07-29-2016 01:10 AM

Just compiled and ran the conversion script. (mine only took 40 minutes)(chuckle)
On Win 7(64) box, everything went smooth. No errors, bugs or insects.
Zoning to Moors took 19-22 seconds.

On Linux box, (Debian 7.8) however, didn't get past compiling, with these errors:

Code:

/home/eqemu/source/zone/raycast_mesh.cpp: In member function ‘void RAYCAST_MESH::MyRaycastMesh::serialize(std::vector<char>&)’:
/home/eqemu/source/zone/raycast_mesh.cpp:1182:42: error: cast from ‘RAYCAST_MESH::NodeAABB*’ to ‘RmUint32 {aka unsigned int}’ loses precision [-fpermissive]
/home/eqemu/source/zone/raycast_mesh.cpp:1182:60: error: cast from ‘RAYCAST_MESH::NodeAABB*’ to ‘RmUint32 {aka unsigned int}’ loses precision [-fpermissive]
/home/eqemu/source/zone/raycast_mesh.cpp:1188:42: error: cast from ‘RAYCAST_MESH::NodeAABB*’ to ‘RmUint32 {aka unsigned int}’ loses precision [-fpermissive]
/home/eqemu/source/zone/raycast_mesh.cpp:1188:61: error: cast from ‘RAYCAST_MESH::NodeAABB*’ to ‘RmUint32 {aka unsigned int}’ loses precision [-fpermissive]
make[2]: *** [zone/CMakeFiles/zone.dir/raycast_mesh.cpp.o] Error 1
make[1]: *** [zone/CMakeFiles/zone.dir/all] Error 2
make: *** [all] Error 2


DanCanDo 07-29-2016 02:11 AM

Quote:

Originally Posted by Uleat (Post 250316)

I have not tested running the conversion script on a running server..but, it may be possible since there are checks in-place
to work around 'partial' files.

I'm thinking that either the conversion's zone instance will fail conversion or the booted zone's save process will fail..but, a crash
shouldn't ensue nor should a mmf not be created by the 'winning' process.

Again, untested theory.

I took the liberty of testing the conversion script on a running server.(win 7)
I logged one toon in and had it running around zoning while the conversion was
going on.
I kept an eye on the progress of the conversion and timed it out to zone in to
a map that was being processed. It all went well without any hitches.

Uleat 07-29-2016 04:57 PM

Quote:

Originally Posted by DanCanDo
Zoning to Moors took 19-22 seconds.

Did you happen to get a 'control' time so that a comparison can be made? :)


I'll take a look at that linux conversion error. Shouldn't be too hard to find the proper casting method.

Thanks!

DanCanDo 07-29-2016 06:10 PM

Quote:

Originally Posted by Uleat (Post 250320)
Did you happen to get a 'control' time so that a comparison can be made? :)

Sorry, you'll have to give me a little guidance on that one. As in Huh? (chuckle)

Uleat 07-29-2016 06:16 PM

Sorry :)

A zoning time comparison using the old versus new methods.

DanCanDo 07-29-2016 06:20 PM

Oh, actually yes, I was paying attention to the old vs new. I based this on zoning to moors
only. Before it was 25-33 seconds on average, sometimes higher, but never below 25
After conversion it was a steady 19-22 secs

Uleat 07-29-2016 06:29 PM

Nice! Thanks for that info!


I pushed a (hopeful) fix for that 64-bit compile issue.

Let me know if that does not work.

DanCanDo 07-29-2016 06:48 PM

I just pulled that fix and compiled on the linux box again. No compile errors this time :)
I'll get the server booted up and get the map conversion done, then test it all out, to
see how the linux server takes to it all.

DanCanDo 07-29-2016 06:53 PM

Oh, ok, it rapidly gave me "conversion failed" on all of them, so I didn't get too far there.
Didn't give me any visible errors in window, so I will see if I can some kind of print.

Uleat 07-29-2016 07:37 PM

Not sure what linux should be reporting with a dll failure.

I'll look around and see if I can find a way to trace it to the dll.


EDIT:

The code that was 'fixed' takes the offset of the base pointer and subtracts it from the pointer reference, then divides by the size of the
object to get an 'index' that can be used to deference the base pointer during the load process.

If that index is computed to be '< 0' or '>= mNodeCount' (which should not occur) .. you should be getting an exception error during the LoadMMF call.

Since this is happening during the conversion/save process, it most likely is dll-related..but, will need more info before confirming that.

DanCanDo 07-29-2016 08:08 PM

Is there a way I can get that script to spit out some kind of log ?
I'm just running it with perl command from cmd window in linux, but
it won't give me any kind of feedback other than failed

Uleat 07-29-2016 08:12 PM

The 'status' message actually comes from zone.exe and is passed to the script.

Try invoking it using the manual format and see what happens.

Code:

./zone convert_map abysmal.map
(or whatever the linux command line requires..)

DanCanDo 07-29-2016 08:23 PM

Sorry, all I got was - abysmal.map conversion failed


All times are GMT -4. The time now is 11:46 AM.

Powered by vBulletin®, Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.