Go Back   EQEmulator Home > EQEmulator Forums > General > General::News

General::News EQemu news posts.

Reply
 
Thread Tools Display Modes
  #1  
Old 07-28-2016, 10:57 PM
Uleat's Avatar
Uleat
Developer
 
Join Date: Apr 2012
Location: North Carolina
Posts: 2,240
Default Zone MMF Files

A memory-mapped file feature has been added to zone maps.


This feature should alleviate some wasted memory in the map loading process and quicken zone boot-up times by a percentage.
(Actual benefit will vary depending hardware, zone size, etc...)


Please consider using this option as experimental for the time being.
[continued...]
__________________
Uleat of Bertoxxulous

Compilin' Dirty

Last edited by Uleat; 07-29-2016 at 09:27 PM..
Reply With Quote
  #2  
Old 07-28-2016, 10:57 PM
Uleat's Avatar
Uleat
Developer
 
Join Date: Apr 2012
Location: North Carolina
Posts: 2,240
Default

[continuation...]

This feature essentially saves the post-load state of the zone map..

..meaning, that once a zone map file has been loaded and the complex transformation has occurred, the in-memory object is saved
to a memory-mapped file for future loading.


This is not typical of a 'standard' server..but, in testing, I realized a huge difference in map loading time for my test configuration (in debug build.)

- pre-mmf conversion map load time of moors: ~85 seconds
- post-mmf conversion map load time of moors: ~0.2 seconds (uncompressed format)

Yes, that is not a typo. that's a decrease of ~84.2 seconds.

This is an exaggerated example and is not typical (or even close) to the realized performance increase.
(Obviously, I found a bottleneck in my system..but, its alleviation should help some with most peoples' servers.)


I decided to implement compression into the file storage because I noticed that some of the files were extremely large (77+ megs for moors,)
and this is where, I suspect, most people will encounter problems...

Using compression, the additional file storage appears to be approximately 3.19 gigs total for all files in the maps directory (up 2.67 gigs.)

Non-compressed (raw) files took that requirement up to 8.69 gigs of storage (up 5.5 gigs over compressed MMF usage.)

I think the slight zone boot-up overhead of compressing the MMF is well worth its use.


Problems with the zlib1.dll can summarized to this: "It will crash!"

The server code is not the problem..but, rather the way that the zlib dll is configured when it is built.


These particular zone map conversions were consistent crashes with my system: abysmal, acrylia and moors

There is no guarantee that testing only these three maps will ensure safety with other conversions..but, it's a start.

To test, open a command prompt window and change to your server directory.

Then, use these three commands:
Code:
zone convert_map abysmal.map
zone convert_map acrylia.map
zone convert_map moors.map
These commands will start zone.exe in a special 'map conversion' mode and will produce a mmf file of the zone map - if successful.

If you get a 'zone.exe has stopped' type pop-window, hit continue and the stack should dump to the console window. (Linux systems will probably
complain as well..but, I'm not sure how it responds to issues like this.)

Look just above the 'SaveMMF' call for a zlib crc message.

If the call is to 'crc32_combine64' and you have a 32-bit server build, this is your problem. You will need to compile your own version of zlib
with source code from here: http://www.zlib.net/ (download the source code, not the compiled DLL.)

This solution should solve the problem with the conversion crash. If you have other problems not described in this thread, please post them in a new one.


Please see the changelog for more information.


EDIT:

The batch conversion script will process all map files in the 'maps' sub-directory.

It took over two hours on my system..but, performance will vary according to your system's specs.


I have not tested running the conversion script on a running server..but, it may be possible since there are checks in-place
to work around 'partial' files.

I'm thinking that either the conversion's zone instance will fail conversion or the booted zone's save process will fail..but, a crash
shouldn't ensue nor should a mmf not be created by the 'winning' process.

Again, untested theory.


EDIT2:

I should iterate that this feature is really designed for low-spec hardware/dynamic zone servers.

Due to the nature of those criteria, this feature should really help client load times on those servers..

..but, that's not to say that higher-end servers wouldn't benefit from this feature.

It is a forced recompile option..so, its use is left up to the individual admins to enable it on their servers.
__________________
Uleat of Bertoxxulous

Compilin' Dirty

Last edited by Uleat; 07-29-2016 at 06:41 PM..
Reply With Quote
  #3  
Old 07-29-2016, 01:10 AM
DanCanDo's Avatar
DanCanDo
Discordant
 
Join Date: May 2016
Location: Above Hell
Posts: 436
Default

Just compiled and ran the conversion script. (mine only took 40 minutes)(chuckle)
On Win 7(64) box, everything went smooth. No errors, bugs or insects.
Zoning to Moors took 19-22 seconds.

On Linux box, (Debian 7. however, didn't get past compiling, with these errors:

Code:
/home/eqemu/source/zone/raycast_mesh.cpp: In member function ‘void RAYCAST_MESH::MyRaycastMesh::serialize(std::vector&)’:
/home/eqemu/source/zone/raycast_mesh.cpp:1182:42: error: cast from ‘RAYCAST_MESH::NodeAABB*’ to ‘RmUint32 {aka unsigned int}’ loses precision [-fpermissive]
/home/eqemu/source/zone/raycast_mesh.cpp:1182:60: error: cast from ‘RAYCAST_MESH::NodeAABB*’ to ‘RmUint32 {aka unsigned int}’ loses precision [-fpermissive]
/home/eqemu/source/zone/raycast_mesh.cpp:1188:42: error: cast from ‘RAYCAST_MESH::NodeAABB*’ to ‘RmUint32 {aka unsigned int}’ loses precision [-fpermissive]
/home/eqemu/source/zone/raycast_mesh.cpp:1188:61: error: cast from ‘RAYCAST_MESH::NodeAABB*’ to ‘RmUint32 {aka unsigned int}’ loses precision [-fpermissive]
make[2]: *** [zone/CMakeFiles/zone.dir/raycast_mesh.cpp.o] Error 1
make[1]: *** [zone/CMakeFiles/zone.dir/all] Error 2
make: *** [all] Error 2
Reply With Quote
  #4  
Old 07-29-2016, 02:11 AM
DanCanDo's Avatar
DanCanDo
Discordant
 
Join Date: May 2016
Location: Above Hell
Posts: 436
Default

Quote:
Originally Posted by Uleat View Post

I have not tested running the conversion script on a running server..but, it may be possible since there are checks in-place
to work around 'partial' files.

I'm thinking that either the conversion's zone instance will fail conversion or the booted zone's save process will fail..but, a crash
shouldn't ensue nor should a mmf not be created by the 'winning' process.

Again, untested theory.
I took the liberty of testing the conversion script on a running server.(win 7)
I logged one toon in and had it running around zoning while the conversion was
going on.
I kept an eye on the progress of the conversion and timed it out to zone in to
a map that was being processed. It all went well without any hitches.
Reply With Quote
  #5  
Old 07-29-2016, 04:57 PM
Uleat's Avatar
Uleat
Developer
 
Join Date: Apr 2012
Location: North Carolina
Posts: 2,240
Default

Quote:
Originally Posted by DanCanDo
Zoning to Moors took 19-22 seconds.
Did you happen to get a 'control' time so that a comparison can be made?


I'll take a look at that linux conversion error. Shouldn't be too hard to find the proper casting method.

Thanks!
__________________
Uleat of Bertoxxulous

Compilin' Dirty
Reply With Quote
  #6  
Old 07-29-2016, 06:10 PM
DanCanDo's Avatar
DanCanDo
Discordant
 
Join Date: May 2016
Location: Above Hell
Posts: 436
Default

Quote:
Originally Posted by Uleat View Post
Did you happen to get a 'control' time so that a comparison can be made?
Sorry, you'll have to give me a little guidance on that one. As in Huh? (chuckle)
Reply With Quote
  #7  
Old 07-29-2016, 06:16 PM
Uleat's Avatar
Uleat
Developer
 
Join Date: Apr 2012
Location: North Carolina
Posts: 2,240
Default

Sorry

A zoning time comparison using the old versus new methods.
__________________
Uleat of Bertoxxulous

Compilin' Dirty
Reply With Quote
  #8  
Old 07-29-2016, 06:20 PM
DanCanDo's Avatar
DanCanDo
Discordant
 
Join Date: May 2016
Location: Above Hell
Posts: 436
Default

Oh, actually yes, I was paying attention to the old vs new. I based this on zoning to moors
only. Before it was 25-33 seconds on average, sometimes higher, but never below 25
After conversion it was a steady 19-22 secs
Reply With Quote
  #9  
Old 07-29-2016, 06:29 PM
Uleat's Avatar
Uleat
Developer
 
Join Date: Apr 2012
Location: North Carolina
Posts: 2,240
Default

Nice! Thanks for that info!


I pushed a (hopeful) fix for that 64-bit compile issue.

Let me know if that does not work.
__________________
Uleat of Bertoxxulous

Compilin' Dirty
Reply With Quote
  #10  
Old 07-29-2016, 06:48 PM
DanCanDo's Avatar
DanCanDo
Discordant
 
Join Date: May 2016
Location: Above Hell
Posts: 436
Default

I just pulled that fix and compiled on the linux box again. No compile errors this time
I'll get the server booted up and get the map conversion done, then test it all out, to
see how the linux server takes to it all.
Reply With Quote
  #11  
Old 07-29-2016, 06:53 PM
DanCanDo's Avatar
DanCanDo
Discordant
 
Join Date: May 2016
Location: Above Hell
Posts: 436
Default

Oh, ok, it rapidly gave me "conversion failed" on all of them, so I didn't get too far there.
Didn't give me any visible errors in window, so I will see if I can some kind of print.
Reply With Quote
  #12  
Old 07-29-2016, 07:37 PM
Uleat's Avatar
Uleat
Developer
 
Join Date: Apr 2012
Location: North Carolina
Posts: 2,240
Default

Not sure what linux should be reporting with a dll failure.

I'll look around and see if I can find a way to trace it to the dll.


EDIT:

The code that was 'fixed' takes the offset of the base pointer and subtracts it from the pointer reference, then divides by the size of the
object to get an 'index' that can be used to deference the base pointer during the load process.

If that index is computed to be '< 0' or '>= mNodeCount' (which should not occur) .. you should be getting an exception error during the LoadMMF call.

Since this is happening during the conversion/save process, it most likely is dll-related..but, will need more info before confirming that.
__________________
Uleat of Bertoxxulous

Compilin' Dirty
Reply With Quote
  #13  
Old 07-29-2016, 08:08 PM
DanCanDo's Avatar
DanCanDo
Discordant
 
Join Date: May 2016
Location: Above Hell
Posts: 436
Default

Is there a way I can get that script to spit out some kind of log ?
I'm just running it with perl command from cmd window in linux, but
it won't give me any kind of feedback other than failed
Reply With Quote
  #14  
Old 07-29-2016, 08:12 PM
Uleat's Avatar
Uleat
Developer
 
Join Date: Apr 2012
Location: North Carolina
Posts: 2,240
Default

The 'status' message actually comes from zone.exe and is passed to the script.

Try invoking it using the manual format and see what happens.

Code:
./zone convert_map abysmal.map
(or whatever the linux command line requires..)
__________________
Uleat of Bertoxxulous

Compilin' Dirty
Reply With Quote
  #15  
Old 07-29-2016, 08:23 PM
DanCanDo's Avatar
DanCanDo
Discordant
 
Join Date: May 2016
Location: Above Hell
Posts: 436
Default

Sorry, all I got was - abysmal.map conversion failed
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is On

Forum Jump


All times are GMT -4. The time now is 11:47 AM.


 

Everquest is a registered trademark of Daybreak Game Company LLC.
EQEmulator is not associated or affiliated in any way with Daybreak Game Company LLC.
Except where otherwise noted, this site is licensed under a Creative Commons License.
       
Powered by vBulletin®, Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Template by Bluepearl Design and vBulletin Templates - Ver3.3