Go Back   EQEmulator Home > EQEmulator Forums > General > General::News

General::News EQemu news posts.

Reply
 
Thread Tools Display Modes
  #31  
Old 02-24-2009, 05:55 PM
trevius's Avatar
trevius
Developer
 
Join Date: Aug 2006
Location: USA
Posts: 5,946
Default

Ok, after re-reading the post that cavedude made earlier in this thread, it looks like eqextractor needs something before it that can actually convert the packets into something it can read to do the extractions from. I am guessing that is what eqcollector is, needed for then.

Really, it seems like quite a hassle to have to use 3 programs to accomplish this lol. The log files from ShowEQ should be pretty simple for a parser to be able to handle pulling the data from. They clearly show the opcodes for each packet and which direction they are being sent from. So, once a parser can identify those 2 things, it should just be a matter of running the data in the packet through a struct to have it converted. Then, once it is converted, it would just need something to tell the program which tables/fields in the database to put each bit of data into.

I am not really sure why these 3 programs are requiring so many extra files and stuff from the source code. Really, I would think that if they are just parsing something and converting it to something else, it should be pretty straight forward and not require a bunch of code to do it. It is too bad we don't have FNW here anymore to maybe give some advise, or at least to get the apps able to compile again After that point, I am sure someone can get them working with Live again.
__________________
Trevazar/Trevius Owner of: Storm Haven
Everquest Emulator FAQ (Frequently Asked Questions) - Read It!
Reply With Quote
  #32  
Old 02-24-2009, 06:20 PM
cavedude's Avatar
cavedude
The PEQ Dude
 
Join Date: Apr 2003
Location: -
Posts: 1,988
Default

It is silly to have 3 programs, but at the same time it makes some sense. The packet collector produces pf files (I think, it's been some time) which eqextractor is able to read. Eqextractor can then dump the doors, zone data, zone points, books, aas, objects, etc to a sql that source right into the DB. It also has the ability to export to a file format that EQBuilder understands. EQBuilder is what creates the NPCs, spawns, grids, and merchantlists. Its accuracy is really pretty terrific. Eqextractor and EQBuilder can probably be merged into a single program, but that would probably require a bit of annoying work, it might just be easier to get the two of them working as is.

If eqextractor can be updated to import ShowEQ logs, then our collector would no longer be needed. That's probably for the best, as that will be one less program we need to maintain. At that point we would just need to update eqextractor and eqbuilder with the new structs, opcodes, and database structure. I'll talk to FNW and see if he could possibly rewrite the programs so they at least compile in a modern environment. From there, Trevius if you can update the structs and like, I can update the database structure portion of the code and then we all collect the hell out of Live
Reply With Quote
  #33  
Old 05-27-2009, 10:30 PM
Sylaei
Hill Giant
 
Join Date: Jan 2007
Posts: 124
Default

I was just wondering if Andrew80K was still working on eqcollector/eqextractor/eqbuilder?

I have the source but didn't want to duplicate effort.
__________________
Syl

"The significant problems we have cannot be solved at the same level of thinking with which we created them."
Albert Einstein
Reply With Quote
  #34  
Old 05-27-2009, 10:38 PM
trevius's Avatar
trevius
Developer
 
Join Date: Aug 2006
Location: USA
Posts: 5,946
Default

Quote:
Originally Posted by Sylaei View Post
I was just wondering if Andrew80K was still working on eqcollector/eqextractor/eqbuilder?

I have the source but didn't want to duplicate effort.
I highly doubt it. I think he gave up shortly after seeing how badly the compiles failed :P
__________________
Trevazar/Trevius Owner of: Storm Haven
Everquest Emulator FAQ (Frequently Asked Questions) - Read It!
Reply With Quote
  #35  
Old 05-27-2009, 11:36 PM
Andrew80k
Dragon
 
Join Date: Feb 2007
Posts: 659
Default

I'm still working on it. I actually got it to compile and run fine. I have the opcodes updated and about half way through the structures, but since I'm using ShowEQ to get the structures from and they are not as interested in some of the stuff as we are, the going is kind of slow. I have had to use some of the struct info from the SoF stuff and that's only been recently where it's getting pretty solid. Then I have to sort of wing it.

I should say that I have only got it working on my Linux box. I don't have an MFC compiler to use for the windows compile. Apparently, the MFC classes don't come with the express version. /shrug.
Reply With Quote
  #36  
Old 05-28-2009, 01:12 AM
Sylaei
Hill Giant
 
Join Date: Jan 2007
Posts: 124
Default

I have vs2k3 pro, it has the MFC classes. I'll go ahead and see if I can get it to compile, at least get the dependancies worked out, then can take the code you have and try to compile it. Let me know if this sounds ok to you or let me know what you need.

Just a thought if its not too much work we'll want to remove the MFC stuff and get it back to pure c++ code? More or less a question.


Ps. I got the source from sourceforge tonight.
__________________
Syl

"The significant problems we have cannot be solved at the same level of thinking with which we created them."
Albert Einstein
Reply With Quote
  #37  
Old 05-28-2009, 01:14 AM
cavedude's Avatar
cavedude
The PEQ Dude
 
Join Date: Apr 2003
Location: -
Posts: 1,988
Default

Quote:
Originally Posted by Sylaei View Post
Just a thought if its not too much work we'll want to remove the MFC stuff and get it back to pure c++ code? More or less a question.
That's the crap I'm failing on, so yes!
Reply With Quote
  #38  
Old 05-28-2009, 11:01 AM
Andrew80k
Dragon
 
Join Date: Feb 2007
Posts: 659
Default

Quote:
Originally Posted by cavedude View Post
That's the crap I'm failing on, so yes!
Agreed. My C++ is kinda rusty so it's taking me a bit longer than I'd hoped. I'll try to see if I can get the first whack at a complete update done by next week. I'll have to come up with a decent way to keep it updated without having to hand jam in all the struct changes and opcode changes everytime. The opcodes aren't too bad as there is a script out there that does and should just take some minor tweaking to get at least workable. The structs are a bit more problematic since they aren't one to one with showeq. Also keep in mind I'm only working on the collector, not the other pieces just yet. First things first.
Reply With Quote
  #39  
Old 05-28-2009, 11:59 AM
Dibalamin
Hill Giant
 
Join Date: Dec 2007
Posts: 182
Default

Wow, this is some really good news. Nice work!
__________________
Retired EMarr
Project1999 Developer
Reply With Quote
  #40  
Old 05-28-2009, 12:20 PM
gaeorn
Developer
 
Join Date: Apr 2009
Location: USA
Posts: 478
Default

Would it be good to put this into a project someplace so those interested can combine efforts?
Reply With Quote
  #41  
Old 06-14-2009, 11:49 PM
Andrew80k
Dragon
 
Join Date: Feb 2007
Posts: 659
Default

Small update....

I got the code mostly updated and working on Linux but I'm still stuck on a windows version. I took a break to try to get the windows compile going. I had it down to 5 errors and in fixing those I now am up to around 70. I'll get it eventually, but I'm going to take a break from that and start my live collects to see where I need to start adjusting the structs and see if I can get it enough along to distribute back out for folks to start collecting...
Reply With Quote
  #42  
Old 06-15-2009, 12:45 AM
Sylaei
Hill Giant
 
Join Date: Jan 2007
Posts: 124
Default

I have gotten eqemucollector to compile but I am still getting errors on the linking.
This is one of the errors, but I haven't had much time to research them.
Code:
EQStreamPair.obj : error LNK2001: unresolved external symbol "class std::vector > StreamDestroyHandlers" (?StreamDestroyHandlers@@3V?$vecto
r@P6AXPBVEQStreamPair@@@ZV?$allocator@P6AXPBVEQStreamPair@@@Z@std@@@std@@A)
I am building it on a virtual pc running xp and visual c++ 6.0. I didn't have any luck building it with visual studio 2003.

I have had to move some files around and things of that nature. I assume this means that the source I downloaded wouldn't have compiled back when it was commited to sourceforge.

Also I am only working on the CLI version at this point. There is a DumpHeaders project that is missing from the cvs. At this point I don't think it is required.
__________________
Syl

"The significant problems we have cannot be solved at the same level of thinking with which we created them."
Albert Einstein
Reply With Quote
  #43  
Old 06-15-2009, 10:54 AM
Andrew80k
Dragon
 
Join Date: Feb 2007
Posts: 659
Default

I'm using Visual C++ Pro 2008 to work on it but I'm running into many of the issues you are on Windows I'm sure. Are you removing the MFC stuff? I'm considering going that route, but if you are going to work on the Windows version I may just defer to you and stick with the Linux version since it is actually working...
Reply With Quote
  #44  
Old 06-15-2009, 05:28 PM
trevius's Avatar
trevius
Developer
 
Join Date: Aug 2006
Location: USA
Posts: 5,946
Default

If you have the collector or any of the other tools compiling under Linux, maybe we should get an SVN setup for them or add them somewhere in the existing SVN. I think all of those programs are pretty small, so it could probably be added directly into the trunk of the /utils directory of the EQEmu server source code. That way, people can work together to get them working easier. I think CD and KLS would be ok with adding you guys to the SVN if you are working on the collector/builder tools.

I assume that if the collector is working under Linux, that it can just watch the packets going to a Windows box if both the Windows PC and Linux PC are connected to a hub. That is how ShowEQ works anyway. I already have this setup going for ShowEQ packet collects, so it would be very easy for me to use it. It would be nice to have the packet sniffer working on Windows to make it so anyone could use it, but the others are fine being Linux only IMO.

Interesting stuff!
__________________
Trevazar/Trevius Owner of: Storm Haven
Everquest Emulator FAQ (Frequently Asked Questions) - Read It!
Reply With Quote
  #45  
Old 06-15-2009, 05:53 PM
Andrew80k
Dragon
 
Join Date: Feb 2007
Posts: 659
Default

Quote:
Originally Posted by trevius View Post
If you have the collector or any of the other tools compiling under Linux, maybe we should get an SVN setup for them or add them somewhere in the existing SVN. I think all of those programs are pretty small, so it could probably be added directly into the trunk of the /utils directory of the EQEmu server source code. That way, people can work together to get them working easier. I think CD and KLS would be ok with adding you guys to the SVN if you are working on the collector/builder tools.

I assume that if the collector is working under Linux, that it can just watch the packets going to a Windows box if both the Windows PC and Linux PC are connected to a hub. That is how ShowEQ works anyway. I already have this setup going for ShowEQ packet collects, so it would be very easy for me to use it. It would be nice to have the packet sniffer working on Windows to make it so anyone could use it, but the others are fine being Linux only IMO.

Interesting stuff!
Yeah, that's pretty much exactly the way the linux version works. That's been one of my hangups. I couldn't locate my hub. I've been using a pair of routers and the hub was never unpacked from my last move. So it was MIA. I've actually had the linux version running for a few weeks, but couldn't test it until I located the hub. Finally found it and will be starting the collection testing here in the next few days. It is my fervent hope to get the collector out publicly in the next 2 weeks. Matters not at all to me how people want it disseminated. It's up to the community. I can post a diff of the changes, or post it in it's entirety and then someone can update the svn (probably the best way). It uses a bunch of stuff from common, but will need it's own version of the common files since we don't chase live, the common stuff that needs to be changed for it to monitor live can't be changed in the emu version. I have sort of done that, but not completely. I will do it prior to posting it. I'd rather wait until I get some testing done to post it. That way I can adjust the structs/opcodes a bit more that I'm pretty sure are off a little. Shouldn't really be that much changing to be honest.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is On

Forum Jump


All times are GMT -4. The time now is 01:41 AM.


 

Everquest is a registered trademark of Daybreak Game Company LLC.
EQEmulator is not associated or affiliated in any way with Daybreak Game Company LLC.
Except where otherwise noted, this site is licensed under a Creative Commons License.
       
Powered by vBulletin®, Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.
Template by Bluepearl Design and vBulletin Templates - Ver3.3