Ok, after re-reading the post that cavedude made earlier in this thread, it looks like eqextractor needs something before it that can actually convert the packets into something it can read to do the extractions from. I am guessing that is what eqcollector is, needed for then.

Really, it seems like quite a hassle to have to use 3 programs to accomplish this lol. The log files from ShowEQ should be pretty simple for a parser to be able to handle pulling the data from. They clearly show the opcodes for each packet and which direction they are being sent from. So, once a parser can identify those 2 things, it should just be a matter of running the data in the packet through a struct to have it converted. Then, once it is converted, it would just need something to tell the program which tables/fields in the database to put each bit of data into.

I am not really sure why these 3 programs are requiring so many extra files and stuff from the source code. Really, I would think that if they are just parsing something and converting it to something else, it should be pretty straight forward and not require a bunch of code to do it. It is too bad we don't have FNW here anymore to maybe give some advise, or at least to get the apps able to compile again :) After that point, I am sure someone can get them working with Live again.

It is silly to have 3 programs, but at the same time it makes some sense. The packet collector produces pf files (I think, it's been some time) which eqextractor is able to read. Eqextractor can then dump the doors, zone data, zone points, books, aas, objects, etc to a sql that source right into the DB. It also has the ability to export to a file format that EQBuilder understands. EQBuilder is what creates the NPCs, spawns, grids, and merchantlists. Its accuracy is really pretty terrific. Eqextractor and EQBuilder can probably be merged into a single program, but that would probably require a bit of annoying work, it might just be easier to get the two of them working as is.

If eqextractor can be updated to import ShowEQ logs, then our collector would no longer be needed. That's probably for the best, as that will be one less program we need to maintain. At that point we would just need to update eqextractor and eqbuilder with the new structs, opcodes, and database structure. I'll talk to FNW and see if he could possibly rewrite the programs so they at least compile in a modern environment. From there, Trevius if you can update the structs and like, I can update the database structure portion of the code and then we all collect the hell out of Live ;)

I was just wondering if Andrew80K was still working on eqcollector/eqextractor/eqbuilder?

I have the source but didn't want to duplicate effort. :-)

I highly doubt it. I think he gave up shortly after seeing how badly the compiles failed :P

I'm still working on it. I actually got it to compile and run fine. I have the opcodes updated and about half way through the structures, but since I'm using ShowEQ to get the structures from and they are not as interested in some of the stuff as we are, the going is kind of slow. I have had to use some of the struct info from the SoF stuff and that's only been recently where it's getting pretty solid. Then I have to sort of wing it.

I should say that I have only got it working on my Linux box. I don't have an MFC compiler to use for the windows compile. Apparently, the MFC classes don't come with the express version. /shrug.

I have vs2k3 pro, it has the MFC classes. I'll go ahead and see if I can get it to compile, at least get the dependancies worked out, then can take the code you have and try to compile it. Let me know if this sounds ok to you or let me know what you need.

Just a thought if its not too much work we'll want to remove the MFC stuff and get it back to pure c++ code? More or less a question. :-)


Ps. I got the source from sourceforge tonight.

That's the crap I'm failing on, so yes!

Agreed. My C++ is kinda rusty so it's taking me a bit longer than I'd hoped. I'll try to see if I can get the first whack at a complete update done by next week. I'll have to come up with a decent way to keep it updated without having to hand jam in all the struct changes and opcode changes everytime. The opcodes aren't too bad as there is a script out there that does and should just take some minor tweaking to get at least workable. The structs are a bit more problematic since they aren't one to one with showeq. Also keep in mind I'm only working on the collector, not the other pieces just yet. First things first. ;)

Wow, this is some really good news. Nice work!

Would it be good to put this into a project someplace so those interested can combine efforts?

Small update....

I got the code mostly updated and working on Linux but I'm still stuck on a windows version. I took a break to try to get the windows compile going. I had it down to 5 errors and in fixing those I now am up to around 70. I'll get it eventually, but I'm going to take a break from that and start my live collects to see where I need to start adjusting the structs and see if I can get it enough along to distribute back out for folks to start collecting...

I have gotten eqemucollector to compile but I am still getting errors on the linking.
This is one of the errors, but I haven't had much time to research them.
I am building it on a virtual pc running xp and visual c++ 6.0. I didn't have any luck building it with visual studio 2003.

I have had to move some files around and things of that nature. I assume this means that the source I downloaded wouldn't have compiled back when it was commited to sourceforge.

Also I am only working on the CLI version at this point. There is a DumpHeaders project that is missing from the cvs. At this point I don't think it is required.

I'm using Visual C++ Pro 2008 to work on it but I'm running into many of the issues you are on Windows I'm sure. Are you removing the MFC stuff? I'm considering going that route, but if you are going to work on the Windows version I may just defer to you and stick with the Linux version since it is actually working...

If you have the collector or any of the other tools compiling under Linux, maybe we should get an SVN setup for them or add them somewhere in the existing SVN. I think all of those programs are pretty small, so it could probably be added directly into the trunk of the /utils directory of the EQEmu server source code. That way, people can work together to get them working easier. I think CD and KLS would be ok with adding you guys to the SVN if you are working on the collector/builder tools.

I assume that if the collector is working under Linux, that it can just watch the packets going to a Windows box if both the Windows PC and Linux PC are connected to a hub. That is how ShowEQ works anyway. I already have this setup going for ShowEQ packet collects, so it would be very easy for me to use it. It would be nice to have the packet sniffer working on Windows to make it so anyone could use it, but the others are fine being Linux only IMO.

Interesting stuff!

Yeah, that's pretty much exactly the way the linux version works. That's been one of my hangups. I couldn't locate my hub. I've been using a pair of routers and the hub was never unpacked from my last move. So it was MIA. I've actually had the linux version running for a few weeks, but couldn't test it until I located the hub. Finally found it and will be starting the collection testing here in the next few days. It is my fervent hope to get the collector out publicly in the next 2 weeks. Matters not at all to me how people want it disseminated. It's up to the community. I can post a diff of the changes, or post it in it's entirety and then someone can update the svn (probably the best way). It uses a bunch of stuff from common, but will need it's own version of the common files since we don't chase live, the common stuff that needs to be changed for it to monitor live can't be changed in the emu version. I have sort of done that, but not completely. I will do it prior to posting it. I'd rather wait until I get some testing done to post it. That way I can adjust the structs/opcodes a bit more that I'm pretty sure are off a little. Shouldn't really be that much changing to be honest.