EQBuilder 2.0
 

so,

I finally got off my ass and made a build of eqbuilder 2.0 for release. Havent tested this build a whole lot, not sure I care, but I figured I would put it out none the less. Source should be in CVS.

More info on the wiki:
http://www.eqemulator.net/wiki/wikka...akka=EqBuilder

Wooohooo, I am downloading this now. Yay, now I can update my zones a bit easier! :D thnx FNW

so, I somewhat failed to mention that eqextractor does not work with live, didnt even think about it. Sorry about that.

?
 

No but who cares Im glad to have what ew got ...woohoo now I can decode all these pdf files I have for over a year.

It's not a .PDF decoder
it's a .PF decoder :D

And thanks FNW!! We were all looking forward to seeing a new builder

I just used to it populate 26 zones, works like a charm. The map features are a very nice change.

thanks fathernitwit! :D

I guess I'm not getting it. I read the wikki description, but I'm still not sure what exactly this tool does, or under what circumstances I would use it. Can someone explain?

Basically, I believe the program was used to read packet collects from EQLive and convert them into usable data for EQEmu Server databases. It is my understanding that the program has been non-functional for a long time now. Keep in mind that this thread is almost 3 years old.

I imagine that the source code for this program could be adjusted to work with EQLive to be usable for EQEmu again, but I don't think anyone with the skills required to do that is interested in actually doing it right now. Plus, that would require the source code for the tool and I don't know where to get it. Though, it would be useful for gathering data for the newer expansions.

The code is on the old CVS. Source/eqbuilder folder.

In order to get this working, we'd not only need its code updated to work with Live, but also that of eqextractor (converts live logs to eqbuilder format, and also extracts data such as zone data, zone points, doors, objects - its code is also on CVS) AND, we'd need a working packet collector for Live that exports in a format eqextractor can use. Quite the daunting task, for even somebody with the skills and ability to do so.

I dunno, I would have to look at the source of those programs. Though, I would think that it would mostly just be putting in opcodes to identify the packets then the location of the information we need like X, Y, Z, Race, Etc within the packet structure. It would mostly just be looking at the spawn structure and probably the movement updates and then the ground spawn and door spawn packets I imagine.

I have been looking at the structures enough and on live as well that I could probably figure it out if things work as I think they would. We could use the ShowEQ structures for EQLive to identify where the info we need is. I don't think it would be too hard to even pull that info directly from a ShowEQ log file from EQLive. If I ever get free time to investigate this at some point, I might just have to give it a look. If anything, I would like to at least get door locations since those are really hard to set perfectly manually.

Gosh, Trev. We got it the first 12 times. :D

I've actually looked at these and they aren't that bad. The biggest problem is that I haven't taken the time to really understand the stream and how it works. Someone with some knowledge of the data stream (and some spare time) should be able to get this going again in a fairly short amount of time I would think.

Cleaned up the thread :P Don't worry, it's happen to me as well.

If somebody can get these guys back up and working, the PEQ team can get on Live and collect the hell out of it. Then, I'll merge the dumps into the DB so when the SoF client is farther along, at least the community will have some data for the newer zones.

Haha, I guess it was a bad idea to try submitting a post as a way to verify if the forum is working or not. I have done it before, but normally am able to catch it and clean it up myself before too many people see it :P

I will try to get my hands on the source for eqextractor and eqbuilder maybe tonight. Since I am limited in what I can work on for SoF from work, maybe I can work on these 2 programs in the meantime. At the very least, I think I should be able to assess how hard it will be or if it will even be something I can handle myself or not.

Cavedude, how exactly do I get to the old CVS to get the source of those 2 programs? I am trying to use WinCVS, and entering the following:
And it gives me this error:
cvs [checkout aborted]: authorization failed: server peq.cvs.sourceforge.net rejected access to /cvsroot/source for user anonymous

I definitely want to start looking at these soon to get an idea if I can help with them or not. I just need help figuring out how to get them :P

Unless you made a typo, it looks like you're trying to grab from PEQ :P

I usually just follow the sourceforge page using the plain ole cvs command which says:

I assume you can tack on /Source/eqbuilder and /Source/eqextractor after EQEmuCVS to grab just those two projects. If you do get them working, might as stick them in SVN under utils.

I took a crack at compiling these 2 over the weekend both on Unix and Windows. I don't have the MFC classes with the Express version of the compiler I have for windows and wasn't able to get anywhere close to getting it going there. I made some more progress compiling it on Linux, but it still failed for me, but I'm using gcc 4+ an older version *might* compile it. But it appeared to have some files missing, namely one of the logsys files, but I don't remember which one off the top of my head. I thought I'd take a little different tack and hack up ShowEQ to get it to give me the information that I was looking for and then hack it up some more to log it properly. That might work, /shrug.

The missing files are probably in common, in fact you may need zone too. Might as well grab all of the source.

ShowEQ is extremely limited when compared with the power of eqextractor and EQBuilder. Between the two, they can handle zone data, doors, objects, ground spawns, traps, NPCs with full stats and their paths, merchantlists, etc. Even I sometimes take things like doors and zone data for granted, but looking at these new zones, I'm starting to see how brutal it will be to do it all manually :(

I downloaded the entire codebase again to make sure I was getting the latest. But these 2 haven't been updated in years, and if there is stuff in zone and common it's probably not even going to compile anyway, but that would likely give clues to where things need to be fixed. Is it possible that the files it's expecting no longer exist? Has the logging changed in the last couple of years? If so they may actually just be gone in favor of something else. I've only been on the emu for a year or so actively so I don't recall.

The newest of the files look to be 18 months old, so:

Should grab them all from that time period. I'll try when I get home.

Cool, I got them now. Just from a quick review of the source for eqextractor and eqbuilder, it looks like it may be fairly easy to get them working pretty well. That is assuming there aren't currently any issues with compiling these builds. It looks like all they need are new opcodes and structs to match eqlive. I think combining some of the structs I have made for SoF, and the structs and opcodes from ShowEQ, it shouldn't be that bad at all. At least, that is if that is all there is to updating them. Really, it should only require just a few structures and opcodes to be accurate. Off the top of my head, I think it would mostly just be the spawn, door, objects, positionupdate, newzone and zonepoints structures (and related opcodes) that would be able to get 90% of what we would want.

I will probably start trying them out soon and see what happens.

Anyone have any luck with this yet. I tried getting the Sep 1 version but I'm getting the same error:

I've reverted to using VS 6.0 and MySQL 4.0 on Windows and I am getting there. Tons of code errors but hopefully I'll get it to compile.

As for your error, just add the file to your makefile ;) But, why you are building world? It isn't required to do so.

Well I've made much more progress. I have the collector compiled now, but the extractor isn't happy. I compiled world because it seemed, at least on linux that it was looking for other stuff. Now that I have everything except extractor compiling, I probably don't really need to. But I tried a couple of ways before to get it to compile by itself and I just kept getting dependency issues. I compiled it all and it seems to like that better. Now it's just time to find out where it's broken...

Let me know if you guys can get these to compile. If I am right and all that is needed is updated Struct/Opcodes, that should be considerably easy to do using ShowEQ source.

Since I am pretty busy with SoF still, I don't have much time to work on this just yet. But, if anyone wants to test out if my theory on how to fix them is correct, I would gladly spend a few minutes to update something easy like Doors so you guys could see if it then works to collect doors. If that doesn't work, then there is probably more to it than I am thinking. If it does work though, it should verify that we could have these great tools up and running again in no time. Really, using the ShowEQ source, almost anyone could do it. They keep opcodes that we would want for this in the /conf/zone_opcodes.xml file and the packet structures are in the /src/everquest.h file. ShowEQ and EQEmu use almost the exact same structure setup, so it will almost be just a copy and paste. Other than that, you may need to compare the Titanium struct/field names with the struct/field names from ShowEQ and make sure that they match. For the ones that don't match, you just need to use your best guess as to which Titanium field would be in place of the ShowEQ field. Most of those are very easy to tell though, because it is the difference of something like "Name" and "name" in most cases.

I will definitely take some time after SoF development settles down some and give these tools a shot. But, if anyone wants to have a try at it before then, feel free! I will definitely try to answer any questions anyone working on these might have. Though, I really can't help much with the actual compile. I suck at that stuff, lol. I would mostly just be able to help with Structs and Opcodes since I have been getting alot of experience with them lately :P

I'm still having trouble getting the extractor to compile. There are some files missing from the cvs download that I can't even find in the svn. I tried to copy things around and comment stuff out but I'm still not having any luck. I may just skip it for now and see if I can just get the collector going. Once I have that going and collecting logs I can work on the extractor to get the data out of them. I'm working on it a bit tonight.

Hmm yeah, I can't get either of these even close to compiling either. If you can get either compiling, or if someone can't find a version that does actually compile, I will start trying out adjusting some structs and opcodes in them and running them to see if they might actually be able to get working with Live again. Once the compile works for both, it won't be hard for me to find out just how rough it will be to get them both updated for EQLive again. I imagine it won't be bad at all.

Really, all I need is something that can identify an opcode and then use a packet structure to parse the fields out and convert that to SQL. ShowEQ already has easily readable log files that could be parsed pretty easily. I know that is basically all these 2 programs are doing. If I knew enough about how to make something that could do that, I would make one myself lol. It would just need to be able to convert hex into strings, floats and int8, int16 and int32s. After that it would be all about putting them into a format that could make an SQL file or go directly into the database even.

I figure that if I have to, I may just use ShowEQ to collect the really hard stuff to get set perfectly like doors and manually break each field down from the packet and add them to my database manually. I could probably do doors in about 5 minutes each, which is much less time than trying to place them the hard way lol.

Yeah, just to test it, I just manually converted all of the doors in Crescent Reach into the database and they seem to work perfectly. Everything is in the right place and even lifts work great and are perfectly aligned. I just broke down each packet to fit the packet structure for doors, and then used hexvis to get all of the info from the hex that I needed for the table.

So, if needed, things like this could be done manually with some work, but it would be extremely nice to get the 2 tools working that were made to do this automatically :)

Well, I was able to get the collector to compile under linux. It includes a bunch of stuff from the common files so I've been trying to make sure it was using the latest version of stuff. My next step is to take what I can of the ShowEQ stuff and make sure the structs and opcodes are correct for live and then try to collect to see what that does. Unfortunately, it's probably going to be slow for me for a bit as I just got into the teeth of my graduate classes. I will be trying to get this going over the next couple of weeks though. I'll post as I progress. I haven't gotten the extractor to compile yet, but I suspect that I know what is wrong and I think I have solved it but I haven't tried it yet. It's not much use without the collector working so my priority is on the collector. Once the collector is working again I can pass that to the PEQ guys and they can start collecting the heck out of live zones and I'll start working on the extractor. I don't think the builder will be hard once the others are working. It actually may work just the way it is, but we'll see.

Oh, you are working on eqcollector? I thought were were only talking about eqextractor and eqbuilder2. I think eqextractor pulls the data from the packet collect and puts it in a certain order or whatever and then eqbuilder takes the output from eqextractor and puts it into the database. I am not sure how eqcollector would fit into all of that, but I didn't think it would be useful to us anymore. Plus, IIRC, the structures and stuff work differently there, so it would be harder to get it going I think.

Ok, after re-reading the post that cavedude made earlier in this thread, it looks like eqextractor needs something before it that can actually convert the packets into something it can read to do the extractions from. I am guessing that is what eqcollector is, needed for then.

Really, it seems like quite a hassle to have to use 3 programs to accomplish this lol. The log files from ShowEQ should be pretty simple for a parser to be able to handle pulling the data from. They clearly show the opcodes for each packet and which direction they are being sent from. So, once a parser can identify those 2 things, it should just be a matter of running the data in the packet through a struct to have it converted. Then, once it is converted, it would just need something to tell the program which tables/fields in the database to put each bit of data into.

I am not really sure why these 3 programs are requiring so many extra files and stuff from the source code. Really, I would think that if they are just parsing something and converting it to something else, it should be pretty straight forward and not require a bunch of code to do it. It is too bad we don't have FNW here anymore to maybe give some advise, or at least to get the apps able to compile again :) After that point, I am sure someone can get them working with Live again.

It is silly to have 3 programs, but at the same time it makes some sense. The packet collector produces pf files (I think, it's been some time) which eqextractor is able to read. Eqextractor can then dump the doors, zone data, zone points, books, aas, objects, etc to a sql that source right into the DB. It also has the ability to export to a file format that EQBuilder understands. EQBuilder is what creates the NPCs, spawns, grids, and merchantlists. Its accuracy is really pretty terrific. Eqextractor and EQBuilder can probably be merged into a single program, but that would probably require a bit of annoying work, it might just be easier to get the two of them working as is.

If eqextractor can be updated to import ShowEQ logs, then our collector would no longer be needed. That's probably for the best, as that will be one less program we need to maintain. At that point we would just need to update eqextractor and eqbuilder with the new structs, opcodes, and database structure. I'll talk to FNW and see if he could possibly rewrite the programs so they at least compile in a modern environment. From there, Trevius if you can update the structs and like, I can update the database structure portion of the code and then we all collect the hell out of Live ;)

I was just wondering if Andrew80K was still working on eqcollector/eqextractor/eqbuilder?

I have the source but didn't want to duplicate effort. :-)

I highly doubt it. I think he gave up shortly after seeing how badly the compiles failed :P

I'm still working on it. I actually got it to compile and run fine. I have the opcodes updated and about half way through the structures, but since I'm using ShowEQ to get the structures from and they are not as interested in some of the stuff as we are, the going is kind of slow. I have had to use some of the struct info from the SoF stuff and that's only been recently where it's getting pretty solid. Then I have to sort of wing it.

I should say that I have only got it working on my Linux box. I don't have an MFC compiler to use for the windows compile. Apparently, the MFC classes don't come with the express version. /shrug.

I have vs2k3 pro, it has the MFC classes. I'll go ahead and see if I can get it to compile, at least get the dependancies worked out, then can take the code you have and try to compile it. Let me know if this sounds ok to you or let me know what you need.

Just a thought if its not too much work we'll want to remove the MFC stuff and get it back to pure c++ code? More or less a question. :-)


Ps. I got the source from sourceforge tonight.

That's the crap I'm failing on, so yes!

Agreed. My C++ is kinda rusty so it's taking me a bit longer than I'd hoped. I'll try to see if I can get the first whack at a complete update done by next week. I'll have to come up with a decent way to keep it updated without having to hand jam in all the struct changes and opcode changes everytime. The opcodes aren't too bad as there is a script out there that does and should just take some minor tweaking to get at least workable. The structs are a bit more problematic since they aren't one to one with showeq. Also keep in mind I'm only working on the collector, not the other pieces just yet. First things first. ;)

Wow, this is some really good news. Nice work!

Would it be good to put this into a project someplace so those interested can combine efforts?