Just as an update, I wanted to let you guys know that I am still working some stuff out before updating it to the SVN. So far, all I can do is log in. After that, my character is just hovering above the ground and cannot move at all. I do see spawns, but they are partially below ground. I was able to get the AA structure aligned well enough that it will load even with sending the AA table. But, for some reason, it doesn't show any AAs in the actual AA window yet.

Before I release it for people to start helping with and playing around with, I would like to get it so the character can at least walk around. So, that is my main focus. Other than that, I have been working on cleaning up and aligning some of the other packet structures so that things work properly.

My other priority is figuring out what a certain opcode that the client is sending me is. For some reason, it is sending 0x7580 to the server about 10X or more per second. I have no clue what the opcode is, but I am hoping that it is just some kind of error from the client that will be fixed by some of the other structure work I am doing. My guess is that it is some kind of underworld message, because the client probably thinks it is under the world.

The last thing I want to do before releasing it is to get the files separated into their own SoF files instead of using the Anniversary files. Then, I want to clean them up a bit so they aren't so sloppy.

So, it might be a few more days before I have anything on the SVN. But, I kinda doubt anyone else will be helping much with it yet at this point anyway. So, it doesn't really matter much when I release it, unless it is fully playable.

They cmpletely revamped the ranking/ names of AA's in SoF as i recall, so that might have something to do with it. Keep up the good work Trev.

I went ahead and stuck what I have so far on the SVN. It is pretty sloppy, but it is the point I am at. I made new SoF patch files for it in the common/patches directory. As I fix more things, I will just continue to add the updates to the SVN when I think something is pretty accurate and complete.

So, if anyone has a copy of SoF and a server, they can now check it out. Though, there isn't much they can do other than log in and float in the air at this point :P Hopefully it won't be long before they can do much more than that :)

You haven't found any snags yet where Sony went in and did major redesign? Sony's normal habbit would suggest they wouldn't redo anything already paid for but I can see a sneak attack on the client combat system or the likes.

There will probably be a big issue with how death is handled since they changed EQ1 to use EQ2esque style release to respawn / wait to get rezed.

would actualy be nice to have eq2 style local respawn spots

I am sure there will be some bumps along the way. But some progress is better than nothing and I think we can get most of the basics functioning. Filling in the rest of the details may be a bit tougher, but I don't think anything is impossible just yet :)

I have already been able to get it to put spawns in the correct places and finally able to move around. I have been experimenting with the newzone struct and that is what has let me start moving finally lol. Unfortunately, after I had it working, I tried adjusting something else and now it doesn't work anymore and I can't seem to get it working again yet. But, I am sure I can get it soon. The newzone struct is probably only off by a very small amount currently. As soon as I have it correct (or at least better), I will get it updated on the SVN. Being able to move is pretty important lol.

Oh, and I dunno about the new death system yet, but I know I already died once. I think it kicked me out of the game when I did though. I don't think it should be too hard to figure out a death window. It should probably work similar to rez or teleports. And, I can use my EQLive account to find exactly what the Live server sends to compare to SoF IDA output and find the opcode.

The code for the 'hover on death' and respawn window is also in the Titanium client as I was playing with it before Christmas, so it's use for SoF client support may also be optional.

In Titanium, OP_RespawnFromHover=0x011d

This is the code I was using to play with it, it is far from usable in it's current form.

When you die, it brings up the respawn box with two hardcoded options, respawn at bind in Pok, or Rez in Great Divide. It doesn't handle the Client reply which is OP_RespawnFromHover, with a uint32, 0 for respawn at bind, or 1 for the Rez option.

I didn't progress it any further because I didn't think it was particularly useful.

It seems I am kinda stuck on another thing I can't quite figure out just yet. I may be wrong, but it seems like some of the packet structures change for the client between a couple different structure versions each time I log in. One of the main ones I have notices so far are the clientupdate packets. I don't know why, but it seems like sometimes it swaps the 0 - 15 part of the struct with the 16-31 section. My guess is this is an attempt for SoE to keep us (or the SEQ project) from being able to make structures that work every time. I also see that the newzone struct seems to change as well unless I am just missing something. For some reason, sometimes NPCs show up exactly where they should be and I can move around just fine, but if I /quit out and log in again, they are almost always all gone. No NPCs show up at all. This could also be an issue with the newspawn struct or opcode though, but I just haven't figured it out for sure. It seems that if I restart the server, the first time I log in, I see spawns, but logging in any time after that, I don't.

I got the clientupdate struct partially figured out. I know for sure where a few things are for it like X Position. The weird thing is that even though the hex coming from the client seems to be reversed, the server still reports the correct location for X when I do a #loc. The odd thing is that even if the hex for the section where X should be in the struct is all 0s in the hex, it still reports properly. I don't know why that is yet.

The other odd thing I have noticed is that wearchange struct size seems to change coming from the client as well. The weird part is that if I change the struct to size 19, which I see coming from the client and then restart the server with the new struct size, the client starts sending wearchange as size 9 instead of 19. Then, if I change it back to size 9 and restart the server (after recompiling of course), the client starts sending wearchange as size 19 again. The only thing I can think of is that maybe the logs for the hex isn't showing up properly in the server logs for some reason. Either that, or I need to make separate wearchange structs depending on if it is incoming from the clients, or outgoing from the server.

I still have plenty of other things to work on at this point, so I am definitely not completely stuck. It isn't anywhere near as bad as when I couldn't get all of the way in game for a month, lol. I will keep working on things from different angles and maybe one of the things will fix my other issues. I am hoping that there isn't some new kinda security or encryption happening that I am not aware of. But, as far as I can tell, SEQ doesn't seem to have a problem even with Live and I don't see anything new and special for them to handle it. So, maybe I am just having other unrelated issues :P

I accidentally a SoF is this bad?

http://i43.tinypic.com/1zb857n.jpg

fueling the war effort.

I think you have been spending too much time around Mort. You are starting to post like him and not making any sense :P

I think i'm going to start crying in a corner now -- feel free to delete our three posts to remove clutter, i'm not sure what I was smoking at that time if you know what I mean~

I would actually recommend creating a branch, like I did with loading the spells via the database. That way, you can upload changes that may sorta work so that others can work on it also, but won't affect the trunk ("stable", if you want to call it that). Then, once it's a little more stable/accurate, we can update the branch back into the trunk.

Given that the patch files are pretty much self contained I don't think it really needs a new branch. At least not unless you need to modify core parts of the server to make it all work, in which case yes I'd agree with a SoF branch.

Any modifications of the core files that are needed shouldn't affect the other clients. So far, the only core file changes I have made are to add a new opcode. The only other change I currently have planned for core files is to add a second way to handle the sendexpzonein opcode. Since Titanium sends the same op and expects it right back, the current files work fine for Titanium. They also work for SoF, but the order in the EQ Debugs is a bit off, which might be a concern. So, I plan to copy the function in client_packet.cpp for sendexpzonein and make a different version for SoF. That won't effect how the other clients work, only how it handles SoF logins. Basically, instead of sending the sendexpzonein and getting it back from the client, SoF will send the send the sendexpzonein and send back the new worldobjectssent opcode and handle it by sending that same opcode back to the client again.

Sorry if that sounds a little confusing, but I have it tested on my test box and it works fine and doesn't change anything about how the other clients currently work. At this time, I don't think another branch is needed really. I will just leave it set so you have to uncomment a #define to enable using/testing SoF. So, by default, servers won't be able to use it unless they make that change. This is really only because of the possible crash issues with the SoF client that should probably be avoided for now on non-test servers.

As KLS said, since the patch files are handled in their own separate way, the SVN updates I make for SoF should be ok.

Right now, I am still working on and testing some structure changes to see if I can get more stuff working. I think figuring out the clientupdate structure is an important one to focus on for now. It is a pretty small struct (only 36 in size), but it uses some split signed/unsigned blocks that I don't really understand yet.

Here is the structure of it that I am currently working on:
The actual packet example from that looks like this:
Really, once the spawn struct and clientupdate structs are worked out, I think that should have most of the struct work done. I am sure there will still be more little things to work out in the other structs, but after that, it will mostly be finding missing opcodes. I still haven't worked out a way to find all missing opcodes just yet, but I have been verifying more new ones almost every day.

If I only had a packet collect from 9/7/2007 or shortly after that, it would make this whole process ALOT easier lol.

That's not really what I was talking about, that all really falls under patch files. For example when we wanted to fix inter-zoning we had to actually modify how you zone as 6.2 sent a different series of packets overall than titanium and so required changes to both the patch files and the zone server. When I talk about "core changes" that's things that aren't contained to a certain version and could potentially affect other versions. =p

Just do us a flavor and make SoF commits contain only SoF related changes! :)

As if I had time to make any other changes while I am working on SoF. This thing is like a full time job that I don't get paid for lol.

I will probably be doing an SVN commit for SoF tonight. There is alot more progress since the initial release. Spawns now show up (missing names though), you can actually move around in the zone instead of being stuck in place, doors spawn and ground spawns spawn (but still need work), and a few new opcodes are verified.

Items are a big one that needs work, but those shouldn't be too bad once I know how to do the serialization like a structure so it sends it the way it needs to. I think it is almost to the point where other people might actually be able to start helping out on it soon. I am probably going to try to write a beginner's guide in the wiki for packet structures and maybe some opcode stuff.

You just described the entire EQemu project!

I do so enjoy documentation~

Back to working on the item serialization again, I broke down items into as close to an exact structure as I can figure out. This is for EQLive, but no new fields have changed from SoF to Live as far as I know, so it should also apply to SoF. This structure may not be 100%, but I think it is at least 95% accurate. This is based off of what I could find from 13th floor and packet collects from EQLive.

Since items are handled in a different way than any other struct, I am not exactly sure how to implement this properly. Currently, the emulator sends items as a long string. For SoF, they need to be sent like any other normal structure. Right now, I can't figure out how to add these structs in without having to mess with the item serialization for other clients. I will keep working on it, but it is a bit above my skill level right now. For starters, I need to figure out how to change the serialization from using MakeAnyLengthString and instead for it to send the actual data as it is written in the struct.

I was able to get Targeting working finally and also able to get names to show up now too. The only other major thing to get working from the spawn structure is size. Most of the rest isn't really required and could be done later at any point. These are a couple of major things that were holding me back from making much more progress, so having those working will help alot to getting the game playable.

The actual process I used to find those settings is a bit of a pain, but seems to actually work fairly well. Basically, I took all of the unknowns in the spawn structure and have been setting each one to send 1s for all fields in them. So, by doing this for each section of unknowns 1 at a time, I can compile the changes and then start the server and log in to observe if there were any differences and what they are.

Here is the actual spawn encoding that I am using to test:

The section noted in blue are the new fields I found and am forcing them to send all spawns with a certain setting in that field. The section noted in green are the actual settings to test the unknown fields. I just uncomment the line of the unknown section I want to test. There are some notes next to some of them as to what I noticed when testing them. There are also numbers next to some as well, which correspond to the section in orange below that. Basically, I tried picking the unknowns that were smallest first. I was doing more than 1 at a time to reduce the number of times I would have to recompile. The numbers tell which ones I turned on for each test number I ran. So, the ones that have 1 next to them were the first ones I tried and the notes in orange below are what I saw as a result from that first test. The notes could have been done better, but at least it was something to reference to help keep track of what was being done and being seen.

The reason I am doing a little write-up on how I did this is so others may be able to use the same technique in the future to help refine the structures even further. Other than being able to figure out the structure from reading the IDA output, which is well above my skill level, this guess and test technique is the only other way I could think of to figure out these needed structure fields.

The same technique should be able to work fine for figuring out the rest of the fields. I think it should be fairly easy to figure out stuff like beard and beard color, face, etc with this technique. So, almost anyone could help refine this list of field positions in the structure.

I will most likely be doing a write-up wiki page on what I know about working with packet structures. It really isn't all too complex, but it is important to understand a few key things that took me a while to figure out on my own.

I will probably do an SVN update with the new targeting and name stuff tonight, or very soon. It is finally getting to the point where it is almost playable. I think getting items working is probably one of the biggest things left to do. I have the item structure worked out, but may need help in finding the best way to implement it.

I was finally able to try attacking a mob for the first time in SoF, and of course it caused EQ to crash :P I am pretty sure the crash was due to the fact that items hadn't been loaded though.

For a while, I was a bit concerned that we would never be able to get SoF functioning well enough to actually use as an upgrade to Titanium. Now, I am fairly confident that all possible issues can be resolved and that it will almost certainly be completed and at least as functional as Titanium is now. It is all really just a matter of time and work. I am really hoping that once I have the game somewhat playable and most systems functioning, that others will start assisting in the project. I know most people think this stuff is well beyond their skill level, but if I can learn it, most people with technical skill can as well. And, I think that if I write a couple of wiki pages to help people understand what to do, it will make a huge difference in who can start helping.

So much like myself, you too have little room for your other hobbies. :(

Just wanted to remind you that AugSlotUnk; is now known as AugSlotVisible.

Thanks. Yeah, I noticed that from 13th floor back when I first started working on serialization. I went ahead and changed it in the structure I posted above so it should be accurate now.

Using the same method I described before, I was able to find the helm field in the spawn structure and also in the illusion structure last night. I think this method could even be useful for finding more stuff in Titanium if we needed it. Even though it is a bit of a pain, it isn't really too hard to do. It just takes time.

I also wanted to mention that I was changing through races last night and noticed that there are considerably more global races in SoF than there are in Titanium. There are a few new races as globals that aren't in Titanium at all, including a dragon model with 5 textures! And also many of the models that exist in Titanium are now set as globals as well like Shilskins, all Pirates (LoY) and at least a few others. This is great for custom servers that want to use a variety of races in custom zones and don't want to require any changes on the client side. I also saw some new global races that were objects. One was a big floating skull with flies and a centipede crawling around it and some of the others looked like runes and floating things. The new races look really cool from what I have seen so far. I am really excited to be able to start making content using them.

The last major thing I want to find from the spawn structure is model size. Other than that, I do need to figure out why texture doesn't seem to work either. I don't see texture as one of the fields in Titanium, so I am guessing that it comes from the appearance packet instead of from the spawn struct.

Items and spawn/client position updates are still the 2 most important things that need to be done right now. Once those 2 are complete, I think the rest will be just filling in the little stuff and working out the details to refine it. Once I can fight and kill an NPC, I think it will be ready for player beta testing. The only things that I think are holding that up right now are the items and position updates I already mentioned. Basically, things are getting really close to being playable.

If anyone is interested in helping, let me know. I could use help from someone with more experience in packet structures. I don't quite understand what unions are. Most importantly, I don't understand how signed and unsigned fields work. Here is an example of the client position update structure from Titanium and it uses signed and unsigned, which seems to combine a into an int32 somehow, but I don't understand exactly how...

That struct is pretty straight foward, but I don't understand the sections in green and how those actually work. The client update comes from the client, so it is easy to look at them and see what is being changed and what the order is. I just don't know how to figure out how to write those green section types.

Here is the struct I am working on for SoF so far:

As you can see, I only have a small bit of that structure verified. I am really just guessing with the sections in green since I have no clue how to figure that part out yet. If someone can explain how those signed and unsigned with sizes work, that may help. All I know is that they all have to add up to 32 for each section of them.

Unfortunately, the positioning stuff is one of the few things that SoE seems to completely switch around almost every time they patch. The only good thing is that it is a fairly small packet, so even just guessing, it shouldn't be way too hard to get it set correctly.

I will probably try to get an update onto SVN tonight. I know there isn't much reason to update it regularly for SoF changes at this point, but when I make major breakthroughs, I think it is worth it. The main reason I am doing the updates at all is that hopefully at some point, it will be working enough that someone else might realize that they could start helping so we could get this expansion done much quicker. I think once people start realizing that SoF will actually get completed, they might be more willing to put some of their own time into helping. Up until recently, I think it was questionable if it would ever work or not, but at this point, I am very confident that it will be able to fully replace Titanium (though Titanium will still be usable of course).

I still don't have an ETA on when it will be playable or widely used on servers, but it is getting close to time to maybe consider picking up that last copy of SoF at your local retailer if you happen to see a copy and have an extra 20$ laying around that you don't mind spending :P At least anyone interested in testing or helping out with the SoF development should definitely get a copy if they find one. Of course, don't blame me if for some freak reason it never gets finalized!

I can help a little here.

This is a bit packed signed integer 4 bytes long. that's 32 bits. The first 10 bits are the animation, hence the animation:10, the second 10 is delta_heading and the last 12 is padding.

Oh, so I need to break that stuff down into actual bits to figure it out? LOL, that is gonna suck. Maybe hexvis will help me lol... I will have to try it later tonight.

I think part of the problem is that I don't know how to calculate stuff like floats. So, when an X, Y, or Z gets sent, they are always floats, but how do I make that into an actual X Y or Z location? If I try converting the hex into decimal, it almost always shows something like 17,000 or higher. I don't know how to figure out the actual loc from that. If I could figure that out, it would probably be easier to break this stuff down. Then, I should be able to just compare my /loc in game with the hex in the packet and figure out what matches with what. The way I was doing it was by moving around and watching what changed more when I did certain things. Needless to say, that was sloppy at best.

It truly is amazing how much of a noob I am with this stuff to have come this far even! :P

And I'll add for signed and unsigned and this is how I look at them, maybe not a technical definition:

signed and unsigned integers are integers that can be positive and negative. A signed integer of a certain size can go from a certain negative value to a certain positive value, depending on the size of the integer so -n to +n. Now an unsigned integer is only positive and can go from 0 to n. In the case above where you have unsigned and signed in the same 4-byte block, you have say the first 10 bits are signed so the value can go from -2^9 to 2^9-1. And then you have the unsigned piece that can go from 0 to 2^20 -1 since it is 20 bits.

Any help?

On a semi-related note. SoF at NewEgg is $6. I just got a copy of it. As soon as I get it I might try to run a separate instance and fool around with it some myself.

Thank you very much, sir!

Wow...

http://www.newegg.com/Product/Produc...20of%20faydwer

Very nice find, Andrew80k! Free shipping too! Honestly, I would recommend that anyone who thinks they will be playing the emulator for a while to come should pick this up now. If you may only be playing it for a short time from now, you probably shouldn't bother, as SoF probably won't be fully playable quite that soon. But, at 6$ on a brand new retail copy with free shipping, I think it is a good idea to pick them up while supplies last! At the very worst, you are out 6$, so it is hard to argue with that, lol.

The only reason I would even consider pushing that people start buying it now is because it is only a matter of time before it is nearly impossible to get a legal retail copy of just like Titanium is now. And even then, it will almost certainly cost more than 6$ to get a legal copy (shipped).

I think the fact that not many people actually have SoF yet is part of the reason I have been solo working on the project so far. The stuff I have been doing isn't rocket science. Sure, some of it is a bit complex, especially for someone who has no idea what to do, but I was able to figure it all out up to this point. I am persistent and consider myself to be a good trouble-shooter/problem solver, so that helps alot. But as you can see, my knowledge of this stuff is very crude. I am certain that others would be able to look at it and figure things out in no time that would have taken me considerably longer. One example is getting items to work. I have already laid out the structure of how the info needs to be sent. All someone has to do is figure out how to send it properly without effecting how the other clients handle items. I can probably figure it out eventually, but someone with more knowledge could probably do it pretty quickly. I don't mind doing any of this work, but with help from others, it can all get done much quicker. Packet structures are pretty simple for the most part. I am still planning to get a simple wiki page up that explains them enough that almost anyone with some technical skill can understand.

And before anyone asks, no I won't be giving refunds to anyone that buys SoF in the case that it never gets completed!

Yeah, some of that stuff is sounding vaguely familiar from reading I have done on it in the past. I just have to figure out how to calculate this stuff. It does look like Hexvis can be a little helpful. I am guessing that the "Real" field in Hexvis is for floats? I don't really understand how to convert X Y Z info into floats to compare my /loc to verify which section might be what. And converting the combined signed and unsigned stuff is even harder. I can just guess and test different structure orders, but I don't even know what size to break up the int32s into for signed and unsigned.

I wouldn't mind reading up on this stuff, but it is really hard to find a good site that explains ints and floats and unions and stuff. Searching for those things yields way too many coding posts, lol.

One thing I am not 100% sure about, but I think I have figured out is that it seems like uints send the hex starting on the left and sints send it starting on the right, though both still read the bits from right to left. What I mean by that is when looking at the hex output from packet collects, I see that some int32 fields of the packet structure are sent like 01 00 00 00, and some are send like 00 00 00 01. Though, maybe it is the floats that are sent backwards like that now that I think about it more. It seems like it is normally X Y Z coordinates or something that is sent that way (00 00 CD AB vs CD AB 00 00).

I just started working on a wiki page for packet structures. It still needs to be formatted and have alot more info added to it, but I think it is a good start. I will try to fill in as much info as I can think of. Feel free to edit it and correct anything that might be wrong, or add to it. Also, feel free to format it so it is easy to read and with an index. I will probably do that at some point, but haven't gotten to it just yet. Here is the link:

http://www.eqemulator.net/wiki/wikka...=PacketStructs

Score~ it was $10 at Amazon but I still didn't want to fork it up just yet...but the $6 + free DHL smart mail = win. I'll help when I can!

Great to hear! Knowing that some other people might actually start trying to help soon, I will make sure to get any progress I make updated to the SVN nightly if it is important enough for an update.

Also, once a couple people are working on it, I will probably start a new thread to keep track of what needs work and what is being worked on by who. This thread is getting a bit long, but it served my needs to keep people updated and maybe explain a little of the process I have taken to progress this far with it :)

I am really excited to be using SoF content hopefully soon :D

I'll spare whatever time I can to this project once my client arrives. PEQ and general EQEmu stuff keeps me busy, but I'd like to see this succeed.

I was wondering why you went for the Secrets of Faydwer expansion pack instead of the new Seeds of Destruction that also is a complete all expansions package.

SoF is currently download only. There is no disk. So you don't have a static client to work from. Working with live is problematic in that it changes so often and it's difficult to keep up. SoF would be a LARGE step toward bringing the emu up to (more) modern EQ. That's quite a bit of additional content and functionality to add. Should keep us busy for a while...

The reason to use SoF instead of SoD is because as far as I have read and heard, SoD is a digital download only. That means you patch to get it. Basically, it means that we would be trying to keep up with Live in order to use SoD unless we picked a certain patch revision and stuck with it, but that brings up legal issues since there is no way for new players to get an old patch legally. And since SoD is digital download only, then I think it is fairly safe to assume that all future expansions probably will be as well.

It was all of that that pushed me to finally decide that this was our last option as an upgrade and it was time to start working on it :P

Also, another reason is that EQLive apparently does something new with their Login Server and will not connect to the server list of EQEmu anymore. I am sure we could fix that if we had access to the source code of the Login Server, but that isn't much of an option. Plus, chasing live is way more work than anyone around here have time for. Otherwise, it would be a million times easier for me to get the emu working with EQLive than it is to get it working with SoF.

I think SoF is a pretty huge upgrade and will be worth all of the work. We are getting almost 2 years of updates over what Titanium has.

BTW, I am going to try to get the zonechange and requestclientzonechange structs working tonight. That way, at least once you guys get your copies of SoF, you will be able to zone around to check out the new zones, and spawn the new races and fun stuff like that. Those structs may depend on clientpositionupdate being correct though, so no guarantees. I was having a blast last night just checking out the new global races. Having a global dragon race is just awesome, and he looks great too! Oh, and I will update #race and #fixmob to both go up to 600 or whatever I find the max race to be. I know it is well into the 500s now at least.

This is going really fast so I apologize, I didn't see either of these issues resolved but maybe I missed it:

equip_chest2 is the field that we fill with "texture".

Also you talked some about serializing items and I'm not sure I understand the problem. We pass an internal serialized item struct to the struct strategy which is:

signed short
ItemInst*

Which we then proceed to turn into a string, turning it into a binary structure instead would be trivial in theory. Is there something I'm missing?