SoF Development Tracking
 

I figured I should start a thread to help keep track of the development of SoF. This can be used as a reference for what needs work, and who is working on what. I will try to keep this post updated as people update the thread with any progress or notes. Please try to keep posts in this thread for developer work only. All other SoF development discussion should be posted here, or a new thread created for it. For reporting Bug in Secrets of Faydwer that are not already listed in this post, please post them in the Secrets of Faydwer - Bug Reports thread.


Last Update - March 5th 2010

Top Priority Work:
1. AAs - All AAs in SoF are now fully functional. A temporary work-around is in place for the Improved/Advanced upgraded versions of AAs. For now, we just see double of the base AA for the upgraded version, but they work and can be purchased. This is because SOE consolidated those into single AAs. We just need to figure out how Live handles it to deal with training them higher instead. More notes in the post below.

Medium Priority Work:

Low Priority Work:
1. Adventure Leaderboard - The stats of the Adventure Leaderboard don't work properly yet. I think the correct opcode is set, but it just needs an encode and structure adjustment.
2. Levitate - Levitate seems to cause PCs and NPCs to hop if it was cast prior to you zoning in. Levitate is now supposed to go in the spawn struct as "2" in the flymode field instead of from a separate appearance packet like Titanium uses.
3. Animation - Animations such as sitting, crouching, death do not show up if someone zones in after the animation was already used. This is because of the new StandStance field in the spawn struct for SoF being hard set to 0x64, which is standing. We need to set it to put the actual animation being used in that field instead of forcing it to standing.
4. Inspect Notes - Inspect works, but the player notes in the inspect window do not work for SoF clients yet. I think this may need to be set in the player profile now or something.

Database and Source Code Changes that will be needed at some point:
1. AA Table - The alt_vars table may need further adjustments to allow it to be fully functional for both clients and their differences.

Other Issues:
1. The SoF client seems to have a few crash issues. No real details at this time on what causes crashes, but it is still a bit less stable than the Titanium client is.

New Systems to SoF that did not exist in Titanium:
1. Blocked Buffs - This should be pretty straight forward to get going at some point.
2. Auras - These appear to mostly work like Songs, but also have an aura NPC that can zone with characters.
3. Fellowships - The NPC class that gives fellowships is Class 69. The window opens up for the client already, but nothing is actually coded for handling them yet.
4. Pirate Combat - No Information
5. New Power Source Equipment Slot - Power Source Slot added and stats from it are functional, but it does not yet work the same way it does on Live.
6. Guild Banners - Banners are spawned as NPCs and are a global race. Custom Guild Banner info should be in the spawn structure somewhere.
7. Destructible Objects - No Information
8. Player Traps - These are spawned as new global NPC races. Most likely, they are just quest scripts in the default.pl to control the trap actions.
9. Spheres of Influence - No Information
10. New Skills - Unsure how many new skills there are in SoF, but live has Triple Attack as a new skill (skill 76). Not sure how to add these to work with both SoF and Titanium.
11. Respawn Window - Have started work to getting this added. It isn't functional yet, but getting there.
12. New Item Stats - All possible item stats now show up on items when a value is set in the database. The new SoF stats still need to be coded into the server to actually be used. KLS Added Backstab Damage, but the others still need to be coded.

Recently Resolved Issues:
1. Consumption - Food and Drink consumption is now working and is now consumed at the correct rate.
2. Trading (to NPC or PC) - This is now working.
3. Objects - Interacting with objects (tradeskill, aug pools, etc) is now working.
4. Interrupts - Spell interrupt is now functional.
5. Groups - Grouping should now be fully functional.
6. Banking - Banking is now fully functional including the 8 new bank slots, shared bank and slots within bags in any of those slots. This also resolved a fairly major crash related to banking.
7. Clickies - Right Click item effects now seem to be fully functional, including observing recast delays when they apply as well as using item charges when they apply.
8. Drakkin Corpses - Drakkin and Froglok corpses are now showing up properly after zonedumps (thanks Derision!)
9. Tasks - The Task system is now fully functional (thanks Derision!)
10. Pet Buff Window - Pet buff window is now fully functional (thanks Derision!)
11. Animations - Animations are now fully functional (thanks Derision!)
12. Tracking - Tracking is now fully functional (thanks Derision!)
13. Skill Points - Skill training points now report properly (thanks Derision!)
14. /who - The /who command is now fully functional (thanks Derision!)
15. Item Links - Item Links are now fully functional, including Item Links from command outputs.
16. Item Augments - Item Augments are now fully functional in every way (HUGE thanks to Derision!)
17. Money Trading - Traded money now updates without having to zone (thanks Derision)
18. Bazaar - Bazaar trading is now fully functional (thanks Derision!)
19. Loot Slots - Looting corpses with multiple items will now always loot the correct item
20. Tribute - The Tribute system for personal tributes is now fully functional (thanks Derision!)
21. Rest System - After 30 seconds of being out of combat, extra regen kicks in to regenerate characters mana/hp/end at faster speeds. (completely new system - thanks Derision and drakelord!)
22. Books - Books are now functional (thanks Derision!)
23. Attunable Items - Attunable items now function properly in SoF (thanks realityincarnate!)
24. Spell Interrupts - Spell Interrupts and Fizzles no longer cause abnormally long refresh times on the spells being cast.
25. Stun - Melee stun/bash no longer causes unintended spinning
26. Spawn Structure - All of the currently desired fields should now be identified (thanks for the help, Shendare!)
27. Drakkin Specific Features - All Drakkin Related fields (Heritage, Tattoo, and Details) are now fully functional for NPCs and PCs.
28. Potion Belt - Potion Belt seems to function perfectly fine now.
29. /bug - /bug is now functional (thanks KLS!). Verified that it works in SoF with no modifications required.
30. /petition - The /petition command in SoF has changed to now open a browser to the SOE support site. Apparently /guidehelp has replaced /petition and functions in exactly the same way.
31. Chat Channels - Chat Channels and Mail are now fully functional in SoF with the addition of the new UCS (Thanks Derision!!!)
32. Item Stats - The item table now contains all of the fields for SoF including Heroic Resists which aren't used on Live but are built into the client. All stats now show up on items if a value is set, but no code is in yet for the server to actually use the new stats.
33. Tradeskill Combines - Tradeskill combines now work in tradeskill containers within your inventory like medicine bag or warrior epic 1.0 sword combing/split.
34. Recast Count-Down - The recast count-down on clicky items with recast delays no longer leaves items greyed out all of the time. This feature is available in Titanium as well, but doesn't work for Titanium yet either, so I am marking this off of the SoF specific issues list. Though, it would be really nice to see this get added at some point.
35. Raids - Raids are now functional in SoF (Thanks KLS!)
36. Death Disconnects - Dying in the same zone you are bound in no longer causes you to be disconnected (Thanks KLS!)
37. Guild Management Tool - The guild management now shows members and member notes and should be as functional as the one in the Titanium client (Thanks Derision!)
38. Stamina/Endurance - Live now sends endurance update packets and SoF probably does as well, so it wouldn't hurt to add an update packet for SoF. For now, endurance seems ok in SoF though.
39. Azone2 - Now functional for the EQGv4 zone files, so every SoF zone is now usable, though watermaps still do not work for EQGv3 or EQGv4 zone files yet (Thanks Derision!!!)
40. Max Level 75 - A work-around for level 76+ was added for SoF to correct HP/Mana/End by sending a tribute packet that sends what the stats should be (Thanks KLS/Secrets!)
41. Inspect - Inspecting SoF clients is now functional. The only functionality it is still lacking is the ability to set a note in SoF client inspect windows.
42. Clicky Checks - Items with click effects now have the proper checks for all clients to verify that they are allowed to be used.

Attaining Secrets of Faydwer:
Everquest: Secrets of Faydwer can currently be purchased from www.newegg.com for $19.99 with free shipping.
http://www.newegg.com/Product/Produc...20of%20faydwer
There are also various other sites that have SoF for sale at lower prices than Newegg.

This post will be updated again regularly with current development status.

Detailed Notes on Subjects Above

AAs - Currently, there is a fair amount of work that needs to be done to get AAs fully functional and up-to-date in SoF. Most of them now work thanks to a structure issue that Derision found and corrected. In order to handle AAs properly for SoF, we need to create and use new tables just for SoF AAs. Right now, AAs load into the AA window and can be purchased without having to zone for them to update.

The total spent AA count in the AA window is temporarily being forced to show 500. This is because I do not know how to get an accurate count of the total spent AA points yet. This is a fairly minor issue, but worth mentioning. Derision found an issue with the encode of the RespondAA packet, which is what sends the spent AA points total. So far, I now have it able to show the correct number of AAs purchased, but not the total of AA Points spent to make those purchases.

The main priority on AAs right now is to get the table issues worked out. I have already collected every AA that a rogue gets on Live and have put them into a table format. I was able to get them working in game as well, but need to do some clean up on them before they are ready for other admins to use. The table I have so far for this is posted later in this thread. At some point, we will need to collect the AA info for each class and put them into the table as well. It seems that the AA data for Titanium still mostly exists in SoF (since AAs show up), but SoF seems to get descriptions and names and such from different places than Titanium does. Titanium uses the eqstr_en.txt file to get most of it's descriptions, but SoF uses the dbstr_us.txt file for them.

Something worth mentioning is that it seems like getting AAs set in the database might be easier for SoF. The dbstr_us.txt file is setup differently than the files that Titanium uses for AA names, descriptions, and maybe other info. Basically, it seems to use a list format for each ID. For example; Innate Strength is skill_id 2, and in SoF, the name and description in the table can both be set to 2. This is because the client knows that the name is always the 1st line in the list and description is always the 4th. Here is an example from the file to explain what I mean a bit better:

So, skill_id, title_sid, and desc_sid all get set to 2. This just means a little less work for getting the AAs added to the alt_adv table. Nothing major, but seems like a better way to do it.

I think I have the alt_var table I have so far is looking pretty decent, but still need to look into the aa_effect and aa_action tables. Once all of the AAs are collected for all classes from Live, I am sure we will have to do some clean up to make sure everything matches up and is set properly.

Once the new tables are ready, I will probably need help from someone more experienced to get the SoF encode to use the new AA tables instead of the old ones. We will also need to have it send a couple new fields that didn't exist in Titanium. I figure I will deal with that once the tables are more ready, though it won't be long now.

It seems that with the change to how the AA window works since Titanium, it means that the way that the AA tables are sent works a bit differently. As far as I can tell, none of the actual AA IDs listed in AA.h have changed. But, since many AAs were combined/consolidated to reduce overhead, the way they are handled had to be changed as well. The most noticeable change is for stuff like the Innate stats, and other AAs that used to have direct upgrade line AAs for them, but are now combined into a single AA. The issue is that with the IDs remaining the same, it doesn't allow for them to be combined in the same manor that Titanium handles them individually. To use Innate Strength as an example, the ID for it is set to 2, and the next AA is Innate Stamina, which has an ID of 7. This means that there is only room for 5 levels of that particular AA. In Titanium, the max_level for that AA is 5, so that works fine. In EQLive (and probably SoF), the max is now 15 for Innate Strength. It seems that each time you train a point, it just moves up 1 in the AA effects table. And, to compensate for this restriction, it looks like EQLive just sends that same AA table 2 or 3 times so that the effects stack. So, if you had 1-5 points trained, it would send that AA 1 time, if you had 6-10 trained, it would send it 2 times, and if you had 11-15, it would send that same AA 3 times. I am not 100% sure that this is how it works, but it definitely sends multiples of some of these, and it only seems to do it when you have points trained in that stat. This will definitely be something we will have to understand fully before the new SoF AA tables can be completely finalized. I still have to figure out exactly how each table is supposed to be formatted to handle this new way of doing it. I think most things are still the same, but changes will definitely have to be made here and there.

I just noticed that the unknown0092 field in the SendAA structure seems to actually be a secondary category field. The first category field is currently labeled as "expansion", because it is the number that correlates to the expansion that the AA originated from. The unknown0092 field seems to override the expansion field if it is set to anything other than -1. I am going to do some testing with this in SoF and if it works, I will label the field "special_category". So far, I have identified that it will show the following if the field is set to any of these for SoF:

As with the other new fields, this one will have to be added to the code for SoF AAs to have it sent properly. I already have them listed in the new tables I am working on.

That leaves the unknown0096 field as one of the only new unknown fields that hasn't been identified yet. The only AA I show as setting that field to anything other than 0 is Harmonic Dissonance. That is an AA on live that is free to purchase and will port the player to a zone. It is currently set to 16777216, but I am not sure what that number is supposed to relate to yet. This field probably isn't very important to us.

Recast Count-Down
Clicky items that have a recast delay on them all seem to show up as grayed out all of the time. This may be due to an issue with the serialization portion of the struct, or possibly something in the click effect portion that is not getting set properly. Interestingly, it seems that after my recent changes to the manachange encode, my click type 2 items now count down as they should. This could possibly mean that the manachange packet is what updates items to start the countdown after the click happens. If that is true, then we should just need to figure out the correct values to use in the unknown fields to correct this. I have it hard set to -1 right now, so I don't know why it would effect recast type 2 only. Maybe it is just coincidence that they work now or I may not have noticed that they started working after some other recent change. Either way, if we can figure out why type 2 works, then we should be able to figure out how to get the others all working. And if we can get it working in SoF, it could possibly be fixed in Titanium as well, because I know that feature was in Titanium. This system isn't crucial to get working, but it would be handy to have :)

Azone2 probably wont load some of the newer zones as they moved to version 4 eqg at some point and azone2 can only do <=3.

I'm gonna take a stab at item packets soonish once I can get client setup.

I'm at a loss on how to handle drakkin, since titanium and 6.2 wont be able to display them at all. I guess we could just have them pretend to be human for <SoF

Thanks, KLS. The items are one of the last major remaining items to get working before players may be able to start testing it and reporting bugs. I think I should hopefully have the AA and /who issues worked out really soon. Once items are done, I think combat just needs to get worked out and basic play should be functioning. For all other stuff, it will mostly just be opcode finding I think, which shouldn't be too bad at all.

I am actually surprised how close to being ready this is already. As long as there are no more major bumps, I don't think it is far off at all. Maybe Azone can be updated sometime in the future to work with the new EQG files. Other than that, I think a couple minor code adjustments to handle the new player race and maybe a couple other things, and then a couple minor database adjustments to handle some of the new stuff as well. Then it should be fully functional. At least as well as Titanium is for us now.

And yeah, for Titanium players, they will just have to see Drakkin as a human model or something. The Drakkin race is model 522, which is well beyond what Titanium can use. By default, they would see the frozen arms and legs out human model. But, we can probably force them to be any race in the Titanium/6.2 client cpp files in the spawn encodes. I think it would work doing a race check when the spawn packet is sent, and then just convert any race that is > 473 to be race 1 instead. Really, it isn't a huge issue, because they won't be able to see the new weapon models and stuff that the SoF users will be able to use anyway.

Eventually, I imagine most people will just move onto using SoF instead of Titanium. Especially once it is just as functional as Titanium currently is.

I must say that after going through each of the 81 new zones in SoF that have been added since Titanium, I am really amazed. SoE has done an excellent job with their zone design. I didn't care much for the Buried Sea zones, but other than those, all of the other expansions really wow'd me. And with 103 new races to use, I think this will open up alot of doors for custom servers. I haven't even got to seeing the new weapon models yet, but I imagine they are great as well.

After SoF is done, I will most likely look into seeing if I can possibly fix EQExtractor and EQBuilder2 to work with EQLive again. Then, we can use them to populate many of the new zones. I think that combined with SoF will help keep the emulator thriving for a long time to come. And since it is pretty likely that it will be the final client version for the emulator, it should make it worthwhile to work towards a 100% functional emulator :)

Now that I'm SoF enabled, I'll pick up Character Creation if no-one else is working on it.

Thanks Derision! I am glad to start seeing some dev interest in this project. It has been a pretty huge one so far lol. The amount of hours I have put in over the past couple of months is pretty insane. I was hoping once I got it to a somewhat playable point that others might start realizing that it wasn't just a pipe dream anymore and that it was going to actually get done!

If you need any help or suggestions for character creation let me know. I can easily get ShowEQ logs of the character creation process from EQLive to share with you. Those will be much easier to read than packet sniffs from WireShark or something. That is assuming that EQLive and SoF handle it the same way, but I am pretty sure it would be the same, or at least much closer than how Titanium handles it.

I have been focusing on AAs mostly over the past couple nights. I am pretty sure that the structure is just wrong. I was basing it off of the EQLive AA structure, but I bet SoF was slightly different. The good thing is that there is minimal difference between EQLive and Titanium, so it shouldn't be hard to figure out what to change/remove from the current struct I have that is based off of EQLive.

I have also filled in a few more Opcodes, but haven't updated the SVN with them yet. Probably 70% of the Opcodes we need to fill them all out will come directly from the client in a tail log (as an OP_Unknown), so those will be very easy to get. For the rest of them, they may be a bit trickier, but those can be handled when the time comes.

I got Character Creation working, mostly.

There is a new OpCode the client sends when you click on 'Create New Character'. I got a trace from live and the server responds with a 22KB packet, with what I assume must have valid starting stats, starting zones, deities etc, for each race/class.

I didn't bother trying to interpret what is in the packet just yet, I just have a hardcoded response with the same 22KB of data that live sends which is good enough to get into the character creation screen on SoF.

I can currently only select races from the original game ... the Expansions data must have moved within whatever packet it gets sent in, also the Enter Tutorial button seemed to be missing. Both will hopefully be easy to sort out.

The way the initial start zone is requested also seems to have changed, so everyone starts in South Qeynos for now.

FYI, in case no one noticed yet, Newegg is now selling Secrets of Faydwer for $4.99 with free shipping! That is down a dollar from the recent price of $5.99 with free shipping. If you don't have a copy of it already, you almost certainly won't find a better deal than this and should get it while they are still in stock. There is a link in the first post of this thread to the page on newegg of where to get it.

I've been working on tasks today. Most of the problems were missing/incorrect opcodes, although the task selector struct has also changed slightly.

I have everything functional, although I think one of the structs is still slightly off, leading to the reward item/link sometimes not displaying correctly.

I still need to write the encode so it works with all clients, but should probably be done this weekend.

I'll probably work on the pet buff window after that.

If you are making a hash for the item links, I wanted to note that I confirmed that there are now 5 more 0s in the hash used for Titanium to show up properly in SoF. If you do a #itemsearch in SoF right now, it cuts off the first 5 letters of the name. I added the 0s to it to test and then it worked fine, but then Titanium was showing the 5 0s in the name output. So, I imagine we will have to do something tricky to get it working properly for both. Not sure what the best solution is yet, but I wish there was a way to check if a client was SoF or not. Then, we could just do an if before making the output.

Dunno if that is related to the reward links in tasks, but figured it was worth mentioning.

Thanks for helping, Derision and KLS! It has been a ton of work to get to the point things are at now, but the help you 2 have given has definitely made a big difference. There isn't that much left to iron out now, but, all help is much appreciated!

Thanks for the info on the item links, that may be my problem.

There is a way to check the client version. The client network stream has a method which returns a string describing which patch file that client is using (Client62, Titanium, SoF). I'll post an example of how to check it later today. It would be best though if everything could be done in SoF.cpp, to keep the main source as version agnostic as possible.

EDIT: Lol, this is Derision, just happened to be logged on my alternate account :)

Hah, was wondering if that was you while I was reading that! Funny :)

Yeah, I agree that it would be best if we could do it all in SoF.cpp, but I think it would be tricky in some cases like this where stuff is being hard coded and not put into any sort of structure. Unless maybe we could recreate the same commands with alternate settings in SoF.cpp, I dunno any other way to do it. I was thinking about if it might be possible to set encodes for the special/formatted/common or whatever messages packets that would look for a string that looked like an item link and then add the required 5 0s in there. But, that would be quite a hack IMO, and might cause more issues than it would be worth. One solution I was thinking of would be to make alternate commands just for SoF (#sofitemsearch, #sofpeekinv, etc), but that would be much less than ideal imo. The only other thing I could think of would be to create a rule that would let admins set if they wanted to use SoF settings for stuff like command output and anything else that might need to be changed. Then, we could have a simple rule check, but again that would be less than ideal as well.

I don't think there are many things that would have to be changed outside of the SoF.cpp that would effect the other clients. For most things that are hard coded, I have just been able to just create new opcodes and handling that doesn't effect Titanium users. In this case, that isn't really an option as far as I can think of unfortunately.

Oh, and a client version check function would be nice to have handy for certain situations. I would really like to have a way for a porting NPC to check if the client is SoF or not so it doesn't port Titanium users to zones they don't have. I am sure there are plenty of other times that would come in handy as well if we are going to have 2 compatible and fully functional clients (not counting 6.2).

Here is the new SoF table that I have so far. It still needs some adjustments to be perfect, but it is getting there. This table was made directly from a Live collect while playing a rogue, so it should have every Rogue AA available. Once this is fully functional, I will collect the rest of the classes AA info and get them added to the table. It won't be as much work for the other classes, since they should all have the same General AAs and I think there are only a few Archetype AA changes between classes, so it would mostly just be collecting class specific AAs.

This table is NOT the final version, I am only posting it for development purposes at this point. To test it, you can rename your current AA table and rename this one to "alt_vars" so that it uses it in place of the current one. Once code is written for it, SoF will most likely use AA tables completely separate from the standard Titanium and other client AA tables. I have prefixed the table name with "sof_" in case we require more SoF specific tables in the future so they will all be grouped together if prefixed like this.

And here is the actual table itself:
With some recent discoveries from Derision, SoF AAs are almost completely functional on my test server. There are still a few minor things to work out, but once those are done, the tables will just need to be filled in the rest of the way and they will be ready to go.

Also, I think it would be a very good idea to maybe add 1 more field to this table as a bool. The new field could be called something like "enabled" with 1 being enabled and 0 being disabled (enabled is default). This new field would then be checked before sending the AA with the AA tables. That way, we could populate the entire table to include every single AA, but then decide which ones we want to send to the client so we don't send them a bunch of AAs that have no coding support for them just yet.

That's a really good idea, actually, if it could be worked out in the code. Everyone's well aware that most of the EQEmu project is a work in progress, it'd be nice to save them the trouble of buying non-working AAs or at least be able to warn them of them if possible. At worst (though it wouldn't be quite Live-like), the AA's description could have a note added to explain that it isn't functional yet.

EDIT: On second thought, I suppose a note like that couldn't be added. The actually descriptions aren't loaded from the database, are they? Sorry, I still have a lot to learn.

Well, I am sure we could make a standard setting to either disable them in game or to warn about them. Actually, we could just set a bogus prereq_skill skill for any we want to disable. That is probably the easiest and best way to deal with this. Then, as AAs are coded, we just remove that prereq_skill or set it to whatever it is supposed to be set to. I didn't really think about doing it that way before, but it would be simple and require no extra coding.

I think once the issue with consolidated AAs is figured out, I should be able to collect the rest of the class' AAs and get them all added. Then, it would just mean some clean up to make sure everything is set properly.

Once the tables are ready, we would just need to get it coded to be able to handle separate tables for SoF AAs. I am not exactly sure how to do that, but I am sure it is possible.

I've just started incorporating the mail server functionality into the chatserver (it will be a new program called UCS).

It's complicated a bit by the fact that Titanium clients establish two connections to it, one for mail and one for chat. I've added an extra character to the Mailkey, M = Mail connection, C = Chat Connection, S = (SoF) Combined connection. This is set by world, since it can determine what version the client is.

I should add that in SoF, the friends list is now maintained by the server. If you add or remove a friend it sends a command to the UCS which must be responded to for the friend to be added or removed. The friends list is also sent to the client when a connection to the UCS is established.

Has there been a release of UCS yet ?

Not yet. I have it working, but I need to clean the code up a bit before releasing it. It will probably be a couple of weeks.

Your code still has to look cleaner than mine.

Was actually trying to get the opcodes for ldon objects today.

-On character select my characters hold their items in the opposite hands. Looks kinda odd with shields.

-I can confirm raids don't work; the initial raid create packet does seem to but the add players to raid packet does not.

-Something is off about LDoN objects they can't be sensed/disarmed/picked. On titanium to do this you needed a certain bodytype set but this doesn't appear to be the case. You can cast ldon spells on any target now so I assume this has moved from body type to some other field in the spawn structure.

Thanks KLS. The char select issue with primary/secondary should be really easy to correct by adjusting the struct and the encode and just reversing the order of those 2 fields. I hadn't noticed this, but I also don't have any chars I use that have shields.

Thanks for confirming raids don't work. I was just wondering if anyone had even tried it yet. Judging by the difference in group structs, I figured that raids were almost certainly going to have similar changes. Maybe we can review the group struct changes and use them to help figure out what might have been changed about raids. Otherwise, we will probably need to try to create a raid on Live and collect the packets from it with SEQ.

Judging by the other changes to the spawn struct to break down certain things like Bodytype into multiple categories so that it is easier to set an NPC to spawn exactly how you want, I figured that there are other new fields like that. It shouldn't be too hard to figure out which field is required for LDoN traps by just using spawn struct hacking code that Shendare wrote. It is already in the encode for the spawn struct and just needs to be uncommented to be able to use it. It really makes identifying fields very easy, though a bit tedious since you basically have to go through each unknown until you find the right one. It is still 100X better/faster than the old way I was using to identify fields!

If I have time tonight, I plan to try to finally get the encode working for inspecting players, so that will be 1 less thing on the list of stuff that needs to be fixed for SoF. I don't think that will take very long. I tried it before, but couldn't get the strings working properly. I think with the new encodes from Derision, I should now have enough examples to get the inspect encode working. And, if I have time after doing that, I really want to knock out the stamina/endurance update packets that should now be sent and probably the manachange packets need to be encoded since they should only be dealing with mana now, and not include stamina. That is of course if SoF handles the updates the same way Live does now, which I am pretty sure it does.

After trying to add in code to support separate sets of AA related tables for SoF and Titanium, I found that it was going to be quite a bit harder to do it that way than I initially though. So, I changed the plan to simply add more fields to the existing alt_vars table and try to create a couple work-arounds. Probably 90% of the Titanium AAs are also in SoF. The few that are not, are the upgraded versions of earlier AAs. As explained earlier in this thread, this is because SOE consolidated all upgrading AAs into single AAs with more max train points in them.

It is going to take a bit of thinking and work to get the new idea working properly, but I think it will do what we need with the least amount of issues. I added some essential new fields to the alt_vars table that SoF requires, but I also added 2 new fields that it does not require, but will be used to handle the change from Titanium to SoF.

The new clientver field is for setting which client versions you want to see that particular AA in their AA Window. Here are the list of options for the clientver field atm:

0 = No Client Versions
1 = All Client Versions
2 = Titanium Only
3 = SoF Only

This field is pretty simple and is used for removing the Unknown DB messages in the AA Window for AAs that no longer exist. Currently, only SoF has this issue, but Titanium would have it as well if we ever add SoF only AAs in. This new field makes it easy to resolve it. I probably need to code something in for 6.2 Client, but I didn't yet.

I also made another field that isn't being used yet, but I have an idea about how it might be able to be used. The other field is sof_next_skill. This is essentially the opposite of what the pre_reqskill field does. The sof_next_skill will be populated with the skill ID of the upgraded version of that AA if one exists. I am not sure how to use it yet, but the idea is that we would then send the upgrade, but send it with the skill ID of the original, since I think that is what SoF wants. We would also need to do something similar to save newly purchased AAs to the player profile. This is because we need SoF and Titanium to both match up properly (or at least close) which is another reason I wanted to try using a single table for both. As far as the server is concerned, the player would essentially have the same AAs as they do on Titanium when they are playing on the SoF client. But, the client would be sent some slightly modified info so that it can show itself as having higher max train points in certain AAs that Titanium just gets upgrades for.

Anyway, the first part of this change is now on the SVN. I added an SQL update to update existing tables, but it doesn't really update all skills. So, here is the complete table that is the latest from PEQ and has the added fields that have mostly been modified to be set to what they are supposed to be set to. From what I have seen, it looks like this removed all of the Unknown DB messages and also sets many of the expansions and tabs properly. Note that running this SQL will remove your existing alt_vars table and replace it, so if you have customized AAs, you probably don't want to run this. If you just want the latest AAs, you can run this, but make sure you get the aa_effects and aa_actions from the current PEQ as well to match up:

I actually think I finally may have figured this whole thing out, lol.

Right before going to bed last night, it occurred to me that the way Live currently sends AA information may be the same way they always have, just with a bit of a conversion. Basically, the increase in max_level for each AA that has an increase is causing it to send an additional separate AA table packet with the same skill id. This is a bit hard to explain clearly without showing someone the exact packet stuff I am talking about, which I am not going to do in this post :P

I was thinking about the Player Profile and how to use both clients with the same PP without causing issues. That is why I originally added the sof_next_skill field to the alt_vars table. I was hoping to be able to make some code that would be able to figure out when and where to do conversions similar to how to convert slots between SoF and Titanium, but quite a bit more involved. If my latest theory is correct, it might not be very involved at all!

How it will work is we will need to adjust the alt_vars table to fill in the sof_next_skill field for all AAs that have upgraded versions. I may need to rename that field since if this works, it would actually be more like the pre-req field, accept that all further upgraded AAs would also have the same ID as the previous ones for this field, and that ID is always going to be the original AA that now has increased max_level instead of upgraded versions.

I guess I should just give up on trying to explain this for now, lol. I will just test it and verify if it actually works as I think or not. If so, then I will make the required adjustments and make additional updates to the Wiki in relation to these new fields so people creating AAs will know how to populate them. I think this change might be a very simple one, just doing a simple check to see if the AAs are an upgraded version and if so, then send the skill id of the original AA they are upgrades to instead of the new skill ID. I am sure this post is hard to understand, but my theory makes clear sense to me and it makes sense of why I see things from Live packet collects that I didn't previously understand. It also clears up how to deal with the table changes as well as how to deal with the Player Profile all in 1 fix. There may need to be a few extra tweaks after this to get all AAs fully functional in SoF that are currently working in Titanium, but it shouldn't be anything too major.

After I have the table working correctly for SoF, we will be able to finish off populating it. I am sure some of the AAs will need to be moved to different tabs in the AA window and also some probably have the incorrect expansion listed for them as well, but those are just minor issues. I will post the full table again once I have the upgrade stuff worked out. I am very glad to finally be making progress on AAs in SoF as they are IMO the last remaining major issue with SoF. And even though they might be the biggest issue, probably 90% of them already work, so it isn't even really that huge of an issue.

With a little luck, I will have some good news about SoF AAs later tonight :D

Well, it wasn't as simple as I thought, so I really didn't make any breakthroughs last night. I will give it another shot tonight. I think I am close, but there is just something keeping it from working. Might still have to do some tricky conversion stuff to get AAs working between both clients properly. I am still pretty confident that we can do it using the same tables as they are now setup.

Here are the AA table packets from EQLive that make me think it is just tricking the client:

The first packet is Innate Str, which I didn't have anything spent in, so it just sent the one packet. The second and third packets both fall into the Innate Stamina skill ID range. Those 2 packets are what make me think the server just uses the old separate AA information but sends them twice to stack the effects. Basically, the AA Effects are the same between the 2 aas for Innate Stamina and Improved Innate Stamina. But since the improved version no longer exists in SoF, it simply sends the normal innate 2 times and the client knows to double the stats up. I am sure that the extra effects work, because if I train Chaotic Potential (doesn't exist in SoF) and Planar Power while using a Titanium client, and then log in with SoF, I still get +50 to all caps as if I had both AAs. So, the only thing that is left to figure out is how to stack the AAs so it doesn't show the unknown DB entries in the AA Window. Also, we need to have something that will do conversion stuff for when an AA is purchased above what the base AA allows. An example of what I mean by that is Innate Stamina has a max of 5 train points allowed, but Improved Innate Stamina has 10, so that totals 15 between the 2 which is what should show up for normal Innate Stamina in SoF. But, since they are actually stored separately and sent separately just like in Titanium, we need something to tell the server that after 5 are trained into Innate Stamina that it needs to start putting the remaining 10 points into the improved version.

Hopefully this is starting to make more sense. Anyone who wants to chime in or give a shot at figuring this all out is more than welcome LMAO! I think I am on to something, but just need to get 1 working so I know exactly what needs to be done. Currently, I emptied out the alt_vars AA table on my server and left only 2 entries. One being Innate Str and the other is the improved version of it. This should make it easiest for testing as I only need to worry about getting 1 working. Once one is working, the hard part should be over, but no luck just yet.

Have we thought about using a bitmask instead of a number for clientver? Since we're only working with 3 clients (6.2, Titanium, & SoF), it would only use 3 bits (values 0-7), and allow for more flexibility (and fewer ORs in our IF statements):

That's the way to do it, I think AndMetal. It has the advantage too that if one day another client gets added it can just be assigned 8, and the next 16, etc.

Yeah, I thought about that, but the main reason for having the clientver field is to correct an issue caused by the change in the layout of the AA window and the consolidation of improved AA versions. It would still be good to have a bitmask setup, but atm the only thing it would be good for would be for that it doesn't already do as-is would be to give the option to remove AAs from being sent to 6.2.

Personally, I don't think that there are many (if any) people who run current server code that are using the 6.2 client. And even though it would be nice to continue support for all clients, I don't really know if it is still worth the extra effort. And if we ever had a 4th client, that would just mean that much more work to ensure that all 4 stay fully functional with all updates. I am not saying that I think we should break the 6.2 client completely, I just don't think we should waste time on making sure it gets all of the new features we are adding in.

I also have my doubts that we will ever get a client newer than SoF, which is one of the big reasons I started working on it in the first place. A newer client would require updating the LS code. It would also mean we would need to be able to get retail disks for it, which I don't think SOE plans to do. I guess we won't know for sure until they release the next expansion probably in a couple of months.

That is a whole other topic in itself though. For the matter at hand, if someone wants to change it to a bitmask, feel free. I don't really know what all is needed in order to set one up fully, or I would do it myself.

Another point about not really need a bitmask for the AA stuff is that there will never be a scenario when you need AAs for 6.2 and SoF but not in Titanium. You will almost always have AAs starting in 1 expansion and continuing on through all later expansions. The one exception IMO is Titanium to SoF, since they consolidated AAs to reduce the number of individual AAs by combining improved versions with the base AA they are improving. This means that the number of possible combinations is really limited and even if we added more expansions, we would still not need more than 6 or 7 options. Bitmask is fine if someone wants to add it, but again, I don't think it matters that much.

On a couple of side notes; I played Live for a few hours the other night and made some AAs to help figure out how they are handled differently now. In the process, I found out about SOEs new 51/50 server, which starts players at level 51 with 50 AAs. I don't think that server is up fully yet (been locked when I checked), but once it is, I think it might help a lot for collecting what I need about AAs. Another cool thing I found out about is the new group button on the map that shows the location of your group members. I checked SoF on the emu to see if it was there, and sure enough it was. Not only was it there, but it already works! One more selling point for people to make the switch :)

Trev is right. I am curious if there is anyone left who is running 6.2. Given there was ultra limited time frame under which 6.2 could have been installed, YEARS ago.

Beta has been announced for next expansion. So we might get another retail disc set yet. Knowing SoE, that shower of bloodsucking leeches will slurp the remaining playerbase for every penny / dime / rupee, while its still viable to do so.

We shall see:)

Take the scenario of a complete newcomer to EQ, having to DL a shedload of content on a slow connection. Aint it better to have a disc solution.

I know some BB is fast but for the some it just aint that fast yet! Unless SoE know better~

from what I see SOE makes "all in one" pack every 3 expansions now.

-Titanium*
-Serpent Spine
-Prophecy of Ro
-Aniversary*
-Barren Sea
-Secrets of Faydwer*


so best gues is next all in pack will be either after next expansion, or even after the one after that one.
Which means at least 1 year from now.
Now given the tremendous ammount of effort it took to get SoF working, we do not yet know if it will be fully operational by that time frame to quickly swith to yet another all in one pack.

I also want to point out, that Trev was a starting force behind move onto SoF, and mainly cuase he found it in obudant supply at only 5.95 :D
WHich of course gives opportunity to all curent emu users to cheaply and quickly upgrade their entire software.

Now if next all in one package going to start out at like $45+ - far fewer people will go after it until it significantly drops in price, or become available for download on torents. WHich could be YEARS from now.

Really, the discussion on other clients should be saved for another thread as it doesn't relate to this topic at all. This thread is for tracking SoF development :P

Ok, so I am making to head-way with the AA stuff again. Apparently many of the fields we didn't think mattered in Titanium actually do matter in SoF. So, the tables may need further adjusting.

It seems that some of the problems I have been running into were caused by the code hard setting certain fields instead of using what is in the tables. One thing that surprised me is that the title and description fields aren't even being used lol! Instead, we just have it sending the AA ID.

Since the desc and title fields are already in the tables, we don't have to add those. We will just need to assign the correct values to them, since all of them currently have the wrong settings. Titanium won't care if they are changed, because it doesn't even use them. The values for those on the client comes from the dbstr_us.txt file, but it always matches the skill ID anyway, so it will be very easy for us to set them in the tables if needed. We may not need them at all if I can figure out how to make the server send what the client is wanting.

I will need to collect more info on how the table gets sent when actually purchasing AAs. I am hoping the new 51/50 SOE server will make this easy to do. It starts you at level 51 with 50 AAs, so if they are untrained AAs, I can spend them and watch how it works. Then, it is just a matter of having the code support sending the adjusted table each time to allow it to train past the max points in Titanium.

Right now, I can get it to send a single AA even if I send 2 tables for the same AA matching what I have seen on Live. I can then purchase past the max point on Titanium, but each time I spend a point, it copies the AA in the window and makes a second one of the same AA listed in there. So, after training 10 points, I see 10 Innate Strengths listed. It sends the AA table again each time you spend a point, so I just need to figure out what it is changing each time to make them stack and not separate like they are doing.

Getting closer anyway, and with more research I should be able to at least figure out how they need to be sent to work properly. Once I know that, I may need some coding help to make that happen correctly. Depends on how complicated the system for dealing with it is, I guess.

EDIT:

Actually, things might be easier than I have been thinking. Part of the problem is that the code is set to use max_level to know when to stop sending table updates, which explains why the client doesn't think it can train past the max level set for Titanium AAs in the table. I have started working to add in extra code for SoF clients to let it send them properly with the right max levels for the AAs. Part of the problem is that I still don't understand all of the AA code yet, so I need to get a better grasp on that before I can do what needs to be done. Basically, I am thinking that for SoF, once the char has trained to the max level setting in Titanium for a certain AA, then the next one to be sent would be for the improved version. Basically, buying SoF AAs will be the same as buying Titanium AAs, accept the client will need to be tricked into thinking it is buy more of the base version of the AA. I am pretty sure this is possible to do, but need to figure out what to do to properly trick the client and to get it coded and working.

I started looking into recast timers on clicky items that have recast delays on them along with a recast type. I think I actually got it figured out now, but really don't know how to code it to do what it needs to do.

Basically, the field we have labeled as PotionType in the item Serialization for SoF is actually supposed to be for all items that have a recast delay/type set on them. It appears that this field of the serialization is supposed to pull the recast timers from the player profile for the recast type of that item. I think I have the structure set properly for where to save the recast timers, but I don't know how to set it to save a unix timestamp to those fields of the profile, so it isn't coded yet.

Once the PP is coded to save the timestamps, we would pull those timestamps and send them in the potiontype field (which probably needs to be renamed to something more like LastCastTime) for any item that has a recast delay > 0. This should tell the item whether it should be greyed out/counting down or if it should be shown as ready to use. Setting a time/date definitely stop items with recast delays from being greyed out. Here is the serialization packet of an item I have on Live broken down into the structure that shows an example:

Book of Knowledge
So far, it seems like items with recasttype of 10 will count down properly after being clicked, but the others do not. I believe this is probably something to do with us not sending the manachange or some other spell casting packet with the proper information. I would guess that we need to send the recast type somewhere after casting a spell.

It would probably help to figure the recast stuff out if anyone with a Live account that has an epic 1.5 could collect a packet log of their player profile as well as the item packet for their epic so I can see how those are serialized. It could be that one of the other unknowns sends the recasttype to go along with the last cast time from the player profile.

BTW, thanks to some help from AndMetal while working on a solution for the AA issues in SoF, I was able to figure out a nice temporary work-around for handling SoF AAs until we have a final solution to handle them properly.

Basically, what it does now is it sends the Title and Description of the AAs according to what is set in the sof_next_skill field. In most cases, that info will match the skill ID of the AA. In the cases where we were previously getting the unknown DB string message in the AA window, we can now adjust the table to trick the client a bit to allow full functionality of all AAs. Since the issue was that many upgraded versions of AAs were removed in SoF and consolidated into the base version by raising the max that can be trained in them, we could no longer send the client the descriptions for those removed AAs. The work-around is to force it to send the client the information for the upgraded AAs, but send it with the Title and Description of the Base AA. An example is that it now sends Innate Strength 2 times instead of sending Innate Strength and Advanced Innate Strength. The result is that you will see 2 entries for Innate Strength in the AA window. You can tell which is which by looking at the requirements when you select the AA. The upgraded version will show the requirement to be trained in the base version for a certain amount. It will also normally show a higher level requirement and will remain greyed out until you meet the requirements to train the upgraded version. Essentially, it will work exactly like it does for us in Titanium, it just won't show the upgraded name and description of the AAs.

I have included some SQL changes with the update that should make the AA window look quite a bit better in SoF now as far as the ordering of the AAs and which tabs they are displayed on. I tried to get all of the AAs with Prereq_skills set properly, but it is possible I could have missed one or 2. It is really easy to fix that if you see any. You just look at the number it gives in the unknown DB string message and that number will correspond with the skill ID of the AA it is trying to show. Then, you just look at the prereq that is set for that AA in the altadv_vars table and copy that prereq_skill number into the sof_next_skill column for that AA. That should correct the problem. If it still doesn't show up correctly, you might have to look at the AA that is the prereq and see if it also has a prereq. If it does, you can try putting that AAs prereq into the sof_next_skill field of the AA you are seeing the error message for.

There is still a fair amount of work to be done to get AAs to display as they should in SoF, but this work-around should do for now. I think AndMetal and I were able to figure out what needs to be done and it probably isn't too extremely complicated. Basically, we just need a function that can add up the max_level of any AAs that need to be combined into a single one. It would have to send it to the client that way and would also have to be able to reverse what it did when the client purchases AAs that are combined so it knows where to disperse the points to in the player profile. Once that is working, I think the only other thing we might need to do is to have it send the upgraded versions of an AA immediately after it sends the base version and both of them need to use the same sequence number. I think the sequence number is how the client knows to stack multiple AA tables being sent to it into a single AA being displayed on the AA window. I am still not sure on how to handle the coding for all of this, but hopefully it can be done at some point.

Here is some code that AndMetal wrote that should be a good place to start as far as working on totaling the max_level of multiple AAs:
I am unsure where the best place is to do that calculation, but I know the reverse would need to be done in the buyAA code.

Did some work on SoF raids today, seems they changed the general struct some; which is the struct for the majority of raid actions: everything from adding members to changing loot types is through a single packet.

is the current struct from a quick playing around with #optest it appears the sof struct will look something like:

I'll have to spend more time on it, the opcodes seem to work though and it shouldn't be too hard to encode this packet to make raids work, I just gotta fiddle around with it some more to figure the rest of everything out.

Is the correct raid general struct for sof.

Problem with the raid join though:
OP_RaidJoin=0x0000 #RaidJoin and RaidUpdate seem to be using the same opcode on Live

This is a little problematic: not impossible to circumvent but I doubt this is entirely true because after finding the raid general struct I can see the behavior isn't changed. I can send a raid general action 8 then a raid general action 30 with the group leader in the encode though and it should do the same thing.

Raids should be working in my next commit :smile:

That's awesome, KLS! One more issue with SoF bites the dust :)

I will mark it off the list, thanks.

I figured out a work around for disc on zoning to bind too.

I tried to figure out the can't level over 75 thing but failed, it's gotta be there though: one of the main features of this expansion was the ability to level over 75... =(
I did note some parts of the PP that were unknown and do stuff though.

unknown12852[0]:
if set to 1 you act like a NPC, your hp / mana / endurance is fixed off some value in the PP that I haven't found yet.
You have no experience bar, no face button and your AA window shows monster abilities instead of AA abilities, also when you open the AA window it sends an opcode, probably something like OP_MonsterAbilityRequest.

unknown12864 is a field of data we send fixed data for but it includes resists in there:
const uint8 bytes[] = {
0x01,0x00,0x00,0x00,
0x02,0x00,0x00,0x00,
0x03,0x00,0x00,0x00,
0x04,0x00,0x00,0x00, //cold resist
0x05,0x00,0x00,0x00, //fire resist
0x06,0x00,0x00,0x00, //magic resist
0x07,0x00,0x00,0x00, //disease resist
0x08,0x00,0x00,0x00, //poison resist
0x09,0x00,0x00,0x00,
0x0a,0x00,0x00,0x00, //corruption
0x4b,0x00,0x00,0x00,
0x4c,0x00,0x00,0x00,
0x4d,0x00,0x00,0x00,
0x4e,0x00,0x00,0x00,
0x4f,0x00,0x00,0x00,
0x50,0x00,0x00,0x00,
0x51,0x00,0x00,0x00,
0x52,0x00,0x00,0x00,
0x53,0x00,0x00,0x00,
0x54,0x00,0x00,0x00,
};

I suspect 6.2 and titanium also have resists in the same chunk of data, and we could probably have some way for servers to change the default resist levels if they wanted.

Wow, KLS! Knocking out 2 fairly major issues in 2 days is amazing :D I was actually thinking about going back and trying to do more work on the respawn window, but atm I am working on trying to make commands to spawn/create/despawn objects in real-time. The commands shouldn't be too hard, I just need to get them written up and tested. Once I am done with that, I will probably try to work on the respawn window again if I can figure anything else out with it. If you can stop the client from being disconnected, I might be able to look at what you did and slip the respawn window packet in there. Then, I think we can do the disconnect/zoning when we get the respawn packet back from the client when they make their selection. The packet handling for the opcodes is in place, it just doesn't have code to actually do anything atm.

Once you have the fix for disconnects up on the SVN, I can remove that one from the list of issues as well. The disconnect was the actual issue. The respawn window is just a new feature that we can put in at any point, but it isn't an actual issue. I think the main point is to get caught up to be as fully functional as Titanium is and we can add in the new features for SoF at any time after that.

There may be a couple more issues with SoF that I don't have on the list just yet, but I haven't confirmed any of them yet. I have seen reports of randomly disappearing items, but I have been unable to verify it and don't really see what would cause it yet. I also noticed that some particular items I have on my server that are set to stack are not stackable. They also are unable to be moved around in the inventory unless you place another item from your cursor into their spot, then it will pick them up and they can be moved. I am pretty sure this is still an issue with the serialization header, but I haven't been able to figure this one out yet. While I was working on it, I did happen to find that the spot we thought was for potions_type is actually for setting the last cast of the items' recast_type, and should be pulled as Unix Time from the Player Profile. I did add the recast types field into the player profile, but I don't know how to make it save the current time upon being cast just yet. Sorry, kinda going off on a tangent here :P

Edit: Just saw the extra notes you made about the PP!

Those are some nice finds!!! I know that we had to be storing resists there somewhere. I think probably half of the PP is for shrouding, since most of it gets ignored due to other packets overwritting the info when they come in after the PP is sent. So, the NPC field you found is probably either for toggling shrouding off and on or for the old code where they made it so players could become NPCs and fight players briefly years ago, or maybe for both.

I think ShowEQ has a few more shrouding structures that we don't have or use yet. I may have copied a couple over into the SoF_structs.h file, but not all of them I am sure. If we ever try to get shrouding working, we could probably find some good info from ShowEQs files for it.

That is really interesting that we get a monster ability window that already has abilities in it without even having to send tables to do it like AAs and such. I assume it works more like skills, where we don't really need to send anything for them to be listed, the client just knows them and it is a static list.