Starting Work on SoF Opcodes/Structs

[Derision]

Originally Posted by trevius

I'm not exactly sure what the OUT stuff means, but I am guessing that all of the stuff set to go OUT is the stuff that gets encoded. If so, then I wonder what happens to stuff in the structure that isn't set to go OUT in the ENCODE. Does that stuff just get ignored?

The ENCODE/DECODE is used when the struct is different between Client versions.

ENCODE 'encodes' packets on the way out, from the server to the client, and DECODE does the reverse.

The ENCODE is a translation from the struct in common/eq_packet_structs.h to the struct in the client specific struct in patches/<Client version>_structs.h.

To take your example for OP_ManaChange. The 'EMU' struct for this, in common/eq_packet_structs.h is:

struct ManaChange_Struct
{
        int32   new_mana;                  // New Mana AMount
        int32   stamina;
        int32   spell_id;
        int32   unknown12;
};

while the Anniversary struct is:

struct ManaChange_Struct
{
        int32   new_mana;                  // New Mana AMount
        int32   stamina;
        int32   spell_id;
        int32   unknown12;
        int32   unknown16;
};

As you can see, Anniversary has an extra field at the end of the struct.

Essentially, the ENCODE section in Anniversary.cpp is copying the fields from the Emulator version of the struct to the version that Anniversary edition needs. I think the outgoing packet is filled with zeroes before the ENCODE takes place, so unknown12 and unknown16 would be zero.

For each struct that has changed in SoF, you would need to map out the fields in the new structure and add it to Anniversary_structs.h, add the Opcode to Anniversary_ops.h and do an ENCODE in Anniversary.cpp.

You would also need to do the same for Client to Server opcodes where the struct has changed, but this time do a DECODE from the SoF client structure to the Emu struct.

[trevius]

@ greldor - Thanks. I wound up paying to get my old EQLive account active again. I am currently using ShowEQ to watch all opcodes from Client to Server on Live where I need to be looking. Right now, I am still working on just logging in. Once I can get past that point, I think it won't be too hard for me to break down each individual structure that has changed and get them updated as well as start filling in missing opcodes. I have switched directions for now and am going to see if I can get the emu working with EQLive. It should be much easier to get it working with Live, because I can see exactly what the client wants and is expecting from looking at the SEQ logs. Then, even if I don't know what a certain packet is, I should be able to just duplicate what I saw the Live server sending from the SEQ logs and trick the client to log all of the way in until I can fill in the missing information. There isn't really much other people can help with right now until I can get logged in all of the way. Once I do that, I will update the SVN so others can assist in filling in and updating opcodes and packet structures and making any other needed tweaks to finalize it. I am not planning to try to chase Live by making the emu work with it, I am only trying to get in on an EQLive version because it should be easier than SoF. Once I get in game with the EQLive version, I will try to work backwards and get SoF working with it. I think that will be much easier than trying to work from Titanium to SoF, since Live seems much closer to SoF than Titanium is.

@ unknownhost - No, no relation to the name. My avatar name is named after my nephew Trever, but with a fantasy twist to it :P I am definitely no machine, I am just persistent enough to keep going even when I am failing miserably. There are certainly a few people on this project who have much more experience than me and probably would have had this working by now already. Unfortunately, none of them have SoF, and more importantly, I don't think they have the time needed to do it. I am more than willing to do all of the grunt work in getting this stuff to work, but I admit it would be nice if one of the more experienced people were able to help figure out some of the more problematic issues (like currently being stuck at the first main loop). The big time involvement here is getting each of the needed opcodes and verifying they are accurate, and also getting each of the packet structures and verifying they are as accurate as possible. To the point I am currently failing at, I think I have all of the required opcodes and most of the packet structures should be good enough to be working for it. I haven't worked much past that point yet, because there isn't a point to until I get to the point where I need them. I didn't know anything about coding 8 months ago and have only learned by examples from reading the emulator source code and working on it. I have learned alot and learn new things every day, but I am sure the more experienced people would probably be able to help alot if they had the time. I will keep working away at it and will get it eventually, hopefully. All of the time I am working on this has really forced me to learn things I wouldn't have otherwise, so that is actually a good thing, but VERY time involved. It has officially been more than a month that I have been working on this so far, and I have worked on it for hours every single night. I made alot of progress in the first week or so, but since I got stuck on this main loop, I haven't been able to get past it. At this point, I still don't know what is causing it to hang up at that point. It could be anything from a wrong opcode, packet structure, something else missing, or whatever. Without logs directly from a packet collect (preferrably from SEQ) from when the SoF eqgame.exe was being used (patched Sep 7th 2007), it is hard to know exactly what might be missing. That is why I am hoping getting it working for EQLive will help me work back to getting SoF working.

@ Derision - Thanks for the info So, any opcode that isn't added to the patch_ops.h will use the eq_packet_structs.h for their structure? I have tried adding a few new opcodes to the anniversary_ops.h and it requires me to put structs into the eq_packet_structs.h for it to compile. If it is supposed to be using the anniversary_structs.h, then why does it require to have a struct in eq_packet_structs.h as well?

The main thing I am trying to figure out is how to know which packets are encoded/decoded. Is it only the ones that are put in the patch_ops.h and set to be encoded/decoded, or are all packets automatically encoded/decoded by default? The only raw packets I see from live are the acks and some of the sessions and network status stuff, everything else is encoded and some is also combined.

I am also trying to understand why anniversary.cpp seems to be telling the server how to handle the opcode packets, but we also have client_packet.cpp, which has handling instructions for each opcode as well. It seems that anything I add to anniversary_ops.h and anniversary.cpp also has to be added to client_packet.cpp. I am not really too clear on what is going on with that just yet. It seems like there is redundancy in some of this stuff and it makes it more complex for me, heh.

I verified most of the opcodes I need to get logged into the emu with an EQLive client last night. I just have to fill in a couple more and then adjust a couple of structures slightly and I should be able to at least get to the same point I am at trying to get in with SoF. I was able to get almost all of the player profile for EQLive mapped out just by looking at the SEQ hex output. I still have a bit more to do to finish that struct 100%, but it should be pretty accurate when I am done with it, at least for Live. But, Live and SoF have very similar structs. They are much closer than Titanium and SoF are.

[Derision]

Originally Posted by trevius

@ Derision - Thanks for the info So, any opcode that isn't added to the patch_ops.h will use the eq_packet_structs.h for their structure?

First thing to say is that what ShowEQ shows as a 'Decoded' packet has nothing to do with the with the EQEmu ENCODE/DECODE terminology.

You are correct, anything that isn't in <patch>_ops.h just gets passed straight through using the common/eq_packet_structs.h structure for that Opcode.

I have tried adding a few new opcodes to the anniversary_ops.h and it requires me to put structs into the eq_packet_structs.h for it to compile. If it is supposed to be using the anniversary_structs.h, then why does it require to have a struct in eq_packet_structs.h as well?

By putting an Opcode in anniversary_ops.h, you are saying that this Opcode requires the struct to be translated from the version in common/eq_packet_structs.h to the version in Anniversary_structs.h, hence why you need a copy of the struct in both files. If the Opcode is a new Anniversary-Only opcode, you don't need to put it in Anniversary_ops.h, and if there is a struct associated with it, you can just put it in common/eq_packet_structs.h.

The only time you need to put something in Anniversary_ops.h is if the same opcode exists in 6.2/Titanium and/or Anniversary, but the structs are different. In that case you need to put the SoF struct in Anniversary_structs.h and code in Anniversary.cpp to encode the struct from the version in common/eq_packet_structs.h to the version in patches/Anniversary_structs.h to account for a different order of fields in the structure, or new fields, etc.

The main thing I am trying to figure out is how to know which packets are encoded/decoded. Is it only the ones that are put in the patch_ops.h and set to be encoded/decoded, or are all packets automatically encoded/decoded by default?

The only packets that are encoded/decoded for a particular client version are those listed in patches/<patch>_ops.h . All other packets are passed along using the struct in common/eq_packet_structs.h. As I said, encoding/decoding is only required if the packet struct for a client version (SoF in this case) is different than that specified in common/eq_packet_structs.h

The only raw packets I see from live are the acks and some of the sessions and network status stuff, everything else is encoded and some is also combined.

As above, when SEQ says it has 'decoded' a packet, this just means it recoginised the EQ Application Opcode and doesn't correspond to the encode/decode terminology used in EQEmu.

I am also trying to understand why anniversary.cpp seems to be telling the server how to handle the opcode packets, but we also have client_packet.cpp, which has handling instructions for each opcode as well.

Anniversary.cpp isn't telling the server how to handle the packet. It is telling the server how to translate the struct associated with that Opcode from the version that particular version of the client is sending into the structure that is in common/eq_packet_structs.h. Once the struct the client has sent has been translated into the 'emu struct', the Opcode is then passed through to client_packet.cpp to handle.

[trevius]

Thank you VERY much, Derision! Understanding that better should help me quite a bit in getting more work done on this. It sounds like I will just have to look at the current examples of how the conversions from struct to struct depending on the patch version are being done. That sounds like it might be easier to do than I originally thought it was. I will check the files now that I have your explanation and hopefully I can make more sense of what is going on so I can get things adjusted properly where needed. When I got into working on this, I was under the assumption that it would just be finding opcodes for it and redoing structs where needed, but it seems there is a bit more to doing that than I originally thought.

All I know is that when I finally get past the point I have been stuck at for the past few weeks, I am probably going to yell loud enough to wake up my wife and daughter :P

[trevius]

After just looking at it quickly to see what is happening, it gave me a couple more questions;

It looks like unknowns are commented out in the Titanium.cpp ENCODEs, so I am wondering if the unknowns are somehow handled/converted automatically, or if they aren't needed at all. Doesn't it still need to know where to send unknowns out at to fill in the gaps?

The other question is about (OUT) vs eq-> usage. Some of the ENCODEs have (OUT) and then the struct field name. But, some of them have something like this "eq->deity = emu->deity;" instead. My guess is that they are both doing the same thing. If I am right, then "eq" is saying to get the eq_packet_structs.h structure and "emu" is saying to put that item from the structure into this position for this patch version. If I understand correctly, that is the same thing that (OUT) is doing. So maybe the eq = emu way is just the old way?

I have seen some warnings about eq and emu not being used when I am compiling. I am guessing that is because I was making blank structs, which also means blank encodes/decodes. And since they are blank, the eq and emu variables aren't being used, so it warns about it. Now that I think I understand it better, I don't think I should need those blank structs now to begin with. If I want to send a 0 size packet, I guess I can just make the new op and then have it get sent when needed directly from the client_packet.cpp. I don't think I would have to add anything anywhere else for them. I will mess with that more later tonight.

[KLS]

OUT(x) is the same thing as: eq->x = emu->x;

When you do your var allocation at the top it allocates both emu and eq; emu being the source struct from the emulator, eq being the dest struct that's going out on the wire.

That's for outgoing packets, as you imagine it's reversed for incoming. When you're decoding an inc packet IN(x) is the same thing as: emu->x = eq->x; and eq is the source from the wire and emu is the dest to the emulator.

[greldor]

As a side note, If anyone needs SoF to aid Trevius:

Newegg.com has Secrets of Faydwer for $19.99
with free shipping!



Greldor
The Grand Creation

[klamath]

If you just want live packets dumped out into your eqemu server its simple to connect. EQmain.dll is everquest loginserver. Swap the live one with your titanium file and your live patch will allow you to login to eqemu login server. Dunno how far you will get from there tho.

[KLS]

My EQLive folder is 5.63GB, eqgame.exe modified on jan 9th, eqmain.dll modified on jan 9th.

The acct I patched with has only a few expansions I think whatever titanium had with it. =p

[Derision]

To add another data point, I can't connect to the LS with the live client either. I ran the patcher today, but it looks like eqgame/eqmain haven't changed since December:

23/12/2008  10:06         1,015,808 eqmain.dll
23/12/2008  10:05         4,874,240 eqgame.exe

I bought the Anniversary digital download a year or two back, so that's all the expansions I have.

[klamath]

Titanium EQmain.dll is 2005 Live patch is 2008 thats why you are not connecting, So swap your live one with the titanium one then you will connect.

[KLS]

They did change the netcode for the live LS recently but it shouldn't of broken anything. They added the ability to force boot your account offline. The basic code is the same, or should be, very odd that it works for both live and emu for me.

Maybe there's something settings dependent, or as mentioned you can just use the older netcode with the titanium eqmain.

Edit: My patch brings me an eqgame and eqmain.dll that don't match those sizes. Puzzling.