Finding Opcodes

[trevius]

I have been working the past week to try to understand the processes of finding opcodes in an attempt to help fill out some missing ones for Titanium. And, to see if it would be possible to find the Anniversary opcodes without having packet collects from Live when Anniversary was being used.

The wiki pages we have on finding opcodes are far from complete and don't give quite as much information as I would like.

Opcode Questions:

1. Are all opcodes 2-way? What I mean is; are the opcodes that the client sends to the server the same ones that the server sends back to the client? I need to know if there is something we actually MUST have live packet collects for, or if everything we need should be somewhere in the client itself. If everything is in the client, then maybe we can find them without using Live packet collects.

2. Do all opcodes change with every release of the client, or do some stay the same? Did SoE purposely change them so that it was hard for programs like MQ, Showeq, and EQEmu to keep up with live? Or is it the source that changes these automatically?

3. What would it take to get the packet collector updated so that it could at least collect opcodes as they came in? Does anyone have the source for any of the Packet Collectors? I imagine that if the packet collector has some logic or filters to pull opcodes from packet sniffs, then even if the packet collector itself would be too much work to get updated, maybe we could use another packet sniffer like Wireshark and just put a filter on it to only pull opcode packets. Using Wireshark, I can definitely see the opcodes, but without already knowing what you are looking for, it would be near impossible to get the ones that are still unknown.

My Tests So far:

Here is what I did to start trying to find them using Wireshark. Note that this is a learning process for me, so I used one that we are already aware of (Rewind - 0x4cfa):

I loaded up Wireshark and ran EQ (connecting to my EQEmu Server, NOT Live) and then I set a couple of filters just to keep unnecessary traffic from showing up. Mainly, I set it to only show UDP packets and also only anything from 192.168.1.101 or to 192.168.1.100, which are my server and client IPs on my LAN. This blocked out most other web or whatever traffic so I was only seeing client to server packets. Then, I got EQ ready by creating a hotkey for /rewind. At this point, I started the sniffer collecting packets, and immediately switched back to EQ and spammed the crap out of my /rewind hotkey for about 30 secs. Then, I stopped the packet sniff so I could analyze.

What I found was that the client was sending alot of packets to the server that had the opcode in them, but the opcode was backwards with the FA first and then the 4C, so it was showing up as "FA4C". But, this code wasn't just standing alone, it was mixed in with packets that were mostly 17 in size, but sometimes varied to be even bigger. For the most part, the FA4C showed up right before the last 4 characters in the data of the packet. So, if the packet was (not a real example), "FB 04 16 00 A9 90 00 20 B8 09 C2 85 3B FA 4C 8B 20", then the text in bold is where the opcode normally was (but not every time).

More Notes:

I can't seem to get the emu collector to run on my PC, even for Titanium. But, it seems to be a problem with it not finding my NIC card for some reason, and not the version I am trying to collect from. But, I think that if I could get it running, maybe it would show all of the missing opcodes we need just by sniffing the communication from emu server to emu client. So, the emu collector packet sniffers might not need to be updated very much to get to the point that they are needed. Though, it would probably be more work to get them updated to sniff Anniversary.

Another thing I noticed is that some of the log files actually make note of opcodes, and I think even ones that are unknown. So, the source must have some way to convert these so they are easily readable. Maybe that part of the source could be used to help develop a working opcode packet collector.

Idea to get Anniversary Edition working:

My idea for Anniversary is that if we had a working sniffer that can see the opcodes and pull them out so they are easy to find, then we could use a step-by-step process to get anniversary working. By that, I mean that we could start an Anniversary client while already running a packet sniff and then collect the opcodes as they come in. It will get to a point where it fails, and we just find the opcode before that and figure out which opcode it is and add it to the .conf file and then start again. Basically just trying over and over until it gets logged in. Then, it is just figuring out the commands and what not, which should probably be much quicker since you shouldn't have to restart every time it fails.

Another possibility would maybe be using IDA Pro to check the code of the client for opcodes listed. But, so far I have been completely unsuccessful in finding even a single known opcode in there. And, I have done quite a lot of searching and research. Though, my understanding of disassembly files is still very weak. If I could figure out how to get opcodes this way, I think it would make this a piece of cake.

Some references I used when researching how to find opcodes:

Method for hacking from the hackersquest forums that has some good disassembly information:
http://hackersquest.org/boards/viewtopic.php?t=3112&highlight=opcode

Page in the wiki for finding Opcodes, that I can't really make much sense from at all:
http://www.eqemulator.net/wiki/wikka.php?wakka=DevOpcodeFinding

ASM Hunting:
http://www.eqemulator.net/wiki/wikka.php?wakka=DevAsmRoutines

Assembly Tools and Information Links:
http://www.eqemulator.net/wiki/wikka.php?wakka=DevAsmTools

HexVis - A little tool for converting hex or other data to different types of output:
http://www.eqemulator.net/wiki/wikka.php?wakka=HexVis

To convert Decimal to Hex, you can just use windows Calculator and set it to Scientific mode. Then, put the number you want to convert and change the radio button to hex and it automatically converts it.

Also, the eqstr_us.txt and eqls_us.txt both have some information in them that may relate to opcodes and could help in defining what the opcode itself relates to in the code. The hackersquest post above mentions the lines in these files. Though, I haven't been able to find their related hex versions in the ASM of the eqgame file.

If anyone out there has info that might help this process, please respond here or PM me. I am sure we could all appreciate a working Anniversary edition, and it would be worth the work to get to that point. Once we find a good procedure for it, I don't think it will be too bad to get these. But, for now, I am still trying to refine a way to actually gather the opcodes without having to do too much guessing.

[trevius]

After doing a bit more research on Anniversary Edition, it seems like we might be almost too late to use it to put a lot of work in getting an updated version of the emu working. It seems like copies are still easier to get than Platinum, but apparently many stores have already stopped selling the Anniversary Edition and it can even be a little hard to find any online. I am sure it will get even harder soon.

Here are some links to places that should normally be easy to get games from:

http://www.newegg.com/Product/Product.aspx?Item=N82E16832135022

http://www.ebgames.com/Catalog/ProductDetails.aspx?sku=647003

http://www.amazon.com/Everquest-I-The-Anniversary-Edition/dp/B000OU66JU/ref=pd_bbs_sr_1?ie=UTF8&s=videogames&qid=1214879400&sr=8-1

As you can see, it appears that the games aren't being shipped much if at all anymore. It seems like they are just trying to sell off remaining stock and that will be it.

I don't quite understand how SoE expects people to get their game if it isn't on shelves or even available to buy the CDs online. I don't suspect that everyone would want to just get a downloaded game file to play it.

The only way I see to reliably get the game is online from SoE directly as a download only. But, we would need to decide on the version to find opcodes for so everyone could get it.

https://store.station.sony.com/

Click on Everquest under the Digital Download section on the lower left. Then, you have to login and you should be able to click cancel on the subscription section. That will take you right to the digital downloads.

I am wondering if those are just standard unpatched downloads, which I imagine they are. If so, then for 20$ we could have the option to go with the ""Starter Pack" which seems to be the exact same thing as Anniversary Edition. Or for 40$, we could go with "The Serpent Spine All-In-One" package that includes every expansion currently out.

If I can figure out how to find opcodes well enough, I will either just get a copy of anniversary, or maybe one of these download packages and start working on finding the opcodes right away. Though, I would prefer an actual CD pack so we don't have to worry about SoE changing the opcodes by changing the versions of the download files. Maybe they will have another CD release soon :P

[Andrew80k]

From what I recall, vaguely, they downloads are, or at least used to be, unpatched versions. They don't update the downloads.

[KLS]

We use a protocol similar to the one used in eq2 I believe though it really isn't documented anywhere outside of the code. It's kind of difficult to put together via the code because of the layers of abstraction on abstraction. It would just be a matter of decoding the packets further to get them to load from something like WireShark.

Usually the opcodes change from exe version to version though some don't; sony does seem to like to change a lot of the ingame ones every patch; probably to hinder stuff like showeq.

Usually if the client sends a packet and expects a direct response packet for it they will use the same opcode; example would be pickpocket or click object.

It's a shame nothing ever came of openeq and we still have to deal with this. =/

[John Adams]

Heh KLS... this sounds hauntingly familiar.

[KLS]

Sir, I thought I was leaving you alone with this!

Anyway basically the packets are encrypted or compressed, there's a flag on the session request for the stream that indicates which it will be; I believe it can be both as well. The server generates a key to send to the client to decode the data as well.

You have larger packets that make up the protocol, things like session requests, whole packets and combined packets etc. Contained within can be the smaller packets, encoded with the key sent from the server etc.

I think we send the key always as 11223344 but obviously were you to do this from live or something you would need to retrieve the actual one to decode the data into readable packets.

in common/EQPacket.cpp there are several functions that deal with compression and encoding of the data as well.

[trevius]

Thanks for the info KLS, it seems like it has been a while since anyone has considered taking on opcodes and I wouldn't mind giving it a try as long as I can find a viable option for it.

Here are some code sections from the source common directory that just skimming through caught my eye as being related to helping find opcodes:

packetfile.h - Guessing this holds the string you were mentioning that the server sends the client, maybe?
#ifndef PACKET_FILE_H
#define PACKET_FILE_H

#include "../common/types.h"
#include <stdio.h>
#include <time.h>
//#include <zlib.h>

//constants used in the packet file header
#define PACKET_FILE_MAGIC 0x93a7b6f6
#define OLD_PACKET_FILE_MAGIC 0x93a7b6f7

#define PACKET_FILE_CURRENT_VERSION 1

#pragma pack(1)
//old structs from when I forgot to put the version number in
struct OldPacketFileHeader {
uint32 packet_file_magic;
uint32 packet_file_stamp;
};
struct OldPacketFileSection {
uint16 opcode;
uint32 len;
};

struct PacketFileHeader {
uint32 packet_file_magic;
uint16 packet_file_version;
uint32 packet_file_stamp;
};

struct PacketFileSection {
uint16 opcode;
uint8 flags; //mainly for client->server, but others could be added
uint32 tv_sec;
uint16 tv_msec;
uint32 len;
};
#pragma pack()

#define TO_SERVER_FLAG 0x01
#define SetToClient(pfs) pfs.flags = pfs.flags&~TO_SERVER_FLAG
#define SetToServer(pfs) pfs.flags = pfs.flags|TO_SERVER_FLAG
#define IsToClient(pfs) (pfs.flags&TO_SERVER_FLAG == 0)
#define IsToServer(pfs) (pfs.flags&TO_SERVER_FLAG != 0)


class PacketFileWriter {
public:
PacketFileWriter(bool force_flush);
~PacketFileWriter();

bool OpenFile(const char *name);
void CloseFile();

void WritePacket(uint16 eq_op, uint32 packlen, const unsigned char *packet, bool to_server, const struct timeval &tv);

static bool SetPacketStamp(const char *file, uint32 stamp);

protected:
bool _WriteBlock(uint16 eq_op, const void *d, uint16 len, bool to_server, const struct timeval &tv);

//gzFile out;
FILE *out;
bool force_flush;
};


class PacketFileReader {
public:
PacketFileReader();

virtual bool OpenFile(const char *name) = 0;
virtual void CloseFile() = 0;
virtual bool ResetFile() = 0; //aka rewind

virtual bool ReadPacket(uint16 &eq_op, uint32 &packlen, unsigned char *packet, bool &to_server, struct timeval &tv) = 0;

time_t GetStamp() { return(time_t(packet_file_stamp)); }

//factory method to open the right packet file.
static PacketFileReader *OpenPacketFile(const char *name);

protected:

uint32 packet_file_stamp;
};

class OldPacketFileReader : public PacketFileReader {
public:
OldPacketFileReader();
virtual ~OldPacketFileReader();

bool OpenFile(const char *name);
void CloseFile();
bool ResetFile(); //aka rewind

bool ReadPacket(uint16 &eq_op, uint32 &packlen, unsigned char *packet, bool &to_server, struct timeval &tv);

time_t GetStamp() { return(time_t(packet_file_stamp)); }

protected:

//gzFile in;
FILE *in;
};

class NewPacketFileReader: public PacketFileReader {
public:
NewPacketFileReader();
virtual ~NewPacketFileReader();

bool OpenFile(const char *name);
void CloseFile();
bool ResetFile(); //aka rewind

bool ReadPacket(uint16 &eq_op, uint32 &packlen, unsigned char *packet, bool &to_server, struct timeval &tv);

time_t GetStamp() { return(time_t(packet_file_stamp)); }

protected:

//gzFile in;
FILE *in;
};


#endif

EQPacket.h - Adding this here for reference
#ifndef _EQPACKET_H
#define _EQPACKET_H

#include "BasePacket.h"
#include "EQStreamType.h"
#include "op_codes.h"

#ifdef STATIC_OPCODE
typedef unsigned short EmuOpcode;
static const EmuOpcode OP_Unknown = 0;
#else
#include "emu_opcodes.h"
#endif

using namespace std;

class EQStream;
class EQStreamPair;

class EQPacket : public BasePacket {
friend class EQStream;
public:
virtual ~EQPacket() {}

uint32 Size() const { return size+2; }

virtual void build_raw_header_dump(char *buffer, uint16 seq=0xffff) const;
virtual void build_header_dump(char *buffer) const;
virtual void DumpRawHeader(uint16 seq=0xffff, FILE *to = stdout) const;
virtual void DumpRawHeaderNoTime(uint16 seq=0xffff, FILE *to = stdout) const;

void SetOpcode(EmuOpcode op) { emu_opcode = op; }
const EmuOpcode GetOpcode() const { return(emu_opcode); }
// const char *GetOpcodeName() const;

protected:
//this is just a cache so we dont look it up several times on Get()
//and it is mutable so we can store the cached copy even on a const object
EmuOpcode emu_opcode;

EQPacket(EmuOpcode opcode, const unsigned char *buf, const uint32 len);
// EQPacket(const EQPacket &p) { }
EQPacket() { emu_opcode=OP_Unknown; pBuffer=NULL; size=0; }

};

class EQRawApplicationPacket;

class EQProtocolPacket : public BasePacket {
friend class EQStream;
friend class EQStreamPair;
public:
EQProtocolPacket(uint16 op, const unsigned char *buf, uint32 len) : BasePacket(buf,len), opcode(op) { }
// EQProtocolPacket(const unsigned char *buf, uint32 len);
bool combine(const EQProtocolPacket *rhs);
uint32 serialize (unsigned char *dest) const;
EQProtocolPacket *Copy() { return new EQProtocolPacket(opcode,pBuffer,size); }
EQRawApplicationPacket *MakeAppPacket() const;


virtual void build_raw_header_dump(char *buffer, uint16 seq=0xffff) const;
virtual void build_header_dump(char *buffer) const;
virtual void DumpRawHeader(uint16 seq=0xffff, FILE *to = stdout) const;
virtual void DumpRawHeaderNoTime(uint16 seq=0xffff, FILE *to = stdout) const;

protected:

static bool ValidateCRC(const unsigned char *buffer, int length, uint32 Key);
static uint32 Decompress(const unsigned char *buffer, const uint32 length, unsigned char *newbuf, uint32 newbufsize);
static uint32 Compress(const unsigned char *buffer, const uint32 length, unsigned char *newbuf, uint32 newbufsize);
static void ChatDecode(unsigned char *buffer, int size, int DecodeKey);
static void ChatEncode(unsigned char *buffer, int size, int EncodeKey);

uint16 GetRawOpcode() const { return(opcode); }

uint32 Size() const { return size+2; }

//the actual raw EQ opcode
uint16 opcode;
};

class EQApplicationPacket : public EQPacket {
// friend class EQProtocolPacket;
friend class EQStream;
public:
EQApplicationPacket() : EQPacket(OP_Unknown,NULL,0) { app_opcode_size=2; }
EQApplicationPacket(const EmuOpcode op) : EQPacket(op,NULL,0) { app_opcode_size=2; }
EQApplicationPacket(const EmuOpcode op, const uint32 len) : EQPacket(op,NULL,len) { app_opcode_size=2; }
EQApplicationPacket(const EmuOpcode op, const unsigned char *buf, const uint32 len) : EQPacket(op,buf,len) { app_opcode_size=2; }
bool combine(const EQApplicationPacket *rhs);
uint32 serialize (uint16 opcode, unsigned char *dest) const;
uint32 Size() const { return size+app_opcode_size; }

virtual EQApplicationPacket *Copy() const;

virtual void build_raw_header_dump(char *buffer, uint16 seq=0xffff) const;
virtual void build_header_dump(char *buffer) const;
virtual void DumpRawHeader(uint16 seq=0xffff, FILE *to = stdout) const;
virtual void DumpRawHeaderNoTime(uint16 seq=0xffff, FILE *to = stdout) const;

protected:
int8 app_opcode_size;

private:

EQApplicationPacket(const EQApplicationPacket &p) : EQPacket(p.emu_opcode, p.pBuffer, p.size) { app_opcode_size = p.app_opcode_size; }

};

class EQRawApplicationPacket : public EQApplicationPacket {
friend class EQStream;
public:
EQRawApplicationPacket(uint16 opcode, const unsigned char *buf, const uint32 len);
uint16 GetRawOpcode() const { return(opcode); }

virtual void build_raw_header_dump(char *buffer, uint16 seq=0xffff) const;
virtual void build_header_dump(char *buffer) const;
virtual void DumpRawHeader(uint16 seq=0xffff, FILE *to = stdout) const;
virtual void DumpRawHeaderNoTime(uint16 seq=0xffff, FILE *to = stdout) const;

protected:

//the actual raw EQ opcode
uint16 opcode;

EQRawApplicationPacket(const unsigned char *buf, const uint32 len);
};

extern void DumpPacket(const EQApplicationPacket* app, bool iShowInfo = false);


#endif

StructStrategy.cpp - Maybe this is where the Opcodes get encrypted/decrypted?
#include "debug.h"
#include "StructStrategy.h"
#include "logsys.h"
#include "EQStream.h"
#include <map>


//note: all encoders and decoders must be valid functions.
//so if you specify set_defaults=false
StructStrategy::StructStrategy() {
int r;
for(r = 0; r < _maxEmuOpcode; r++) {
encoders[r] = PassEncoder;
decoders[r] = PassDecoder;
}
}

void StructStrategy::Encode(EQApplicationPacket **p, EQStream *dest, bool ack_req) const {
EmuOpcode op = (*p)->GetOpcode();
Encoder proc = encoders[op];
proc(p, dest, ack_req);
}

void StructStrategy::Decode(EQApplicationPacket *p) const {
EmuOpcode op = p->GetOpcode();
Decoder proc = decoders[op];
proc(p);
}


void StructStrategy::ErrorEncoder(EQApplicationPacket **in_p, EQStream *dest, bool ack_req) {
EQApplicationPacket *p = *in_p;
*in_p = NULL;

_log(NET__STRUCTS, "Error encoding opcode %s: no encoder provided. Dropping.", OpcodeManager::EmuToName(p->GetOpcode()));

delete p;
}

void StructStrategy::ErrorDecoder(EQApplicationPacket *p) {
_log(NET__STRUCTS, "Error decoding opcode %s: no decoder provided. Invalidating.", OpcodeManager::EmuToName(p->GetOpcode()));
p->SetOpcode(OP_Unknown);
}

void StructStrategy::PassEncoder(EQApplicationPacket **p, EQStream *dest, bool ack_req) {
dest->FastQueuePacket(p, ack_req);
}

void StructStrategy::PassDecoder(EQApplicationPacket *p) {
//do nothing since we decode in place
}




//effectively a singleton, but I decided to do it this way for no apparent reason.
namespace StructStrategyFactory {

static map<EmuOpcode, const StructStrategy *> strategies;

void RegisterPatch(EmuOpcode first_opcode, const StructStrategy *structs) {
strategies[first_opcode] = structs;
}

const StructStrategy *FindPatch(EmuOpcode first_opcode) {
map<EmuOpcode, const StructStrategy *>::const_iterator res;
res = strategies.find(first_opcode);
if(res == strategies.end())
return(NULL);
return(res->second);
}

};


/utils/throwpackets.pl - Not sure what this script does, but it sounded like it could be something Opcode utility related.
#copy this file into your server's dir
#include this file with: require "throwpackets.pl";
# and add:
# command_add("throwfile", "[opcode name] [filename] - Send a file's hex contents as a packet", 250);
#to your commands_init to enable this


sub throwfile {
my $op = shift;
my $file = shift;
my $p = FileToPacket($op, $file);
if(!$p) {
$client->Message(13, "Unable to read file or parse contents.");
return;
}
$p->SendTo($client);
$client->Message(0, "Sent.");
}

sub HexToPacket {
my $op = shift;
my $hex = shift;
my @lines = split(/\r?\n/, $hex);
my $body = "";
my @pieces = ();
foreach my $l (@lines) {
if($l =~ /[0-9a-fA-Fx]+:\s*(.*)\s+\|/) {
$l = $1;
}
$l =~ s/\s+-\s+/ /g;
$body .= $l;
}
foreach my $p (split(/\s+/, $body)) {
push(@pieces, "0x$p");
}
my $p = new PerlPacket($op);
$p->FromArray(\@pieces, $#pieces+1);
return($p);
}

sub FileToPacket {
my $op = shift;
my $file = shift;
my $c = "";
open(F, "<$file") || return(undef);
while(<F>) {
$c .= $_;
}
close(F);
return(HexToPacket($op, $c));
}

utils/asmtools/stringids_to_ida.pl - Guessing by the name of this and other .pl files in this directory that these are scripts to help pull info from the IDA Pro files created by examining the game file. Now, if I could only figure out how to use these and if they are updated properly to work with Titanium or maybe later expansions.
#!/usr/bin/perl

#reads the output of locate_stringids.pl from stdin
#produces a .IDC file on stdout which we can feed to IDA

print "#include \"idc.idc\"\n\nstatic main() {\n";

while(<>) {
s/\r?\n//g;
next unless(/^([0-9a-fA-F]+) Sends (.*)/);
next if(hex($1) == 0);
my $off = hex($1);
my $str = substr($2, 0, 200);
$str =~ s/"/\\"/g;
printf("\tMakeComm($off, \"$str\");\n");
}

print "}\n\n";

/utils/asmtools/opcodes_to_ida.pl - Same info as noted above
#!/usr/bin/perl

print "#include \"idc.idc\"\n\nstatic main() {\n";

print "\tauto id;\n";
print "\tDelEnum(GetEnum(\"EQOpcode\"));\n";
print "\tid = AddEnum(GetEnumQty(), \"EQOpcode\", 0);\n";

while(<>) {
next unless(/^(OP_[^= \t]+)=(0x[0-9a-fA-F]+)/);
next if($2 eq "0x0000" || hex($2) == 0);
printf("\tAddConstEx(id, \"$1\", $2, -1);\n");
}

print "}\n\n";

/common/logtypes.h - I noticed that this file contains a bunch of settings for log files. I don't know if this is used anymore, or if this was the old way of logging that has an option in the variables for setting log levels or something.
#ifndef LOG_CATEGORY
#define LOG_CATEGORY(name)
#endif
#ifndef LOG_TYPE
#define LOG_TYPE(cat, type, default_value)
#endif
#ifndef ENABLED
#define ENABLED true
#endif
#ifndef DISABLED
#define DISABLED false
#endif




LOG_CATEGORY( CHAT )
LOG_TYPE( CHAT, SAY, DISABLED )
LOG_TYPE( CHAT, EMOTE, DISABLED )
LOG_TYPE( CHAT, OOC, DISABLED )
LOG_TYPE( CHAT, GROUP, DISABLED )
LOG_TYPE( CHAT, GUILD, DISABLED )

LOG_CATEGORY( SPAWNS )
LOG_TYPE( SPAWNS, MAIN, DISABLED )
LOG_TYPE( SPAWNS, CONDITIONS, DISABLED )
LOG_TYPE( SPAWNS, LIMITS, DISABLED )

LOG_CATEGORY( AI )
LOG_TYPE( AI, ERROR, ENABLED )
LOG_TYPE( AI, WAYPOINTS, DISABLED )
LOG_TYPE( AI, BUFFS, DISABLED )
LOG_TYPE( AI, SPELLS, DISABLED )

LOG_CATEGORY( QUESTS )
LOG_TYPE( QUESTS, PATHING, DISABLED )

LOG_CATEGORY( SPELLS )
LOG_TYPE( SPELLS, LOAD, DISABLED )
LOG_TYPE( SPELLS, LOAD_ERR, DISABLED )
LOG_TYPE( SPELLS, CASTING_ERR, DISABLED )
LOG_TYPE( SPELLS, CASTING, DISABLED )
LOG_TYPE( SPELLS, EFFECT_VALUES, DISABLED )
LOG_TYPE( SPELLS, RESISTS, DISABLED )
LOG_TYPE( SPELLS, STACKING, DISABLED )
LOG_TYPE( SPELLS, BARDS, DISABLED )
LOG_TYPE( SPELLS, BUFFS, DISABLED )
LOG_TYPE( SPELLS, PROCS, DISABLED )
LOG_TYPE( SPELLS, MODIFIERS, DISABLED )

LOG_CATEGORY( FACTION )

LOG_CATEGORY( ZONE )
LOG_TYPE( ZONE, GROUND_SPAWNS, DISABLED )
LOG_TYPE( ZONE, INIT, ENABLED )
LOG_TYPE( ZONE, INIT_ERR, ENABLED )
LOG_TYPE( ZONE, WORLD, ENABLED )
LOG_TYPE( ZONE, WORLD_ERR, ENABLED )
LOG_TYPE( ZONE, WORLD_TRACE, DISABLED )

LOG_CATEGORY( TRADING )
LOG_TYPE( TRADING, ERROR, ENABLED )
LOG_TYPE( TRADING, CLIENT, DISABLED )
LOG_TYPE( TRADING, NPC, DISABLED )
LOG_TYPE( TRADING, HOLDER, DISABLED )

LOG_CATEGORY( INVENTORY )
LOG_TYPE( INVENTORY, ERROR, ENABLED )
LOG_TYPE( INVENTORY, SLOTS, ENABLED )

LOG_CATEGORY( TRADESKILLS )
LOG_TYPE( TRADESKILLS, IN, DISABLED )
LOG_TYPE( TRADESKILLS, OUT, DISABLED )
LOG_TYPE( TRADESKILLS, SQL, DISABLED )
LOG_TYPE( TRADESKILLS, TRACE, DISABLED )

LOG_CATEGORY( TRIBUTE )
LOG_TYPE( TRIBUTE, ERROR, DISABLED )
LOG_TYPE( TRIBUTE, IN, DISABLED )
LOG_TYPE( TRIBUTE, OUT, DISABLED )

LOG_CATEGORY( AA )
LOG_TYPE( AA, ERROR, ENABLED )
LOG_TYPE( AA, MESSAGE, DISABLED )
LOG_TYPE( AA, IN, DISABLED )
LOG_TYPE( AA, OUT, DISABLED )


LOG_CATEGORY( DOORS )
LOG_TYPE( DOORS, INFO, DISABLED )

LOG_CATEGORY( PETS )
LOG_TYPE( PETS, AGGRO, DISABLED )

LOG_CATEGORY( COMBAT )
LOG_TYPE( COMBAT, ATTACKS, DISABLED )
LOG_TYPE( COMBAT, TOHIT, DISABLED )
LOG_TYPE( COMBAT, MISSES, DISABLED )
LOG_TYPE( COMBAT, DAMAGE, DISABLED )
LOG_TYPE( COMBAT, HITS, DISABLED )
LOG_TYPE( COMBAT, RANGED, DISABLED )
LOG_TYPE( COMBAT, SPECIAL_ATTACKS, DISABLED )
LOG_TYPE( COMBAT, PROCS, DISABLED )

LOG_CATEGORY( GUILDS )
LOG_TYPE( GUILDS, ERROR, ENABLED )
LOG_TYPE( GUILDS, ACTIONS, ENABLED )
LOG_TYPE( GUILDS, DB, DISABLED )
LOG_TYPE( GUILDS, PERMISSIONS, DISABLED )
LOG_TYPE( GUILDS, REFRESH, DISABLED ) //inter-zone refresh comm
LOG_TYPE( GUILDS, IN_PACKETS, DISABLED )
LOG_TYPE( GUILDS, OUT_PACKETS, DISABLED )
LOG_TYPE( GUILDS, IN_PACKET_TRACE, DISABLED ) //hex dumps
LOG_TYPE( GUILDS, OUT_PACKET_TRACE, DISABLED ) //hex dumps

LOG_CATEGORY( CLIENT )
LOG_TYPE( CLIENT, ERROR, ENABLED )
LOG_TYPE( CLIENT, DUELING, DISABLED )
LOG_TYPE( CLIENT, SPELLS, DISABLED )
LOG_TYPE( CLIENT, NET_ERR, ENABLED )
LOG_TYPE( CLIENT, NET_IN_TRACE, DISABLED )

LOG_CATEGORY( SKILLS )
LOG_TYPE( SKILLS, GAIN, DISABLED )

LOG_CATEGORY( RULES )
LOG_TYPE( RULES, ERROR, ENABLED )
LOG_TYPE( RULES, CHANGE, ENABLED )

LOG_CATEGORY( NET )
LOG_TYPE( NET, WORLD, ENABLED )
LOG_TYPE( NET, OPCODES, ENABLED )
LOG_TYPE( NET, IDENTIFY, ENABLED )
LOG_TYPE( NET, IDENT_TRACE, ENABLED )
LOG_TYPE( NET, STRUCTS, ENABLED )
LOG_TYPE( NET, STRUCT_HEX, ENABLED )
LOG_TYPE( NET, ERROR, ENABLED )
LOG_TYPE( NET, DEBUG, DISABLED )
LOG_TYPE( NET, APP_TRACE, DISABLED )
LOG_TYPE( NET, APP_CREATE, DISABLED )
LOG_TYPE( NET, APP_CREATE_HEX, DISABLED )
LOG_TYPE( NET, NET_TRACE, DISABLED )
LOG_TYPE( NET, NET_COMBINE, DISABLED )
LOG_TYPE( NET, FRAGMENT, DISABLED )
LOG_TYPE( NET, FRAGMENT_HEX, DISABLED )
LOG_TYPE( NET, NET_CREATE, DISABLED )
LOG_TYPE( NET, NET_CREATE_HEX, DISABLED )
LOG_TYPE( NET, NET_ACKS, DISABLED )
LOG_TYPE( NET, RATES, DISABLED )

LOG_CATEGORY( DATABASE )

LOG_CATEGORY( COMMON )
LOG_TYPE( COMMON, ERROR, ENABLED )
LOG_TYPE( COMMON, THREADS, ENABLED )

LOG_CATEGORY( LAUNCHER )
LOG_TYPE( LAUNCHER, ERROR, ENABLED )
LOG_TYPE( LAUNCHER, INIT, ENABLED )
LOG_TYPE( LAUNCHER, STATUS, ENABLED )
LOG_TYPE( LAUNCHER, NET, ENABLED )
LOG_TYPE( LAUNCHER, WORLD, ENABLED )

LOG_CATEGORY( WORLD )
LOG_TYPE( WORLD, CONFIG, ENABLED )
LOG_TYPE( WORLD, INIT, ENABLED )
LOG_TYPE( WORLD, INIT_ERR, ENABLED )
LOG_TYPE( WORLD, CLIENT, ENABLED )
LOG_TYPE( WORLD, ZONE, ENABLED )
LOG_TYPE( WORLD, LS, ENABLED )
LOG_TYPE( WORLD, CLIENT_ERR, ENABLED )
LOG_TYPE( WORLD, ZONE_ERR, ENABLED )
LOG_TYPE( WORLD, LS_ERR, ENABLED )
LOG_TYPE( WORLD, SHUTDOWN, ENABLED )
LOG_TYPE( WORLD, CLIENTLIST, DISABLED )
LOG_TYPE( WORLD, CLIENTLIST_ERR, ENABLED )
LOG_TYPE( WORLD, ZONELIST, ENABLED )
LOG_TYPE( WORLD, ZONELIST_ERR, ENABLED )
LOG_TYPE( WORLD, CLIENT_TRACE, DISABLED )
LOG_TYPE( WORLD, ZONE_TRACE, DISABLED )
LOG_TYPE( WORLD, LS_TRACE, DISABLED )
LOG_TYPE( WORLD, CONSOLE, ENABLED )
LOG_TYPE( WORLD, HTTP, ENABLED )
LOG_TYPE( WORLD, HTTP_ERR, ENABLED )
LOG_TYPE( WORLD, PERL, ENABLED )
LOG_TYPE( WORLD, PERL_ERR, ENABLED )
LOG_TYPE( WORLD, EQW, ENABLED )
LOG_TYPE( WORLD, LAUNCH, ENABLED )
LOG_TYPE( WORLD, LAUNCH_ERR, ENABLED )
LOG_TYPE( WORLD, LAUNCH_TRACE, ENABLED )

#undef LOG_TYPE
#undef LOG_CATEGORY

I am curious if there is a way to have the emu itself just log all opcodes instead of having to sniff them. Since it already encodes/decodes the opcodes and seems to already show some opcodes in the logs that are unknown, maybe all opcode information could get sent to one log file to help find them. Then you could just run a tail (in linux - "tail -f /home/server/logs/opcodes.log") to watch the opcodes come in real-time. If you are the only person on the server, it should be pretty easy to figure out what opcodes do what by just watching which ones come in when you do something. So, if you type /rewind, you should see opcode 0x4CFA come in right at the same time and you will know that those 2 are related. Then, it is just a matter of doing things to verify which opcode is for what one at a time. You will probably want to do some of them multiple times to make sure you have the right one.

It would also be nice if that was working and you could somehow filter out all of the known opcodes, so only ones you are looking for (unknowns) will show up to make finding them much easier and quicker. Then, once we were able to get enough opcodes to get a client logged all of the way in, the rest would probably be pretty easy.

[trevius]

Hmm, I may have answered my own question about using the logs to find opcodes. To watch them as they come in, simply open a terminal window in Linux and type:

cd /home/eqemu/server/logs
tail -f *.* | grep OpCode

You could probably add more stuff to grep for, but this seems to be a start at least lol. I already see some opcodes coming from my dozens of players that are unknowns. Now, if I can get a test server setup so that it is only me that is on, I can start testing and see if my idea works for defining opcodes. There are too many people on my server to use the system for testing without giving them all the boot for a while, so I will have to get a second one running.

Unfortunately my other PC is windows only, and I don't know how to do a tail in windows so I can grep for OpCodes only. So, I may have to build my other PC as dual boot to Debian Linux. Unless I can figure out how to run another server from the same PC I run my main server on. So, I will have 1 that only I can use. Maybe if I make another directory and set the config to use different zone ports... I dunno, but I don't think it would talk right on port 9000 with both server sessions wanting to use the same port to communicate with the login server.

[trevius]

Looks like I might be able to do this on my server even with players on it after-all. This tail shows the name of the char that sent the opcode, so if I am testing, I will just look for my own.

It has already found a few unknown opcodes from my players and these seem to be somewhat common.

OP_Unknowns:
0x6a5f
0x45ff
0x7085
0x3b21
0x1241

I will do some testing tonight to see if I can force it to produce more opcodes and hopefully get some of the unknowns defined. I will let you all know of any progress I make. If this actually works, it is far easier than I could have ever expected lol. Maybe I was thinking too hard, but I imagine that if it was this easy, someone else would have been using this method already...

[AndMetal]

I took a quick peek into the source, and it looks like this is where those OpCode errors you found are spit out:

zone/client_packet.cpp (http://eqemulator.cvs.sourceforge.net/eqemulator/EQEmuCVS/Source/zone/client_packet.cpp?revision=1.78&view=markup#l_375):

375 case CLIENT_CONNECTED: {
376 ClientPacketProc p;
377 p = ConnectedOpcodes[opcode];
378 if(p == NULL) {
379 char buffer[64];
380 app->build_header_dump(buffer);
381 mlog(CLIENT__NET_ERR, "Unhandled incoming opcode: %s", buffer);
382 if(app->size<1000)
383 DumpPacket(app->pBuffer, app->size);
384 else{
385 cout << "Dump limited to 1000 characters:\n";
386 DumpPacket(app->pBuffer, 1000);
387 }
388 break;
389 }
390
391 //call the processing routine
392 (this->*p)(app);
393 break;
394 }


You could relatively easily put some code in there to spit out the OpCode info to the terminal (cerr (http://www.cplusplus.com/reference/iostream/cerr.html) is used elsewhere in the source). Then again, looking just above the above code, someone already has (http://eqemulator.cvs.sourceforge.net/eqemulator/EQEmuCVS/Source/zone/client_packet.cpp?revision=1.78&view=markup#l_337):

337 #if EQDEBUG >= 9
338 cout << "Received 0x" << hex << setw(4) << setfill('0') << opcode << ", size=" << dec << app->size << endl;
339 #endif
340
341 #ifdef SOLAR
342 if(0 && opcode != OP_ClientUpdate)
343 {
344 LogFile->write(EQEMuLog::Debug,"HandlePacket() OPCODE debug enabled client %s", GetName());
345 cerr << "OPCODE: " << hex << setw(4) << setfill('0') << opcode << dec << ", size: " << app->size << endl;
346 DumpPacket(app);
347 }
348 #endif


So, you could set EQDEBUG=9 to get console output of what looks to be all of the OpCodes. This is done in the zone makefile (http://eqemulator.cvs.sourceforge.net/eqemulator/EQEmuCVS/Source/zone/makefile?revision=1.17&view=markup#l_15) before compiling:

15 DFLAGS=-DEQDEBUG=5 -DCATCH_CRASH -DNO_PIDLOG -DSHAREMEM -DSPELL_EFFECT_SPAM -DFIELD_ITEMS -DCOMBINED -DAPP_OPCODE_SIZE=2 -Di386


If this works, I might put up a VM server to use locally to dig out some OpCodes. It sure beats packet sniffing + creating/fixing a program to decode it all :-) Then, we'll just need to figure out the packet structures if we don't have them already.

[KLS]

I put some code into the server just to see how the code works further. To analyze live packets I believe you would need something to analyze the entire stream; or at least the session request since right now it appears the client receives a key from the server used to encode and decode.

Right now the emu always sends back: 287454020 which is (0x11223344), I actually added some logging to see what it was since the code has no commenting and can be kinda hard to follow at times, so I guess I got that part right the first time.

Basically:

Client creates protocol packet of OP_SessionRequest

pragma pack(1)
struct SessionRequest {
uint32 UnknownA;
uint32 Session;
uint32 MaxLength;
};
pragma pack()

Server replies with a session responce

pragma pack(1)
struct SessionResponse {
uint32 Session;
uint32 Key;
uint8 UnknownA;
uint8 Format;
uint8 UnknownB;
uint32 MaxLength;
uint32 UnknownD;
};
pragma pack()

Client -> OP_SessionRequest -> Server
Client <- OP_SessionResponce <- Server

the uint32 key is what we use to decode the packet if the flag for encoding (0x04) is set in format, and clearly the session would be the session id, format is the bitfield to store the encode and compression flags for the stream.

I'm still trying to understand it myself though, wtb code commenting -.-.

Oh yeah, also if we're trying to find a client -> server opcode that's pretty simple because you can just have it dump to log or terminal over the emu as stated above but a lot of the ops we're missing aren't client -> server and the ones we are is because the functionality hasn't been implemented and if it were finding the opcodes wouldn't be an issue obviously.

[trevius]

Here is what I have found so far just from running the tail I mentioned above. I am sure some things I have tested must be sending opcodes in another format or something, because they don't seem to send the server anything even though I think they would be unknowns. One example would be when you right click an adventure merchant. Unless the opcode is just assigned to the wrong OP_ name in the .conf file for it already. Then, it wouldn't be getting logged by my tail, because it is currently only able to see opcodes that error for one reason or another.

Errors (unhandled):

0x381d - OP_WeaponUnequip2 - unequip weapon 2 or unequip weapon 1
0x63da - OP_WeaponEquip2 - equip weapon 2
0x6c5e - OP_WeaponEquip1 - unequip weapon 1
0x63da - OP_WeaponEquip2 - equip weapon 1
0x6c5e - OP_WeaponEquip1 - equip weapon 1
0x6f0c - OP_Bandolier - Added and named new bandolier name - same when bandolier is used to swap weapons
0x1ee9 - OP_BazaarSearch - /Bazaar Search or clicking "welcome" on the /bazaar window
0x5891 - OP_RaidInvite - Invited a player with /raid - got some packet info as well
0x6f82 - OP_LFPCommand - Shown on the player when they accepted group invite
0x7f9d - OP_Report - /report playername
0x5306 - OP_Feedback - /feedback and filled out a feedback and hit send. Got some packet as well

I know that unhandled doesn't mean there are problems. But, in some cases, it might be useful to have the notes I made just in case someone decides to write some code to handle them.


Unknowns:

0x524e - Begin /trader mode or Closed /trader window even if Begin Trader mode hadn't been started (on off toggle?)
0x6a5f - Comes in after not moving the mouse for exactly 15 minutes. And comes in again as soon as the mouse is moved again - Toggle Auto-AFK?
0x19d8 - /viewpetition command ran
0x5fc7 - Clicked "View Stats" on the Adventure Window - Same for clicking the Refresh button there
0x230a - Clicked "Leaderboard" on the View Stats Window for LDoN Adventures - Same for clicking the Refresh button there
0x48fe - Used the "Who" button in the friends window - Same for typing /who all friends or friend
0x224c - /zone command
0x35e8 - Right click a player in /becomenpc mode in bazaar (basically trader mode on)
0x3d05 - /veteranReward
0x5eba - Used Shift+T to open the Titles window

And some of these probably aren't considered issues, because there are no systems in place for them. But, I am sure someone would love to get bazaar working, and Titles, and the petition manager window would be nice too. Also, I didn't even know EQ had an auto-afk feature, but that one opcode really seems to be auto AFK. It comes in exactly 15 minutes every time after the mouse hasn't moved on that window and comes in again as soon as it is moved again. That would be a cool feature to have IMO.

Notes:

Starting trader mode by opening the /trader window and setting items for sale in a Trader's Satchel didn't make my character show up on the /bazaar search window to be listed under "Traders". BUT, using /becomenpc on myself changes my name to add "Trader" in front of it the way that trader mode is supposed to on Live. Then, once I had /becomenpc on, I now showed up as a trader in the /bazaar search window. It still wouldn't list the items I had for sale from the search results though.

I will continue looking for more opcodes and I am going to mess around with other versions of EQ as well to see how difficult it would be to get them working for the emu. If most of the opcodes to get another version of EQ working can be attained with the tail I have been running, then it really shouldn't be all too tough. I still need to check into those IDA Pro scripts as well to see what they do and if they are still useful in any way.

I guess some progress is better than nothing. Might as well get as many opcodes in place as possible, so if someone wants to write code to get the systems working, they can do so without worrying about opcodes.

[KLS]

the SWGemu people have a pretty decent writeup of the basic UDP transmission protocol sony uses for apparently all their MMOGs. Including a section on encryption, definitely easier to understand the code after reading it. I think it may differ in the footer a bit from the protocol sony uses in EQ, but it's similar enough to give a good idea of what's going on.

http://trac2.assembla.com/swgemu/wiki/Packets

[KLS]

It's really really close actually, I wrote a small collector today that can identify separate streams and reveal their relevant information. Can't decode packets yet but hey that stuff's hard for an afternoon of work. Trying to devise a system for combined packets doesn't sound very fun at least since they can be combined as many times as the client wants apparently.

[trevius]

Unless I am mistaken, it looks like any change to the .conf files requires a server reset to take effect. Or, is there a way to do it via command or something without a complete reload? That would help quicken things up. Especially since I am testing on a live server with lots of players.

For anyone wondering if the Digital Downloads from SoE are a packaged set of files, I wanted to clarify that they are NOT. I went ahead and purchased the 20$ digital download "Starter Pack", but it didn't give me any actual link to download a package. The only way I was able to get the new files was to use the new beta sony patcher, which unfortunately patches to live.

So, it looks like we are currently stuck with Anniversary Edition as being our only option for upgrading the emu unless we want to chase live, which is probably very unlikely. It is unfortunate that the Anniversary Edition is already almost as hard to get as Titanium. Very few stores still have copies on the shelf and many don't even sell it online either. Though, if someone wanted a copy, I don't think it would be too hard to get just yet.

And, thanks KLS, for working on this. If you can come up with something, I am sure it would make a big difference and help the project to keep moving forward. I am still investigating the opcodes as well, but your knowledge and experience far exceeds my own lol.

[KLS]

#opcode should reload all opcodes

[trevius]

Well, I tried #opcode and it doesn't seem to reload any saved changes to my .conf files for opcodes. I still get an Unknown message about opcodes that I have assigned in the config file and saved.

[KLS]

yeah my mistake it's #opcode reload it should give you a client message about reloading all opcode patches.

[trevius]

Thanks, I actually just found that out while experimenting a little after posting that lol. Was just about to come and post the correction.

After figuring that out, I was about to get /zone to work for GMs instead of only #zone working.

Here are the changes to get /zone working. Still working on figuring out more.

#GM/guide opcodes
OP_GMServers=0x3387 #/servers
OP_GMBecomeNPC=0x0000 #/becomenpc 0x7864
OP_GMZoneRequest=0x1306 #/zone
OP_GMSearchCorpse=0x3c32 #/searchcorpse
OP_GMHideMe=0x15b2 #/hideme
OP_GMGoto=0x1cee #/goto
OP_GMDelCorpse=0x0b2f #/delcorpse
OP_GMApproval=0x0c0f #/approval
OP_GMToggle=0x7fea #/toggletell
OP_GMZoneRequest2=0x244c
OP_GMSummon=0x1edc #/summon
OP_GMEmoteZone=0x39f2 #/emotezone
OP_GMEmoteWorld=0x3383 #/emoteworld (not implemented)
OP_GMFind=0x5930 #/find
OP_GMKick=0x692c #/kick
OP_GMNameChange=0x0000


Also, it seems that the #opcode reload only works on a per zone basis. So, you have to reload it in each zone you want to work on opcodes in.

[TheLieka]

That's awesome Trev. I hope that I can get some free time soon - I'd LOVE to come help with OpCodes, it was the last one thing for which I had been trying to make time.

Keep rocking it out, THESE are the things that have been holding back this project from becoming polished. Muchos Kudos!

Dax

[KLS]

Took forever to figure out the correct byte order to get everything to work but finally got the CRC working on live packets and if I can CRC them I can decode them... eventually.

Request session reply(0x0002) found, stream between You and
Server(199.108.9.205:9000)
SessionID = 248095750
Key = 1415749637
MaxSize = 512
CRC Len = 2
FormatFlag = 1
This stream is compressed
This stream is not encoded
00 02 06 A4 C9 0E 05 A0 62 54 02 01 00 00 02 00
00 00 00 00 02 00 6A 25 73 48 7A C6 06 00 3F 00
00 00 3F 00 00 00 14 00 9F 36 00 0C 41 4F E3 9C
00 18 F3 FB 8B A1 08 00 45 00 00 31 7F 19 00

packet passed CRC, SOEProtocol Opcode: 0x0007, key: 1415749637, packetlen: 21
packet passed CRC, SOEProtocol Opcode: 0x0008, key: 1415749637, packetlen: 28
packet passed CRC, SOEProtocol Opcode: 0x0009, key: 1415749637, packetlen: 46
packet passed CRC, SOEProtocol Opcode: 0x0003, key: 1415749637, packetlen: 41
packet passed CRC, SOEProtocol Opcode: 0x000d, key: 1415749637, packetlen: 149


On a live stream so hooray, im going to bed.

[trevius]

Ooh, nice! Sounds like a very nice start. What are you using to get the packets from live? Wireshark?

Then, you are running a script to decode the packets?

Whatever it is, it sounds like a very handy tool when it is complete :D

If I had your knowledge, I could do some real damage :P For now, I will just keep fumbling around and researching. I imagine that getting the friends list and maybe even titles working wouldn't be way too hard. I think I have all of the needed opcodes, so it would be a matter of pulling out the names that are sent for the friends list and doing a who query on them for the client. And for Titles, I imagine we would need a new table that includes the Title name and multiple fields for requirements to meet (like total AAs, etc). I know friends list and titles seem small, but every little thing adds up IMO.

I know that one of the main ones everyone wants to see working are LDoN Adventures, including the points and merchant systems. This would probably be considerably more complex to get working. I think bazaar would be a really cool one as well. And, I don't even think bazaar would be all that hard to get working. Mainly, everything that /becomenpc does is what turning /trader mode on needs to do. Then, just have a table for items for sale, and maybe gather a couple more missing opcodes for purchases and stuff.

Still, getting any of these system working is over my head. Eventually I hope to understand the code well enough to get at least some working. I have already learned quite a bit since I first started looking at the code.

[KLS]

I used pcap; in this winpcap and just made my own program; hardest part was getting emu code to work outside of the emu(could use some portability) and figuring out that ntohl() can change the byte order of const* data if I'm not careful. -.-

The biggest problem probably isn't the opcodes; even on titanium there's only a few opcodes missing here and there, changing structures is what is really hard. And is also why anniv is stalled; we have the opcodes to get in to the game but once you do everything is messed up because the structure of packets has significantly changed.

[trevius]

Ya, I figured that the packet structures were changing and causing issues as well.

So, if we do ever start working on getting anniversary edition or another version of EQ working, we have to update the opcodes in the .conf files, but we also have to update the /common/patches/<client version>_structs.h? Looks like maybe the /common/patches/<client version>.cpp might need to be updated as well.

In general, are the packet structure changes a complete restructure, or just minor changes that still keeps most of the previous packet structure in place? Or would the entire structure need to be redone from scratch for any that change?

Last night I made a discovery that I am sure some of you were already aware of, but I wasn't and I thought it was worth mentioning here; ShowEQ is still being updated and is current with EQLive! I had just assumed that ShowEQ was abandoned in 2003 or 2005 and was replaced by MQ. I know that much of the info in the opcodes .conf files was attained from ShowEQ. So, wouldn't ShowEQ be a useful tool in finding Opcodes and maybe packet structure as well? I would think that it might be a good packet collector for this project, but I could be wrong. Maybe even some of the source code in ShowEQ could be useful in updating the Emulator source to work with later versions of EQ.

I haven't found an archive of ShowEQ versions, so I don't know if there is a version that might work with Titanium or Anniversary Edition. But, I think the current one could be useful in maybe getting fully patched version of EQ working with the emu.

ShowEQ has install instructions for Redhat and Mandrake, but I run Debian, so I am still working to get it to compile properly. I have been able to get the ./configure to run, which is supposed to auto configure so that the make will run without errors, but the make is still erroring on pcab stuff. I will keep messing with it. Hopefully, once I get it running, I can find out more info on if it will be useful for this project or not.

I have already pulled many of the opcodes in the current Live version of ShowEQ and put them in my live.conf file. But, I haven't reset my server yet to see what results from that.

I am trying to look in every direction possible to find a helpful solution for getting the emu updated. Ultimately, I think it would be best if I could understand the output from IDA Pro, but I can't make much of that yet... So, I am trying it from other angles until I learn more.

Here is the link to the showeq website:

http://www.showeq.net/

It includes a link on that page to the current version of the source.

[AndMetal]

Last night I made a discovery that I am sure some of you were already aware of, but I wasn't and I thought it was worth mentioning here; ShowEQ is still being updated and is current with EQLive! I had just assumed that ShowEQ was abandoned in 2003 or 2005 and was replaced by MQ. I know that much of the info in the opcodes .conf files was attained from ShowEQ. So, wouldn't ShowEQ be a useful tool in finding Opcodes and maybe packet structure as well? I would think that it might be a good packet collector for this project, but I could be wrong. Maybe even some of the source code in ShowEQ could be useful in updating the Emulator source to work with later versions of EQ.

I haven't found an archive of ShowEQ versions, so I don't know if there is a version that might work with Titanium or Anniversary Edition. But, I think the current one could be useful in maybe getting fully patched version of EQ working with the emu.

I actually brought this up in another post:

Ah, the billion dollar question... How to find OpCodes :-)

I've thought about this myself, and have run into a few roadblocks. Here are the possibilities I've thought of to get these bad boys from the client:


Packet sniffing
eqgame.exe (http://www.eqemulator.net/wiki/wikka.php?wakka=DevOpcodeFinding)
Grabbing from old versions of SEQ


Option 1 wouldn't be too bad, except you have to decode the packets, which makes just using Ethereal, etc, not very helpful. I think there are some utilities in the server source (using Perl) to decode the packets, but I haven't personally messed with any. Ideally, you could just do something on the client, the look for an OpCode that seems out of place.

Option 2 would probably be the best way to dig out OpCodes, especially ones we have no idea about, but it requires a lot of work and knowledge about Assembly (which I don't). As linked above, there is some info in the Wiki (http://www.eqemulator.net/wiki/wikka.php?wakka=DevOpcodeFinding) that gives some general info about doing this, but it's a little above my head.

Option 3 would probably be the easiest way at this point, since the work has already been done. However, unless something was just overlooked in the original (http://seq.cvs.sourceforge.net/seq/showeq/conf/worldopcodes.xml?revision=1.3&view=markup) SEQ files (http://seq.cvs.sourceforge.net/seq/showeq/conf/zoneopcodes.xml?revision=1.6&view=markup), I'm pretty sure we have all of the Titanium OpCodes we can get from SEQ. It looks like the same goes for Anniversary (http://seq.svn.sourceforge.net/viewvc/seq/showeq/trunk/conf/worldopcodes.xml?revision=675&view=markup) edition (http://seq.svn.sourceforge.net/viewvc/seq/showeq/trunk/conf/zoneopcodes.xml?revision=677&view=markup). On the plus side, it gives us a good place to start for the Live (http://seq.svn.sourceforge.net/viewvc/seq/showeq/trunk/conf/worldopcodes.xml?view=markup) client (http://seq.svn.sourceforge.net/viewvc/seq/showeq/trunk/conf/zoneopcodes.xml?view=markup).

Of course, if it was going to be easy, everything would be working by now ;-)

All of the links at the bottom link to the OpCode files in the ShowEQ CVS/SVN repositories. This is where a lot of the OpCodes came from in the .conf files. Looking at the source (different files of course) from the same time, we should be able to find out at least some of the structures, if we don't know them already. However, taking a quick peek myself, I wasn't able to find any clear cut "structures" file, so it's probably all spread out through the source.

[trevius]

Thanks for the input AndMetal. I saw that post, but I guess I forgot about it lol.

I was looking for packet structure as well, because I assume they have to have at least some of it defined for showeq to be able to read data packets and pull what they need from them. From what I could tell, they do seem to have some kind of packet structure code, but it is done in a completely different format that what the emu uses. Someone with more code knowledge would have to look into that to see if there is anything useful, because I can't tell. Though, they did seem to have some info on systems that the emu still needs to get working, like bazaar, ldon, titles, and others.

I did finally get showeq running on my Debian box, but I need to move both PCs to share a hub so that the packets can be sniffed and I didn't want to do that last night since it would affect my players obviously. I will try to put a hub in the next time I reboot the server. Then, I can get a look at what it shows for Live at least. And see if there is anything useful coming into it.

I would still like to find the source for one of the eq packet collectors that were developed to help find all of this stuff. Maybe the packet decoding, compression and other stuff from the showeq source could be used to update the packet collector.

[KLS]

Talked with fnw a little bit about this in regards to live he listed 3 steps that he would take to find out opcodes and structures for any given client:

1: Steal from showeq
2: eqopfinder used to be good at finding recognizable packets once you had a good foundation and that the info gained from showeq was probably enough.
3: after that just digging through logs with collector and trying to identify any OP_Unknowns.

Of course how well these methods work now is up in the air as well ...it's been a while.

[trevius]

So, other than eqopfinder, I think we might be on the right track so far. If only I could make heads or tails from the SEQ source for packet structure lol.

[steve]

/keys appears to be 0x68c4. I don't see it in the Titanium Opcode file.

Also, /emoteworld and /delcorpse appear to have the correct opcodes, but do not work.

[TheLieka]

/emoteworld and /delcorpse appear to have the correct opcodes, but do not work.

That's just because no one has tied the OpCode into the appropriate function in the source. Since we already have #corpse delete and #emote world <colorid#>, someone just needs to go in and tie the opcode to those functions and be good to go.

The main current issue with the /blahblah client commands (as far as GM level commands) is that they're all flagged at the same level within the source. That is, if you can /zone, then you can also /summon. You would also be able to /delcorpse and /worldemote, etc. At some point, those client commands needs to be added to the commands table (or possibly a separate client_commands table) where permission levels can be assigned more granularly.

Dax

[EvoZak]

This thread delivers.

SEQ should be an excellent source for opcodes, but I wonder about the structures. Would SEQ even be interested in the structures for, say, bazaar or ldon stuff? Isn't SEQ just interested showing you mob pops, loot drops, etc? If that's the case, there's a fair chance they wouldn't implement any of the data structures since they wouldn't have a use for it.

[KLS]

They'd have the hard to find structures like player profile and spawn structures. (They tend to change a lot from patch to patch.) Things like ldon could be figures out via packet collecting and viewing. Though I can't get the packet collector to compile nor the pre-compiled to work. =x Guess I get to keep making mine.

[Derision]

I did finally get showeq running on my Debian box, but I need to move both PCs to share a hub so that the packets can be sniffed and I didn't want to do that last night since it would affect my players obviously. I will try to put a hub in the next time I reboot the server. Then, I can get a look at what it shows for Live at least. And see if there is anything useful coming into it.


I found that for the purposes of packet collecting from live, you don't need to use a hub.

What I did was to run Wireshark on my PC, perform the actions I wanted to collect, then stop the capture and save the packets as a .pcap file.

I then transferred the capture file over to my Linux box and ran ShowEQ in replay mode:

showeq --playback-tcpdump-file=/tmp/livecap.pcap --log-zone --zone-log-file=/tmp/livecap.txt

Once the GUI has loaded and processed the capture, quit and you have a text version of the capture in, e.g. /tmp/livecap.txt


[OPCode: 0x0e98]
[Name: OP_ExpUpdate][Updated: 01/17/08][Type: expUpdateStruct (8) ==]
000 | 42 00 00 00 02 00 00 00 | B.......

Jul 20 2008 19:58:18:849 [Decoded] [Server->Client] [Size: 12]
[OPCode: 0x4656]
[Name: OP_MobUpdate][Updated: 01/17/08][Type: spawnPositionUpdate (12) ==]
000 | 0f 46 e7 1d 61 f4 bf 4f f7 01 0a 00 | .F..a..O....

<snip>

[trevius]

Oh, nice! I will definitely check that out. I already have wireshark and showeq setup, so that wouldn't take much to get going. I guess I better do it soon before the free live thing stops lol...

I still plan to work on Opcodes, but have been pretty busy the past few weeks with RL stuff. Hopefully I can start going on it again really soon :) I know it could make a big difference for the project if we can get more systems working and things like opcodes and packet structures figured out.

[Derision]

Here is a quick and dirty tutorial on how I found the OP_Charm opcode for Titanium, based on information in the Wiki on
disassembly using IDA.

Requirements: The 6.2 Client, The Titanium Client, and an OpCode that is known for the 6.2 client
but unknown in Titanium (this is what we want to find).

Download IDA Freeware 4.9 from: http://www.hex-rays.com/idapro/idadown.htm

Fire up IDA, Select 'New', Select PE Executable, then navigate to your 6.2 eqgame.exe, then let IDA
analyse it. This will take several minutes. It will say 'Idle' in the status bar when it has finished.

In 6.2, Opcodes are handled in the 'Dispatch' routine at address 45A8B6.

To Jump to this address, press G and enter the address 45A8B6.

The routine looks like this ( I added the comments down near the bottom):


.text:0045A8B6
.text:0045A8B6 sub_45A8B6 proc near ; CODE XREF: sub_407D2B+27p
.text:0045A8B6
.text:0045A8B6 var_5980 = dword ptr -5980h
.text:0045A8B6 var_597C = dword ptr -597Ch
.text:0045A8B6 var_5978 = dword ptr -5978h
.text:0045A8B6 var_5974 = dword ptr -5974h
.text:0045A8B6 var_5970 = dword ptr -5970h
.text:0045A8B6 var_596C = dword ptr -596Ch
.text:0045A8B6 var_5968 = dword ptr -5968h
.text:0045A8B6 var_5964 = dword ptr -5964h
.text:0045A8B6
.text:0045A8B6 mov eax, offset loc_5F3AC7
.text:0045A8BB call __EH_prolog
.text:0045A8C0 mov eax, 5954h
.text:0045A8C5 call __alloca_probe
.text:0045A8CA mov eax, dword_75D3D4
.text:0045A8CF mov edx, [ebp+0Ch]
.text:0045A8D2 push ebx
.text:0045A8D3 push esi
.text:0045A8D4 mov esi, [ebp+10h]
.text:0045A8D7 mov [ebp-10h], eax
.text:0045A8DA push edi ; char
.text:0045A8DB mov eax, 4665h
.text:0045A8E0 xor edi, edi
.text:0045A8E2 cmp edx, eax
.text:0045A8E4 mov [ebp-58B8h], ecx
.text:0045A8EA mov ecx, 5ACh
.text:0045A8EF mov ebx, offset byte_95E7B8
.text:0045A8F4 ja loc_45C52C
.text:0045A8FA jz loc_45DA84
.text:0045A900 mov eax, 2089h ; The Opcode of the packet being processed is in register edx
.text:0045A905 cmp edx, eax
.text:0045A907 ja loc_45B969 ; If the Opcode > 0x2089, jump to 45B969
.text:0045A90D jz loc_45B95E
.text:0045A913 mov eax, 1336h
.text:0045A918 cmp edx, eax
.text:0045A91A ja loc_45ADB6 ; If the Opcode > 0x1336, jump to 45ADB6
.text:0045A920 jz loc_45BA1B
.text:0045A926 mov eax, 0A2Ah
.text:0045A92B cmp edx, eax
.text:0045A92D ja loc_45AB94 ; If the Opcode > 0x0A2A, jump to 45AB94
.text:0045A933 jz loc_45AB89




Starting at address 45A900, is where it is checking which routine to jump to, based on the Opcode.

We know that for 6.2, OP_Charm is 0x10A1, so the first test that will pass is, is opcode > 0x0A2A, and
we jump to loc_45AB94.

Move the cursor over the loc_45AB94 in the ja instruction, right click, and select jump to operand, which
takes us to 45AB94:


.text:0045AB94 loc_45AB94: ; CODE XREF: sub_45A8B6+77j
.text:0045AB94 mov eax, 0F40h
.text:0045AB99 cmp edx, eax
.text:0045AB9B ja loc_45ACAC
.text:0045ABA1 jz loc_45DB94


There are some more comparisons. In this case, our opcode 0x10A1 is greater than 0x0F40, so we now jump to
loc_45ACAC. (Jump to operand as before).


.text:0045ACAC loc_45ACAC: ; CODE XREF: sub_45A8B6+2E5j
.text:0045ACAC mov eax, edx
.text:0045ACAE sub eax, 0F66h
.text:0045ACB3 jz loc_45AD6E
.text:0045ACB9 sub eax, 9Eh
.text:0045ACBE jz loc_45AD63
.text:0045ACC4 sub eax, 9Dh
.text:0045ACC9 jz loc_45AD58 ; Jump to Charm Routine
.text:0045ACCF sub eax, 5
.text:0045ACD2 jz short loc_45AD4A
.text:0045ACD4 sub eax, 4Eh
.text:0045ACD7 jz loc_45DE25


What it does now is start subtracting values from the opcode, then jumping to a routine if the result is 0.
0x10A1 - F66 - 9E - 9D = 0, so we jump to 45AD58.


.text:0045AD58 loc_45AD58: ; CODE XREF: sub_45A8B6+413j
.text:0045AD58 push esi
.text:0045AD59 call sub_457270
.text:0045AD5E jmp loc_45B3A0


This just takes us to 457270:


.text:00457270 sub_457270 proc near ; CODE XREF: sub_45A8B6+4A3p
.text:00457270
.text:00457270 arg_0 = dword ptr 10h
.text:00457270
.text:00457270 push ebx
.text:00457271 push esi
.text:00457272 push edi
.text:00457273 mov edi, [esp+arg_0]
.text:00457277 push dword ptr [edi]
.text:00457279 call sub_402DD2
.text:0045727E push dword ptr [edi+4]
.text:00457281 mov esi, eax
.text:00457283 call sub_402DD2
.text:00457288 test esi, esi
.text:0045728A pop ecx
.text:0045728B pop ecx
.text:0045728C mov ebx, eax
.text:0045728E jz short loc_4572EC
.text:00457290 test ebx, ebx
.text:00457292 jz short loc_4572EC
.text:00457294 cmp byte ptr [ebx+1B3h], 0
.text:0045729B jnz short loc_4572EC


This is where it seems to start doing some processing, so we guess this is the guts of the charm routine.

Now we need to find the same routine in Titanium. Close IDA and start it up again, this time selecting the
Titanium eqgame.exe. Let it do it's thing for a few minutes.

When IDA is done processing, our goal is to find a routine in the Titanium client that looks the same, or very
similar to the one above. In this case, what I did was search for the 'immediate' value, 0x1B3.

Press Alt-I, and enter 0x1b3 in the search box. After a few seconds, a list of matches will be displayed. What
we are after is one that is:


cmp byte ptr [ebx+1B3h], 0


to match the instruction in the 6.2 client. In the list of occurences of 0x1B3 in IDA, the second match is the one we are
after. Double click on it and it will take you to the routine. Scroll up a bit and it looks like this:


.text:00453D7B sub_453D7B proc near ; CODE XREF: sub_45B8F0+D1Dp
.text:00453D7B
.text:00453D7B arg_0 = dword ptr 10h
.text:00453D7B
.text:00453D7B push ebx
.text:00453D7C push esi
.text:00453D7D push edi
.text:00453D7E mov edi, [esp+arg_0]
.text:00453D82 push dword ptr [edi]
.text:00453D84 call sub_401B28
.text:00453D89 push dword ptr [edi+4]
.text:00453D8C mov esi, eax
.text:00453D8E call sub_401B28
.text:00453D93 test esi, esi
.text:00453D95 pop ecx
.text:00453D96 pop ecx
.text:00453D97 mov ebx, eax
.text:00453D99 jz short loc_453DF7
.text:00453D9B test ebx, ebx
.text:00453D9D jz short loc_453DF7
.text:00453D9F cmp byte ptr [ebx+1B3h], 0


As you can see, this looks like the right routine, as it is identical in all ways that matter to the 6.2 code.

Now we need to work our way backwards. Click on the 'Sub_453D7B' which precedes 'proc near', and then press
the X key.

This will produce a list of locations that this subroutine is called from. There is only one place, so double
click it, and it takes us to:


.text:0045C60C loc_45C60C: ; CODE XREF: sub_45B8F0+BD0j
.text:0045C60C push ebx
.text:0045C60D call sub_453D7B
.text:0045C612 jmp loc_45F673


Nothing interesting here, so click on loc_45C60C, press X. This will produce a list of where 45C60C is called
from (just the one location). Double click on it in the result dialog box.

This looks more interesting.


.text:0045C4BA loc_45C4BA: ; CODE XREF: sub_45B8F0+B6Fj
.text:0045C4BA sub ecx, 12E5h
.text:0045C4C0 jz loc_45C60C
.text:0045C4C6 sub ecx, 21h
.text:0045C4C9 jz short loc_45C517
.text:0045C4CB sub ecx, 171h
.text:0045C4D1 jnz loc_45F674


Here the code is subtracting 12E5 from ecx (which is the opcode), and following a jump which leads to the
charm routine if the result is zero.

We could continue moving back through the call tree to find if there are any other subtractions being done
on the opcode value, but in this case there isn't, 0x12e5 is the opcode for OP_Charm in Titanium.

[trevius]

Very nice! Much better info than what I was seeing in the Wiki itself lol. So, by using that same logic, I imagine that the known opcodes in Titanium could be used to find opcodes for anniversary or even live. It sounds a bit involved and I don't know much about looking at the IDA output yet, but I think If I can get the hang of it that it might start getting easier.

I guess we would still need to figure out packet structure, and other than packet sniffing, I haven't seen it done really. SOE just needs to give us a peak at their source hehe :P

[MNWatchdog]

As long as Sony doesnt change compilers or their C code for handling Opcodes, you could make what ya might call templates that could easily find any opcodes.

You take the current client, look at the hex before and after the opcode and build these templates to quickly search for new opcodes.

Template for a opcode would be like HH HH HH ?? ?? HH HH, where HHs are the common hex codes and the ?? are the unknown/possible opcodes.

Then you search the new client using these templates.

[AndMetal]

Looking at the source (different files of course) from the same time, we should be able to find out at least some of the structures, if we don't know them already. However, taking a quick peek myself, I wasn't able to find any clear cut "structures" file, so it's probably all spread out through the source.

Found them: showeq/src/everquest.h (http://seq.cvs.sourceforge.net/seq/showeq/src/everquest.h?revision=1.84&view=markup). Maybe some of the later changes to the structs might provide some insight on what the unknown values may be...