Sourcing Cave Dude's Database into PEQ
 

I would imagine this has been thought of already, but is there any way to source certain tables, like the npc_types table into PEQ's without messing everything up or no? Seems the tables have incredibly similiar structure, so with a little tweaking to make them the same, a mass "source everything that is in cavedude's that isn't in PEQ's" type function would insert all the missing npc info to at least have a basis to start from, but i notice the npcid's themselves are not necessarily the same for the same mobs in cave dudes vs. PEQ's ... is there any way around that or is that pretty much an insurmountable barrier?

you would need to go through the DB NPC by NPC.

Was giving this some thought, and here is a revised method...

would it work to maybe eliminate duplications where the mob 'names' are the same, thus eliminating the duplicate mobs from both tables (so i don't have 2 versions of the same gnoll from blackburrow)

After i've eliminated the duplicate entries to ensure that i'm only getting the newer npc's rather than duplications of the old, i could do a run through the tables, picking out when cavedude's npc and PEQ's npcs have the same npcid, then change cavedude's npcid to a unique one...

Finally, source info from the revised cavedude's npc_type table into the PEQ one giving me at least all the information that was lacking in terms of having at least the mobs availible to import

Furthermore i guess in theory you could change the npcid's for all the relevant spawn tables (npc_types, spawn2, spawngroup, spawnentry... am i missing some?) so that they would transfer over with their new unique id's and be able to spawn in the world, still without faction or loot but thats the next task...

So, as a newb programer anyone that could point out a hole to this theory so i could attempt to work on it s'more, it would be appreciated, right now i'm trying to figure out why i can't seem to successfully backup my database :)

They can be merged together. So basically you would probably want the zones that are populated in cavedudes DB after luclin (PEQ has luclin fully spawned already). You will have to go through and Dump and sort out lootdrop ,lootdrop_entries ,loottable,loottable_entries,npc_types,spawn2,spaw nentry,spawngroup,grid, and grid_entries table. You will have to remove all spawns and entries from luclin and prior from all those tables from cavedudes DB. Once you remove all those Entries you can just source in the left over info you have from the dumps you made of cavedudes DB into PEQ DB and you now have a merged PEQ/cavedudes DB. This is the way i did it and it is very tedious and may not be very future proof to when PEQ releases there next DB.

After searching the planet (and this site and affiliates) for that elusive "combined database", I've decided one of the best ways I can learn the Emulator DB structure was to write my own scripts for merging the two. So far, my eyes have started bleeding only a dozen times, because the auto_increments used to _fk some of the tables together are all over the place (I think that was mentioned elsewhere).

If I get my scripts done, I'll link them here. I work in ASP/VBScript and have no intention of writing other versions. Another thing I could do is post the sources my script generates once I am done - but I really think this has been done 100,000 times before "me", and I am just amazed it is unavailable anywhere.

I'll let you know how it turns out. Cuz right now, I have no idea if I am screwing up spawn and loot drops to all PEQ zones just by doing simple insert/replace. :) Probably. We shall see. Meanwhile, I am learning tons, and it's very exciting!

I am easily amused.


PS: I remember reading somewhere about a standard ID scheme of zone * 1000 for entries. This is what I was hoping to end up with by the time I am done renumbering everything. Ambitious, no?

Now, just out of curiosity as i have experienced in a brief run-through it looked like cavedude's database is lacking any faction or loot for anything beyond luclin anyway... and if mobs are factionized/itemized it's likely not enough mobs for me to particularly care about... so this means that 'in theory' all i'd need in order to have the mobs be created and spawn in front of me, would be to source over the npc_types, spawn2, spawnentry, spawngroup, grid, and grid_entries tables right? i don't really care about doors and whatnot either in terms of the 'general zone specifics' mentioned in another similiar post...

As a side note thanks to the great tool by Zephyr325 I've managed to already get the mobs that need to be sourced in listed (need to change all their id's which will be a pain with 2500+ mobs but oh well) and everything i need for spawn2, and spawngroup, are all ready to be sourced once their id's are re-written so things look plausible even though the id switching will likely make me go blind and take forever :)

Won't let me edit previous post so sorry for the double post...


I've managed to finish doing a patch to;

the id and loottable id (just in case) fields of npc_types
the npcid (to match npc_types id field) of spawnentry
the id and spawngroupid fields of spawn2

i tried to change the spawngroup id field and kept getting duplicate field error no matter what i tried to change it to, so i'm not too sure how to go about avoiding that problem, but thats another day

My real question is, i looked into changing id's in grid and grid_entries but i can't find any description of what their fields mean, the wikki is un-written and i didn't see any posts describing their function, so is there any id fields that i would need to change to make them unique against the peq fields?

btw i edited all the above by just multiplying their respective id fields by 1000, putting them theoretically well beyond the id reach of peq's future patches... so it should work fairly well unless i'm missing something that will crash it, which i probably did, and it probably will, but wheres the fun if it didn't right? :)

and i'm done... i got the spawn points in, and i even did the faction tables to see if it helped (not sure they sourced correctly, i got no errors but not sure the correct faction is to the correct mob) but anyway, if anyone wants the set of insert statements to at least populate all teh zones feel free to let me know and i'll try to get someone to host the files for me...

I for one would love to have them if you can find a host to serve them up. I'm assuming it would be too long to just insert as a code block in a post?

I think the problem may be bigger than we think...I did a pretty deep dive into the records.

There are 27,813 unique NPC ID's between the Cavedude and PEQ databases.
- Cave dude has 20,195 npc ID's; PEQ has 23,122 NPC ID's.

Of those:
- 8,482 of those NPC ID's and NPC names match up <- This is good
- 12,305 of those are unique (i.e., its in Cavedude but not in PEQ, or vice versa)
- 7,024 NPC ID's match up, but they have different NPC names. <- This is not good

In other words, there are over 7000 NPC's records that would fail if we tried to bring them over because their NPC_ID number is already being used by the other database.

Ok, so maybe we look at doing things by the NPC names, rather than their id:

- There are a total of 16,095 unique NPC names between the two databases.
- Cavedude has 12,043 unique npc names. PEQ has 13,349 unique names.
- There are 9,297 names that match each other between the two databases
- There are 6,798 names that are in one of the databases, but not the other.

So there are almost 6800 differences in the NPC names.

I'm really thinking that the NPC ID's is the way we should approach it. Otherwise, all the "a_decaying_skeleton" records would just show up as one NPC.

I'm ready to go after this to synchronize them, but my head is swimming trying to figure out the best way to do it.

Heres what i did...

I pulled up a list of NPC names in cavedude that wasn't in peq... pulled those out into a list

Then i ran a program to take every line of the insert statement from dumping cavedude's database where the name from the above list showed up, giving me several insert statements per name at times, all of which were not in the peq database...

Then i adjested the npcid's to make them larger than peq would ever use, so that there was no chance of a duplicate id entry forming when i sourced it over, nor would there be any chance of a future peq update overwriting the id that i assigned to these mobs... so it is (theoretically) 'update proof'

Thats how i got the npc_types table to work in terms of the npc's ids versus name...

I'm sure i missed some, it's not perfect, it's not pretty, but i gotta say i'm wandering around plane of valor right now on my peq database and looking at all the spawns that weren't there a minute ago ;)

I looked around and found a free hosting site for a temporary host until someone else can host it for themselves, here's the link;


http://upload.ohshare.com/v/8413635/...o_PEQ.zip.html

Note, you do NOT need any of cavedude's database on your machine to use this, they are straight insert statements designed to be run by mysql to insert into peq database... there are linebreaks every 80 or so characters in order to allow them to be straight copy/pasted into the mysql command line if you wish to do so (note some of these files are pretty long, i've found trying to paste in more than about 25,000 lines of code at a time can cause the command prompt to crash, so paste in sections)

Enjoy everyone, hope it helps some of you out, if nothing else at least it's a starting point :)

Thanks alot!

This adds 6648 spawns in 46 zones. :)

awk -F"'" '{print $2}' spawngroupnew.sql | awk -F"_|[0-9]" '{print $1}' | sort | uniq -c| sort -n
4 shadowrest
15 hohonorb
46 hohonora
49 mmcj
50 takd
51 pofire
53 tipt
54 poair
54 ruji
55 potorment
55 solrotower
60 qinimi
62 bothunder
64 natimbi
64 potranquility
74 povalor
82 potactics
89 veksar
104 codecay
112 postorms
117 harbingers
119 nedaria
125 hatesfury
128 powater
131 chardokb
132 riwwi
150 bloodfields
154 vxed
155 hateplaneb
156 qvic
187 barindu
189 poinnovation
200 potimea
209 podisease
218 draniksscar
229 pojustice
229 wallofslaughter
232 abysmal
233 ferubi
250 poeartha
255 poknowledge
258 provinggrounds
287 causeway
310 dranik
376 kodtaz
422 ponightmare


Just one question.

Why do formerly not populated zones do have spawngroup entrys like nedaria_63 and additions to PEQ zones entrys like podisease1640021? Is the _ missing or redundant?

Alot of the spawnentries and grid info i did as educated guesswork... like i said it's not perfect, just a point to jump from... if anyone feels so ambitious i'd recommend going through them more meticulously to find which ones specifically are needed and which ones aren't for these tables and clean up those two files a bit more than what i did...

Not quite sure if that answers your question or not, if not, pm me with more specifics

Hi, and thank you for your great work - this was very needed. If you need a space I can fix you up on my webserver, or I can post your stuff un my UGLY web page (ugly, but works). PM me with what you want.
I have a question; Are you using eight digit numbers? I'm asking, becuause I am using the eight digit numbers for my up dates to the zones, and don't want them to cross with yours.

a bunch of them are 8 digit numbers, so you'd have to check those... most of them are 9 digit numbers however i believe, you can always use a 'where id > 100000000' clause to source in most of them then peel through the rest...

all mine end in 000 though since it was a straight *1000 so that should make looking through the id's easier for you.