Sourcing Cave Dude's Database into PEQ

[BobThompson]

I for one would love to have them if you can find a host to serve them up. I'm assuming it would be too long to just insert as a code block in a post?

[zephyr325]

I think the problem may be bigger than we think...I did a pretty deep dive into the records.

There are 27,813 unique NPC ID's between the Cavedude and PEQ databases.
- Cave dude has 20,195 npc ID's; PEQ has 23,122 NPC ID's.

Of those:
- 8,482 of those NPC ID's and NPC names match up <- This is good
- 12,305 of those are unique (i.e., its in Cavedude but not in PEQ, or vice versa)
- 7,024 NPC ID's match up, but they have different NPC names. <- This is not good

In other words, there are over 7000 NPC's records that would fail if we tried to bring them over because their NPC_ID number is already being used by the other database.

Ok, so maybe we look at doing things by the NPC names, rather than their id:

- There are a total of 16,095 unique NPC names between the two databases.
- Cavedude has 12,043 unique npc names. PEQ has 13,349 unique names.
- There are 9,297 names that match each other between the two databases
- There are 6,798 names that are in one of the databases, but not the other.

So there are almost 6800 differences in the NPC names.

I'm really thinking that the NPC ID's is the way we should approach it. Otherwise, all the "a_decaying_skeleton" records would just show up as one NPC.

I'm ready to go after this to synchronize them, but my head is swimming trying to figure out the best way to do it.

[Zengez]

Heres what i did...

I pulled up a list of NPC names in cavedude that wasn't in peq... pulled those out into a list

Then i ran a program to take every line of the insert statement from dumping cavedude's database where the name from the above list showed up, giving me several insert statements per name at times, all of which were not in the peq database...

Then i adjested the npcid's to make them larger than peq would ever use, so that there was no chance of a duplicate id entry forming when i sourced it over, nor would there be any chance of a future peq update overwriting the id that i assigned to these mobs... so it is (theoretically) 'update proof'

Thats how i got the npc_types table to work in terms of the npc's ids versus name...

I'm sure i missed some, it's not perfect, it's not pretty, but i gotta say i'm wandering around plane of valor right now on my peq database and looking at all the spawns that weren't there a minute ago

I looked around and found a free hosting site for a temporary host until someone else can host it for themselves, here's the link;


http://upload.ohshare.com/v/8413635/...o_PEQ.zip.html

Note, you do NOT need any of cavedude's database on your machine to use this, they are straight insert statements designed to be run by mysql to insert into peq database... there are linebreaks every 80 or so characters in order to allow them to be straight copy/pasted into the mysql command line if you wish to do so (note some of these files are pretty long, i've found trying to paste in more than about 25,000 lines of code at a time can cause the command prompt to crash, so paste in sections)

Enjoy everyone, hope it helps some of you out, if nothing else at least it's a starting point

[eq4me]

Thanks alot!

This adds 6648 spawns in 46 zones.

awk -F"'" '{print $2}' spawngroupnew.sql | awk -F"_|[0-9]" '{print $1}' | sort | uniq -c| sort -n
4 shadowrest
15 hohonorb
46 hohonora
49 mmcj
50 takd
51 pofire
53 tipt
54 poair
54 ruji
55 potorment
55 solrotower
60 qinimi
62 bothunder
64 natimbi
64 potranquility
74 povalor
82 potactics
89 veksar
104 codecay
112 postorms
117 harbingers
119 nedaria
125 hatesfury
128 powater
131 chardokb
132 riwwi
150 bloodfields
154 vxed
155 hateplaneb
156 qvic
187 barindu
189 poinnovation
200 potimea
209 podisease
218 draniksscar
229 pojustice
229 wallofslaughter
232 abysmal
233 ferubi
250 poeartha
255 poknowledge
258 provinggrounds
287 causeway
310 dranik
376 kodtaz
422 ponightmare


Just one question.

Why do formerly not populated zones do have spawngroup entrys like nedaria_63 and additions to PEQ zones entrys like podisease1640021? Is the _ missing or redundant?

[Zengez]

Alot of the spawnentries and grid info i did as educated guesswork... like i said it's not perfect, just a point to jump from... if anyone feels so ambitious i'd recommend going through them more meticulously to find which ones specifically are needed and which ones aren't for these tables and clean up those two files a bit more than what i did...

Not quite sure if that answers your question or not, if not, pm me with more specifics

[Angelox]

Originally Posted by Zengez

Heres what i did...

I pulled up a list of NPC names in cavedude that wasn't in peq... pulled those out into a list

Then i ran a program to take every line of the insert statement from dumping cavedude's database where the name from the above list showed up, giving me several insert statements per name at times, all of which were not in the peq database...

Then i adjested the npcid's to make them larger than peq would ever use, so that there was no chance of a duplicate id entry forming when i sourced it over, nor would there be any chance of a future peq update overwriting the id that i assigned to these mobs... so it is (theoretically) 'update proof'

Thats how i got the npc_types table to work in terms of the npc's ids versus name...

I'm sure i missed some, it's not perfect, it's not pretty, but i gotta say i'm wandering around plane of valor right now on my peq database and looking at all the spawns that weren't there a minute ago

I looked around and found a free hosting site for a temporary host until someone else can host it for themselves, here's the link;


http://upload.ohshare.com/v/8413635/...o_PEQ.zip.html

Note, you do NOT need any of cavedude's database on your machine to use this, they are straight insert statements designed to be run by mysql to insert into peq database... there are linebreaks every 80 or so characters in order to allow them to be straight copy/pasted into the mysql command line if you wish to do so (note some of these files are pretty long, i've found trying to paste in more than about 25,000 lines of code at a time can cause the command prompt to crash, so paste in sections)

Enjoy everyone, hope it helps some of you out, if nothing else at least it's a starting point

Hi, and thank you for your great work - this was very needed. If you need a space I can fix you up on my webserver, or I can post your stuff un my UGLY web page (ugly, but works). PM me with what you want.
I have a question; Are you using eight digit numbers? I'm asking, becuause I am using the eight digit numbers for my up dates to the zones, and don't want them to cross with yours.

[Zengez]

a bunch of them are 8 digit numbers, so you'd have to check those... most of them are 9 digit numbers however i believe, you can always use a 'where id > 100000000' clause to source in most of them then peel through the rest...

all mine end in 000 though since it was a straight *1000 so that should make looking through the id's easier for you.

[Angelox]

Originally Posted by Zengez

a bunch of them are 8 digit numbers, so you'd have to check those... most of them are 9 digit numbers however i believe, you can always use a 'where id > 100000000' clause to source in most of them then peel through the rest...

all mine end in 000 though since it was a straight *1000 so that should make looking through the id's easier for you.

You might want to check out nine digit numbers - I had problems to where my EQEMU wouldn't load the database when they were nine digit.

[eq4me]

How about generating the IDs with an fixed offset instead of an multiplier?
EDIT: On second though I dont think its that easy, you might have to discard the old IDs alltogether and give new IDs, counting upwards from the offset.

Be aware that I did not read through the whole documentation yet, so the below could be moot:
Maybe it would be nice to have some conventions what ID ranges to use for certain stuff. Eg. Half the usable range for PEQ itself. A well sized chunk for imports from Cavedudes and other databases(are there any?). And the rest for local server customizations.
I recon it would help server admins to make sure that their custom additions dont clash with further updates of the PEQ databases.

[Zengez]

Originally Posted by Angelox

You might want to check out nine digit numbers - I had problems to where my EQEMU wouldn't load the database when they were nine digit.

Yea, i had that problem originally because the database only supports some of the upper end ranges because of the size limitation of the field, but it won't tell you you've reached the max until you've already hit it, then try to go higher a second time...

basically if you multiply by 1000, then do so again, half yer id's turn out to be the same number which won't let anything load, but it won't error until you try to multiply/divide it 'again' once they are all the same number... which is frustrating...

Anyway i ran into this problem by accident, figured out what the problem was, then resolved it... so (in my case anyway) everything loads just fine and i didn't break anything... I can't imagine anyone else having this problem if i didn't because i haven't actually customized my server very much... but I make no promisses as i don't know other people's code... my recommendation; dump your database files before you source these in, then if it kills something do a kill on the table and resource in your dumped original version...


As to how to source it, i just did a straight copy/paste from the text files into the mysql prompt in order to put it in, but remember not to do any more than 20-25k lines of code at a time or you may crash the prompt (won't destroy your database or anything, just crashed the prompt)