Live Data Collection (Need Community Help)

[Akkadius]

I'm going to make this short and sweet.

We have made huge ground parsing Magelo's site for large amounts of data to catch up PEQ all the way through the current expansion. I spent time to write a web crawler that has gathered all of the following data types from all expansions to now:

Merchantlists
Loot
Tradeskills
Spawnpoints


I will be working on getting these in one by one, my life has been super busy and it has been hard to make time for things but I'm still trying to get these things knocked out because they are rather large. I also am not rushing because Cavedude is in the middle of moving, I am probably going to end up moving here shortly myself.

Asking the Community for Help:
Anyways, I need help from the community to collect logs from live. These are the logs that are generated from /log in-game from any and all time frames that have ever been played.

What will this enable us to do?
We will be able to collect quite a wide variety of data that just simply can't be automated through web sources. This will allow us to collect:

NPC Relative Health
NPC Spell Casts
NPC Faction hits
NPC Damage Data
NPC Aggro Emotes
NPC Death Emotes
Other things I'm not thinking of right now


If we can get log data from PEQ's more recent expansions (GoD, LDON) all the way through the current expansion the more we will be able to fill in the gaps of all of the data that gets manually processed through the development procedures and standards that PEQ has put in place today.

My effort is going to be to take logs from players anonymously through a form and have a script go through and tear through the data existing through those logs. These logs will be kept privately and will only be used for the benefit of this community and will only be seen by trusted developers.

The UPLOAD form can be found here: http://eoc.akkadius.com/eoc/EQEmuDataCollect/LogCollector.php

Simply upload a eqlog file of your EverQuest live characters and once the data parsing process begins, I will try to make a way to have people see what NPC's have been collected in what zones and what hasn't collected so we can take an accurate hitlist against what we need to collect as a community.

This will take a while as we get people to dump their historic logs from the past, I encourage everyone to reach out to anyone they might know to help this process move along so we can get more data.

How/What:


Typing /log in game in eqemu will create a logfile under EverQuest/Logs/eqlog_charname_servername.txt
We will need preferably HUGE logs that contain data spanning several zones, old and big logs are completely fine
The more combat spam the better, I don't care about /gu or /t messages, I won't be caring about any of that



If anyone has any questions about this please feel free to post.

[namini]

Oh boy. Logs, if I can find the old hard drives that I used to play and guide on, shoot. We're talking a crazy amount. But, yeah.

Short and sweet answer, you need logs, from what expansion to which expansion?

[Noport]

what format would you like to receive them in ? what is the max size allowed ? are you talking about the log files an or are you asking for pcap files ? i have several live logs i started in 2003-present

[sorvani]

we are talking about standard log files created by the /log on command.

any and all logs are helpful, but mostly from GoD+ as everything else is already "completed" more or less. Akka will have to respond to file size restrictions if any.

[Akkadius]

The default format in which they are generated in is required:

eqlog_charname_server.txt

I don't have any size restrictions, just load em up!

And yes all expansions GoD to now log data should be uploaded. Can use any and all, this needs to be a big community effort.

[jdoran]

Would you like us to trim the logs to just those expansions, or can your parser sort it out. I have logs going back years.

[demonstar55]

Would you like us to trim the logs to just those expansions, or can your parser sort it out. I have logs going back years.

Well, they did mention wanting to parse NPC Aggro/Death emotes and I know we are missing a lot of them from pretty much every expansion ... But stuff that isn't on PEQ yet would probably still be more useful.

[Akkadius]

Would you like us to trim the logs to just those expansions, or can your parser sort it out. I have logs going back years.

Anything and everything, my parser will be able to differentiate the data.

[boweq]

where would i send them to akka?
, also anything on live that you need really bad?

still got active acounts ( 95sk/95ranger/95sham/95bard/90 rougue)

[demonstar55]

where would i send them to akka?
, also anything on live that you need really bad?

still got active acounts ( 95sk/95ranger/95sham/95bard/90 rougue)

Read first post, there is a URL there and I think logs from pretty much everything is fine

[boweq]

yeahh im asking if theres something paticuller from a expansion that needs logging down cause i can go do that lol.

[lerxst2112]

yeahh im asking if theres something paticuller from a expansion that needs logging down cause i can go do that lol.

It says this in the original post. Pretty self explanatory to me.

If we can get log data from PEQ's more recent expansions (GoD, LDON) all the way through the current expansion the more we will be able to fill in the gaps of all of the data that gets manually processed through the development procedures and standards that PEQ has put in place today.

[Burningsoul]

Bowmaster's a pretty simple being... He's like an orc - point him in the direction you want him to smash, and let him have at it. With your team are you capable of HoT & VoA content Bow? That'd be the place to hit if you're capable. Hell I'd LOVE to see crystallos fully working, just to finally play through it rather than read about it.

[blackdragonsdg]

Hey Akkadius your website rejected the log I attempted to upload tonight. It said it was not a log file but it is a legit log file. I tried to upload the file a second time thinking maybe it was just a fluke but it rejected it again. Any ideas?

[Noport]

we are talking about standard log files created by the /log on command.

ok i'll send you a log of our guild chat i did type /log on in guild chat

[Akkadius]

Hey Akkadius your website rejected the log I attempted to upload tonight. It said it was not a log file but it is a legit log file. I tried to upload the file a second time thinking maybe it was just a fluke but it rejected it again. Any ideas?

It's going to look for the eqlog in the name as well as .txt

[blackdragonsdg]

It's going to look for the eqlog in the name as well as .txt

Well it wasn't the name that caused the problem. Just a guess but I think 100% packet loss around hop 22 may have had something to do with my upload not working.

[Akkadius]

Well it wasn't the name that caused the problem. Just a guess but I think 100% packet loss around hop 22 may have had something to do with my upload not working.

Try it again, I changed some max upload size settings on the webserver end.

How large was the file that you were trying to upload?

[blackdragonsdg]

Try it again, I changed some max upload size settings on the webserver end.

How large was the file that you were trying to upload?

It uploaded successfully today and the file size 2.08MB.

[Akkadius]

Thanks for those who have uploaded logs, but so far this is looking pretty pathetic.

We really need to get a real density of logs before any parsing systems can really be developed around this.

If you know someone with a mother load of logs spanning lots of content please contact them and see if they are willing to contribute to this.

[pfyon]

Haha I was just thinking about this a couple of weeks ago. Glad someone got around to writing a parser before me :)

I found 80 from a variety of characters + another 26 that I renamed as part of archiving back from 2002.

[Akkadius]

Haha I was just thinking about this a couple of weeks ago. Glad someone got around to writing a parser before me :)

I found 80 from a variety of characters + another 26 that I renamed as part of archiving back from 2002.

Oh wow, logs from 03/04. Now we're talking progress.

Once again if I can encourage anyone and everyone to try and get them to upload historic character log files this helps the project quite a bit.

If anyone has questions please let me know.

Thanks,
Akkadius

[demonstar55]

Have you thought about posting this on like the EQ sub-reddit? I'm sure there would be people at least interested in submitting some and there is a larger chance they have logs.

[pfyon]

Any chance you can release some aggregated data once you're done with this project? I'd like to see how older mobs compare (hp/damage wise) to what we have in the peq DB.

[Akkadius]

Any chance you can release some aggregated data once you're done with this project? I'd like to see how older mobs compare (hp/damage wise) to what we have in the peq DB.

Yeah I definitely will, I just need to collect a decent amount of logs before I start putting together the parser.

In the middle of working on some other rather large PEQ parsing projects to get loot, merchantlist, tradeskills from all expansions.

In the meantime I just need to start building the private repository so I have something to work with when the script is being created to parse through all of the text files.

After that I plan on having a web page that people can go to and see a hitlist of what has been collected and what hasn't

[Maze_EQ]

Question....Is test server data useful? I've got access to a multitude of accounts that can basically box everything up to VoA stuff.

[pfyon]

Finished my first 80 using the web interface, found another 46 (in addition to the archived ones).

After sitting in front of my computer for an hour doing those, I decided I'd script it. If you have a linux box and lots of logs, try the following script:

for x in eqlog*.txt; do echo "Starting $x"; curl -F FileName=@$x -F EQEmuForumName=CHANGEME http://eoc.akkadius.com/eoc/EQEmuDataCollect/LogCollector.php &> /dev/null; done


Change or remove the '-F EQEmuForumName=CHANGEME' to your forum name if you want it to be included.

[Akkadius]

That's quite the collection you have there and that definitely will help.

I've updated the form to include how many logs we've collected so far and the size:

http://eoc.akkadius.com/eoc/EQEmuDataCollect/LogCollector.php

[pfyon]

How are you storing the logs? Are you using the original filename? I have multiple logs which share the same filename (I would copy them elsewhere and/or rename them to make parsing of more recent stuff easier) which, if you keep them as the same filename, could get overwritten.

[Akkadius]

How are you storing the logs? Are you using the original filename? I have multiple logs which share the same filename (I would copy them elsewhere and/or rename them to make parsing of more recent stuff easier) which, if you keep them as the same filename, could get overwritten.

No, I thought that through beforehand. They never get saved as the original filename.

[Akkadius]

Update:

We have tons of 2002-2005 logs now, we need a high density of 2004-2012 logs now.

Need your help!

UPLOAD HERE:

http://eoc.akkadius.com/eoc/EQEmuDataCollect/LogCollector.php

[Warking]

I have all my EQ files since I started in 99. I'll start looking through them

[Akkadius]

I've not seen anything uploaded recently, the community needs more help with this than we are getting. People have no idea how much this is going to help the community.

[Caryatis]

this is going to help the community.

You need to re-frame this to what the individual is going to get out of it. Give out some worthless points or some shit and work out a deal with some of the big servers to redeem them for trinkets, then you will be overwhelmed with logs.

[rhyotte]

well... although i quit playing live many years ago, I suppose I could login and ask some of the guilders who still play if they could share some of their logs. In the next few days I will attempt to get ahold of them.

[blackdragonsdg]

I am out of town atm and can not collect logs for the time being. I should be back home in a week or so and will resume log collections.

[Akkadius]

You need to re-frame this to what the individual is going to get out of it. Give out some worthless points or some shit and work out a deal with some of the big servers to redeem them for trinkets, then you will be overwhelmed with logs.

Everyone will get punch and pie, and leprechaun points :D

[RStefan01]

I have 14 gigs of log files from 2004-2009 when I played whatever ridiculous amount of time I played EQ Live (something like 700+ days played 2001-2009). I started running logs some time during PoP, and logged pretty much all of my group/raid gameplay through the last boss of SoD. If something was in the game during that time, I probably killed it... a lot.

But, here's the thing. If I could parse out chat before sending it, and then send them zipped somehow, I'd prefer doing it that way. If your script strips out tells/guild/channels/raid/etc., I'd be willing to set up a server and run it locally, then zip everything up and ftp it somewhere.

Don't take it personally, but there's a lot of chat in there that doesn't have any business being out in the world. I was an officer/leader of a raiding guild for years, so there's a lot of private shit in there that I don't really feel comfortable sharing since the people involved really don't have a say. Then there's just the sheer file size of it all... zipping it down to 1/10 the size would help tons on my end.

Not sure if it's worth all the hassle, let me know ;p

[Caryatis]

there's a lot of chat in there that doesn't have any business being out in the world

Always surprises me how important MMO players think their drama is. Unless you guys were talking about murder, rape or molesting children... nobody gives a shit if you thought some app was gay.

[chrsschb]

Pretty sure the script rips out chat data and only parses the combat data.

[Akkadius]

Pretty sure the script rips out chat data and only parses the combat data.

You are correct, no one will see the chat data.

[pfyon]

As much as I agree that there is little that could be of interest in someones' log (unless you included account or credit card info), but perhaps opening up the source would alleviate some peoples' concerns.

[RStefan01]

I'll take Akka's word that it won't store any chat info... still gonna take forever to upload that much though.

[Akkadius]

I'll take Akka's word that it won't store any chat info... still gonna take forever to upload that much though.

Compress it in chunks if you need to. We can arrange was of getting it to my repo so the parser can process it

[RStefan01]

I've already been storing em in zip files by year. Basically an entire year as a zip file is the size of a monthly log. I've got some web space I could throw it on, or I could ftp it somewhere if you want.

[Akkadius]

I've already been storing em in zip files by year. Basically an entire year as a zip file is the size of a monthly log. I've got some web space I could throw it on, or I could ftp it somewhere if you want.

Sent you private temp FTP information so that you can upload your compressed logs when you can.

Thanks for your help

[elfindal]

Uploaded two, one ~75mb, another ~100mb, both from 2012 only. Will start logging more whilst I play, pending I remember.

[blackdragonsdg]

Any ideas as to what has been logged so far? Given previous posts 2005 and back has already been covered...is that correct? I don't spend much time on live so I would prefer to gather something useful instead of taking shots in the dark on what to log next.
I have already hit parts of Omens and TSS and I am working through SoF atm. Two boxing will only get me so far and I found that stopping point tonight in MMM.

[Akkadius]

Any ideas as to what has been logged so far? Given previous posts 2005 and back has already been covered...is that correct? I don't spend much time on live so I would prefer to gather something useful instead of taking shots in the dark on what to log next.
I have already hit parts of Omens and TSS and I am working through SoF atm. Two boxing will only get me so far and I found that stopping point tonight in MMM.

I have to build the tracking system for it so it can update automatically. I will let you know when I can get to this, in the meantime just log whatever you come across.

[Darknolan11]

Was just inquiring if you still need logs. If you do is there anyway to figure out what has been collected and what needs to be collected?

[Akkadius]

Was just inquiring if you still need logs. If you do is there anyway to figure out what has been collected and what needs to be collected?

I have to find time to write a parser, once I start cracking at one it shouldn't take too terribly long.

[azog]

first file is some of moors, the other is rest of moors, then the hive, then all of messa. i tried to kill at least one of everything.

hope it helps

Is there a way to get faction imported into peq? I thought someone had a script that would take about 5 min to get all the factions for the character in the current zone there in. but I can't find the post now.

Ill make sure I have logs on when I go on live from now on.

[sorvani]

It is posted over on the PEQ forums as it involves mq2
http://www.peqtgc.com/phpBB3/viewtopic.php?f=14&t=13716