WOOT Homepage of projecteq.net is showing now!

There are signs of life! We have hope once again!

Actually it looks like everything but the forums and the game server are up and I'm sure that Cavedude is either taking a well deserved rest right now, or tinkering away now that things are starting to be up and running. Either way, catering to us whiners (me included mind you) is probably low on his priority list. When he is done, we will know. 8)

Bad news
 

Alright, so here's where we are at, and it isn't good.

When the machine came back up today, we noticed our /home directory had reverted to a state it was when this machine was first built. /home is where our website is, and where we run the game server from, along with all our of personal work directories. PEQ is setup using a mirrored array, for those familiar with drive redundancy/backup solutions. At first glance, it appeared a drive in the array failed, and the mirror failed to rebuild the data properly. Further investigation shows we have a controller that isn't responding to the kernel at all. So, without physically inspecting the machine we can conclude that either:

1. We have a bad/loose drive cable.
2. We have a drive that is so dead it is preventing the controller from working properly.
3. We have a failed controller and as such the motherboard will need to be replaced.

I'm not sure which is worse, as on one hand we need to replace a drive, and have a good chance of losing very important data. On the other, we need to replace the motherboard. Though, in that case our data may be safe, unless the controller took out the drive. Either way, replacing hardware has to wait for now.

Fathernitwit is the only team member that has physical access to the box. Unfortunately, he is going away for a month very soon. He is going to try to get to the datacenter this weekend and see about assessing the situation in person. We then have a few options for getting the game server back up in the meantime:

1. A minor problem occurred (cable came loose) FNW fixes it, we are back up to 100% (Wishful thinking, probably will not happen)
2. We have a dead drive, and FNW is able to get data off. (Not likely considering FNW's very limited time)
3. FNW disconnects the drives on the problematic controller and I manually rebuild the game server and get us back up. (We can't risk rebuilding the game server now, having the "bad" controller come back up for some reason, and destroy the old good data when the mirror updates)
4. FNW isn't able to get to the DC, or it turns out to be an issue we didn't expect. In that case, we were offered to use a server for the time being. If the offer still stands, we may have to go that route.

My goal is to get the game server up this weekend, when FNW gets to the DC. Once he returns from his trip, he and I will discuss permanent solutions, assuming it is a hardware issue.

Now, I'm sure you're all wondering what data might we lose? Well, fear not the drive our databases are stored on is healthy, and current. So, your characters, forum data (posts, etc), quest status data, player points data, etc are all safe and sound. Both FNW and I have backed all of that data up to our personal machines as well. What we might lose, is the website code. The forum code, all of our tools, editors, scripts, etc. It can be reconstructed, but it will be painful :( We also might have lost the game server directory, but that is simply a matter of uploading my server directory from my test machine, and changing a few settings. Misc internal scripts, PEQBot, and other such things that made my life easier may also be on the chopping block.

It could be much, much worse of course. But, it still sucks. I'll know more this weekend, so hold tight.

Thats rough news for any big server =\. Hope you guys are able to sort it out with minimal troubles and data loss.

something probably came loose when they droped it durring the move...

i know
 

just re route the incriptions :D

COME ON LOOSE CABLE !!!!! Everyone cross your fingers


.................................................. ...........................

Are you meaning that you had mounted a partition to /home and now that partition no longer mounts so you have an empty /home directory? That would make sense if that partition was on part of the array that failed.
It should be fairly easy to see if a drive failed right? Just check /proc/mdstat and /var/log/messages. But what do you mean it failed to rebuild the data? It sounds like you didn't replace the broken drive yet, so how would you "rebuild the data?" Did removing the broken drive from the array not work?
Is that not working? Maybe thats what you mean by
If it does come to that, why don't you post a list of what you need and I'd be willing to try and get a couple of those items taken care of.

Thanks for all your work!

What about projecteq.net/quests ? that was probably my favorite feature of PEQ. Is that one of the website parts that are going to take a lot of work to rebuild?

Let us know
 

Cavedude, keep us informed as to the servers status. If you do indeed need to purchase new hardware let us know. I don't have tons of money, which is why I play free EQ, but I would not hesitate to paypal you guys a donation to help cover the cost of getting the server back on its feet. If the website comes down again post a link to your paypal here.

BoginDA

No worries. If it's fixable, then great. We will rebuild.

If not, yes, the offer still stands cave, if that is what you are referring to.

Spoon: My terminology was a bit confusing, I'm sorry. "rebuilding" is generally reserved for RAIDs with parity. I think what it came down to is without pointing any fingers, the mirror at some point failed us. When the machine was turned on, the one controller wasn't working, so Linux used the mirror drive for /home which isn't current. I was only given partial information, we were more concerned with damage control than why did this happen last night. Either way, once it comes time to decide permanent solutions, I'm going to push for a hardware RAID5 (I would love RAID6, but that's a dream!)

Fingel: The quest status data (which quest are completed, the authors, etc) is safe. The PHP page, however may be lost and would need to be rewritten.

BoginDA: We're not going to worry about donations or hardware until we have more information and when FNW comes back from his trip. Thank you for the nice offer!

renoofturks1: Yeah, we may need to take you up on your offer as a failsafe plan. Thank you very much! I'll contact you and Cantus through PM.

I would like to make it clear that I am going to do everything I can to ensure the server is up by Monday, so sit tight guys, we're almost there!

We're back up, and didn't lose a thing! Enjoy! :)

The website is currently down, but that looks to be a network error.

Sweet! Just in time for the weekend. So much cleaning house, getting my car fixed, eating, etc. :D

Oh man! Such great news!

What wound up being the problem?

Thanks for all the hard work, cavedude. I'll be on as soon as I finish some paperwork.

Before everybody starts saying we're down again, we brought the server down to do a full backup, It'll be back up after we're done :)

HAHA =P I was waiting for the sky to fall from the air and destroy the planet!

I thought it was our 2 earthquakes! :)

They over now? You had me worried there man.

Yeah. 5.6 and a 6.1. The building was swaying and shaking. Pretty cool!

Not cool man, your on the 31st floor!

Just out of curiosity, how long does a full server backup take?

NOOOOOOOOOOOOOOOOOO............................... . we are down again its noon on saturday................................


NOOOOOOOOOOOOO :(



the tease continues

More updates, or was there an unforeseen problem?

wish i knew lol, sad thing is i was at work when server came up >.< i didnt even get to go oh yay my charies -.- sigh

Now there's a server popped up called "PEQ says change me" any idea what that's about?

That server is a projecteq server pack server, which is set up with MySQL defaults, PEQ DB defaults, etc.

Chances are you shouldn't log into that server till the admin announces what he's doing with it, and chances are the admin should have locked it.

The question remains. What happened? I'm getting scared! :-?

Now I'm officially scared.

We had to do a full backup and ensure the raid was working properly. We did, and it is so cross your fingers, but this ordeal *should* be over. Website is back up, game server is next after I get some changes merged.

well its nice to see PEQ back up hope it stays up now XD hehe ^.^