Fire in Servers for VFK!

Update 1:

From Doug Erwin (The Planet):
"This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost."

"We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department."

"Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock."

"I am sorry that this accident has occurred and I apologize for the impact."
----------------
Update 2:

Hello,

Our various departments continue to work very hard to restore service to our H1 data center. We expect to have ServerCommand access restored within the next several hours as well as access to single homed nameservers out of H1 redirected to another facility.

We will continue to provide status updates once an hour. We do not have an Estimated Time to Repair at present.

Todd Mitchell
-----------
Update 3:

Morning,

We are continuing to work through various issues this morning. We will have additional contractors on-site this morning starting at approx. 7 AM. Some will hand-off from contractors who worked overnight and others will start the recovery/installation of new electrical gear to power the data center.

We are still working through the EV1 DNS and ServerCommand items. We are making progress on both items and expect to have both functional within the next 120 minutes.

In addition to the above, the network engineering group worked overnight to prepare the network for the recovery of H1. We expect the reconvergence of the network to go smooth once H1 comes back online.

We do not have an Estimated Time to Repair at present; we should have a better estimate this morning. Our staff and management continue to work through the night and morning-- we will continue to provide hourly updates.

Todd
-----------
A major update for all of our Beta testers:




As you may have heard in the news, there was a large explosion caused by electrical equipment failure in the H1 data center in Houston that knocked down three walls at about 5:00 PM CDT. Fortunately, and most importantly, no one was injured. In addition, no data was lost and there appears to be no damage to the servers or networking equipment.

You remember when you first started playing that the servers went down? Well that was because of bandwidth, so we went to The Planet, which is recognized as one of the most RELIABLE providers in the world.

As luck would have it we are among an estimated 100,000 websites that are down around the world.

We have been assured that engineers are working around the clock to get the H1 data center back online, and we are told it will resume operation sometime tomorrow, Sunday, June 1st.

We are not going to waste this time though. This gives us the opportunity to complete enhancements we are working on, and add new lands. Our goal is that each day you log into the kingdom, it will be better than the last.

Keep an eye out and make sure you log on when the services are restored, because our world will be better than ever.

We have rerouted the DNS for the WWW server, so that we can share with you what is happening.

However, we do need the bandwidth from the H1 data center in Houston to support all of us in our virtual world.

We will keep you updated about the progress being made in restoring the Houston data center.

Your development team
Virtual Family Kingdom

-------------------

Outage update from the developers

The outage caused by the electrical explosion continues and the best estimates given to us are that the services should be coming back online in the next few hours.

When this outage originally occurred we were told it would be resolved in less than three hours. It was several hours later that the extent of the damage was determined and a new estimate of when the services would be restored was provided.

We had an option to have our services redeployed but this was forgone to allow those services which support critical sites, such as sites related to medical and health support to be moved instead. We know that each of our community would have made the same decision.

We have had a number of questions about how our network is structured and so we have included the following outline of the topology of our network:

(check the site for picture)

Even though there has been an outage, we have been hard at work taking advantage of this time to complete a number of features we know you will love.

We look forward to seeing you again as soon as the outage is resolved, hopefully later this evening.

Your Development Team
Virtual Family Kingdom

-------------------

Update from the developers
12:01 AM pacific time, Monday, June 2nd

Information we have indicates that the section of the Houston data center on which we rely for services will be functional in the next few hours.

Here is the information about the status of the data center:

"Following the restoration of power to the second floor of the data center, we've cooled the data center floor and are now in the process of systematically restoring power to racks."

"We've got a full staff in the data center to power up racks in sections and verify that the server hardware starts up successfully. This process may take a few hours to restore service to all customer servers on the second floor."

Based on this information we plan to be back in service at our regular time, 7:00 AM pacific time, a few short hours from now!

Your development team
Virtual Family Kingdom

-----------