• Categories
• Company News
• Industry
• Seattle Internet Events
• Security
• Technology
• Archives
• November 2008
• October 2008
• September 2008
• August 2008
• July 2008
• June 2008
• May 2008
• March 2008
• February 2008
• January 2008
• December 2007
• November 2007
• October 2007
• September 2007
• Subscribe via RSS
• 
• 
• 

Adhost Blog

Plaza East UPS Event Update

At approximately 4:35 AM (PDT) on Saturday Morning, June 21, 2008, an uninterruptible power supply (UPS) that provides service to approximately 20% of the servers in our Plaza East Data center experienced a significant failure. This UPS unit is in a suite which is dedicated to this purpose and is separate from the Adhost Plaza East Data Center. Very early indications suggest that the underlying nature of this failure was a significant drop in amperage from the battery strings which might have caused, for an as yet unknown reason, the unit to go into an uncontrolled over-voltage situation to compensate. Full analysis of this event will likely take at least several weeks. In any case, the end result was significant heat and smoke damage to the UPS unit.

The heroes at the Seattle Fire department were dispatched based on a smoke alarm, and, with assistance of on-site engineers, were able to get the electrical service to this UPS and the one next to it shut down without injury or other damage to the facility. This caused the “A & B” circuits that are fed from these two UPSs, and which serve the above mentioned portion of our Plaza East Data Center, to go dark at approximately 4:52 AM PDT. No other UPS systems on the floor or anywhere else in Plaza East were affected by this shutdown (including the UPS systems located within the Adhost Plaza East Data Center and other UPS systems owned and operated by Fisher Plaza on the floor).

Adhost and the Fisher Plaza engineering team were able to restore bypass power to the affected circuits in the Adhost Plaza East Data Center (and to other portions of Fisher Plaza East) by approximately 6:15 AM PDT. Adhost personnel then began the process of powering systems back up and working with affected customers to get their systems back to full operation.

Why did both UPSs get shut down? These two units are in close proximity in a suite dedicated to this function. The second, originally unaffected UPS drew in a significant amount of smoke and other possible foreign material through its intake/cooling fans during the event. It was also probably subject to significant heat which may have compromised the components in the device. At this point, the engineers and the manufacturer are unwilling to certify its operation.

In our initial release to customers, we termed this as a fire, as that is what we initially had to work with. Although there was significant heat and smoke, there were no actual flames and, aside from the two UPSs, there appears to be no long term damage that is not easily redressed. The building has had a restoration company in the building around the clock to address the issue of the smoke and odor which resulted from this event. We are also going through not only our data center in this building, but also our offices which are located on this floor and we find little or no remaining evidence of damage, with the obvious exception of the power status.

So, where are we now? The loads that are affected are currently only backed up by generator power. This means that, in the event of interruption of service from Seattle City Light, the load will drop for 30-60 seconds until power is restored by the generators. Our engineers have been in close contact with Seattle City Light and they are well aware of our situation and have agreed to do whatever they can to minimize or eliminate any work on their power distribution plant that might have an adverse impact on our service. Our thanks go out to them for their understanding and cooperation.

Now, there is some good news. Normally, replacing UPS units like these is a question of several weeks or months. However, two new UPS units were recently installed on the floor which are available for us to use. We are working with the building and our vendors to map out a plan for getting the load moved onto these UPSs as quickly as possible. We do not have an exact timeframe for this because there are several components that are currently hard to predict (exact availability of materials, permitting and inspections, etc.) but we are pushing this process as fast as it is safe (and legal) to do so. We’re hoping to set a target date of this weekend. We will make a formal announcement to the affected customers when we have that exact timing. We will, of course, work with our customers to minimize any impact that this transfer may have on their service.

We apologize for any impact this event may have caused to our customers or to their customers. We thank our customers for their understanding as we work through this situation. We also would like to thank the multitude of players for true excellence during this trying time including, the Fisher Communications engineering team, the Egis engineering team, the Adhost engineering and systems administration teams, Prime Electric, the Seattle Fire Department and Seattle City Light. With their collective effort and knowledge, this potentially very bad situation was made considerably better.

We also are very thankful that no one was injured during this event.