Adhost Blog
24/7/365 Support
1-888-323-4678
support@adhost.com
Company News
Adhost East Data Center Update from Adhost CEO
Posted in Company NewsDear Adhost Clients:
I am writing to you to update you on the status of the power infrastructure supporting the Adhost East Data Center located in Fisher Plaza East. Adhost is committed to providing a fault-tolerant, highly available environment to support our data center operations. We are working closely with Fisher Plaza, other tenants and vendor engineers and electricians to get our data center operations back onto that footing as fast as possible. This letter is meant to provide a synopsis of both the actions taken to date as well as a synopsis of short and long-term steps that Adhost and Fisher Plaza are taking to achieve our commitment.
As many of you know, on July 2nd, 2009 there was a significant event in the main electrical switchgear room located in the parking garage. While we are still awaiting final forensic analysis of this event, at this point we know that two 5000 amp bus ducts were severely damaged and that enough heat was generated to activate the pre-action sprinkler system in this room. Of course, the introduction of water into a live electrical switchgear room had predictable results – namely that all of the gear in that room became inoperable. Unfortunately this room was where the automatic election is made between utility and generator power. The loss of this room resulted in a loss of power to the Adhost East Data Center for approximately 27 hours.
The Adhost West Data Center was unaffected during this event and remains in its standard operating configuration of utility power backed by N+1 generator plant.
For approximately one month following the event, our East Data Center was powered by generators as its primary source. We announced several maintenance windows to upgrade the reliability of that generator power including moving from the single-riser/single generator configuration that we initially brought the building back up on to an N+1 generator plant that allowed for continuous operation and accommodated a regular maintenance schedule for the individual generators.
On August 1st and 2nd, we announced and successfully completed another maintenance window to switch-over to a temporary configuration that features utility power as the primary electrical source backed by the N+1 generator plant for emergency backup.
Meanwhile, we are working to develop a long-term permanent solution that will take this event into consideration and which will allow Adhost to operate continuously through a similar loss. This solution entails the following components:
- Diversifying the Adhost East critical equipment loads across the two switchgear rooms that service (or will service) Fisher Plaza East.
- Working with Fisher Plaza to ensure that catastrophic loss of one room will not inhibit our ability to operate.
- Working with the various regulatory agencies to put into place non-water-based fire suppression systems in all critical electrical areas.
These solutions take time to engineer and they also take time to source and install the appropriate equipment. Right now we are working to have the permanent solution in place by the end of October. As we get progress down this path, we will have better timelines which we will make available to our customers.
One other lesson that we have learned and which we are addressing is to implement a better emergency communications plan (ECP). Our previous ECP was shown as inadequate during this event and we already have implemented parts of this plan. Highlights of the plan include:
- An off-site (completely outside of the Adhost network and facility) maintenance mailing list server which is automatically synchronized with our main maintenance mailing list. We will also be introducing three primary lists over the coming months:
- East Data Center Operations
- West Data Center Operations
- Shared Hosting Operations
Once in place, we will announce to all lists events that impact the entire infrastructure such as power, cooling, network, DNS, and email. We will also announce major events that primarily impact one of these subgroups to all three lists with follow-up details being posted to the affected lists.
- Adhost is committed to using Twitter as a quick, off-site communications tool to announce not only information on major events, but also other Adhost news. You can follow our tweets at http://www.twitter.com/adhost
- An off-site Web server which will act in several capacities:
- Allow customers to get status messages on our major services.
- Provide a chat-style interface for customers with status questions and support needs.
- An off-site phone number that will give status updates to callers.
- Moving our phone systems to a more fault-tolerant system that will allow us to communicate via phone with customers during significant events.
Once again, we apologize for the extent of this event. We want to assure you that we are integrating the lessons that we learned from this event into our on-going operations and policies and procedures.
We appreciate your business, patience and understanding as we work to improve our services. If you have any questions or suggestions, please email support@adhost.com.
Will Riffle
President and CEO
Adhost Internet
permalink | digg this | technorati
Adhost CEO Participates in DatacenterDynamics Seattle 2009
Posted in Company News, IndustryAdhost CEO Will Riffle will be participating in a panel discussion at the DatacenterDynamics Seattle 2009 Conference on Thursday, August 6. The topic for the panel discussion is “Case Study: Data Center Disaster Response and Recovery – An Engineering and Operational Review of the July 2009 Fisher Plaza Fire.” The moderator for the panel will be Ed Doyne, Director of Technology Services – Fisher Plaza – Egis Real Estate.
posted by Amy R 12:55 pmpermalink | digg this | technorati
Fisher Plaza East Power Maintenance – August 01, 2009
Posted in Company NewsDate: 08/01/2009
Start Time: 10:00 PM PDT
End Time: 6:00 AM PDT on 08/02/2009
Customer Impact: None Anticipated
Description:
This maintenance window is Phase 2 of 3 in the process of restoring normal power to Fisher Plaza East, delivered from Seattle City Light and backed up by generators within the facility. During this phase, Fisher engineers will install a temporary connection to Seattle City Light. This means Adhost’s East data center will be receiving power from the City with redundant backup generators outside of the building until Phase 3 is completed and permanent power is restored to the facility.
During this event, there will be periods during which Adhost’s East data center cooling infrastructure is running in a degraded mode. Specifically, Adhost has 4 separate Computer Room Air Conditioning (CRAC) units in our East facility. Two have power fed from one riser and two are fed from a different riser. During the maintenance window, power will be interrupted several times to each of the risers that feed the CRAC units for approximately 30 minutes each.
The Method of Procedure for this event requires that a cool down period be observed between each of the riser outages. So, Adhost’s East facility will be pre-cooled to 65 degrees before the event and then re-cooled to 65 degrees between each riser outage. This ensures that the data center temperature will remain well within tolerances throughout the maintenance window.
We do not anticipate any loss of service during the window. However, if you would like to be here during the event, please arrive at Fisher Plaza no later than 9:00 PM on 08/01. During the maintenance window access to the building will be restricted, so anyone arriving after 9:00 PM may not be granted access into the facility until the end of the maintenance window.
Adhost’s Plaza West facility will not be affected by this event, as it has been unaffected throughout the recent power issues localized to the Fisher Plaza East building.
Thank you for your patience and understanding during this event. If you have any questions or concerns please contact our Technical Support Department at (206) 404-9500 or support@adhost.com.
Regards,
Michael Smith
–
Michael K. Smith – CISSP, GISP
Chief Technical Officer – Adhost Internet LLC mksmith@adhost.com
w: +1 (206) 404-9500 f: +1 (206) 404-9050
PGP: B49A DDF5 8611 27F3 08B9 84BB E61E 38C0 (Key ID: 0×9A96777D)
permalink | digg this | technorati
Adhost responds to Techflash Blog
Posted in Company NewsHello Todd:
I wanted to give you an official response to your blog post. I’m only in a position to speak about the maintenance announcement you quoted as it pertains to Adhost customers. Every Fisher Plaza East tenant will have a bit of a different take on the event depending upon the configuration of the equipment in their own space. One thing that I can say is that the fundamental cooling systems for Plaza East will not be affected during the window.
Adhost will have two brief periods where we have reduced cooling capacity in our Plaza East data center. We have 4 separate Computer Room Air Conditioning (CRAC) units in our facility. Two have power fed from one riser and two are fed from a different riser. During the maintenance window, power will be interrupted to each of the risers that feed the CRAC units for approximately 30 minutes each.
Before each riser outage, the data center will be cooled to 65 degrees. In the past, we have seen the temperature in the data center raise less than 5 degrees during this type of events. As such, we are confident the temperature in the data center will remain well within tolerances during the event.
The rundown of the event is:
1) Cool data center to 65 degrees.
2) Perform maintenance on Lights and Power Riser – 2 CRAC units off line.
3) Re-cool data center to 65 degrees.
4) Perform maintenance on Mechanical Riser – 2 CRAC units off line.
5) Re-cool data center to 65 degrees.
6) Perform maintenance on Technical Riser – no Adhost equipment affected.
It should be noted that Plaza East and Plaza West are completely discrete buildings with completely discrete cooling and electrical plants serving them. Because of this, our Plaza West data center was not affected by any of the events of the last month, and will also not be affected by this coming maintenance window.
Thank you for the opportunity to respond. If you have any questions please don’t hesitate to let me know.
Kind Regards,
Michael K. Smith
–
Michael K. Smith – CISSP, GISP
Chief Technical Officer – Adhost Internet LLC mksmith@adhost.com
w: +1 (206) 404-9500 f: +1 (206) 404-9050
PGP: B49A DDF5 8611 27F3 08B9 84BB E61E 38C0 (Key ID: 0×9A96777D)
permalink | digg this | technorati
Adhost Seattle Data Centers Pass SAS 70 Type II Audit
Posted in Company News, Security
We are pleased to announce Adhost successfully passed a SAS 70 Type II audit for the time period July 16, 2008 – July 15, 2009. During a week-long on-site visit in June, an independent service auditor examined the policies, procedures and business processes in place at Adhost over the one-year time period. The following controls related to the colocation services provided in our Seattle data centers were audited:
- Control Environment: discipline and structure are a key component of the organization and influence the control consciousness of its personnel.
- Physical Security: business premises and information systems are protected from unauthorized access, damage and interference.
- Environmental security: critical information technology infrastructure is protected from certain environmental threats.
- Availability: colocation facility is available and accessible to its customers
- Operations: Operations department identifies and resolves problems affecting the colocation facility or customers in a timely manner.
The examination was performed in accordance with standards established by the American Institute of Certified Public Accountants. Existing Adhost clients, as well as potential clients, can request a copy of the SAS70 audit report by emailing sales@adhost.com.
To learn more about SAS 70 and the details of the audit, visit our SAS 70 page.
posted by Michael K. Smith 12:12 pm July 23, 2009permalink | digg this | technorati
Fisher Plaza East Power Maintenance
Posted in Company NewsDate: 07/20/2009
Start Time: 12:01 AM PDT
End Time: 4:45 AM PDT
Customer Impact: None Anticipated
Description:
Fisher Engineers will be performing maintenance on the power systems supporting the Adhost East Data Center. This maintenance is addressing an issue we are having with the redundant generator set powering the Fisher East building. During this window, the three central electrical risers will, one at a time, be powered off and re-engergized a short time later.
We do not anticipate that there will be any interruption to the electrical service to the Adhost East Data Center during this window. Adhost and its vendors have reviewed the potential impact to our operations of this procedure. The electrical systems that service the Adhost East Data Center technical loads will be able to transfer from one riser to the other in an uninterrupted fashion. Because our mechanical systems (AC) are tied into the risers differently, we will have reduced cooling capacity for two intervals during the window, up to 30 minutes each. We anticipate that there may be brief periods where temperatures in the data center may increase, but we do not expect to reach temperature levels that will impact server performance. We will be fully staffed during this window to monitor all environmental systems and respond to any issues that may arise.
If you would like to be on site during the maintenance window, please arrive at the Adhost facility no later than 11:30 PM on Sunday, July 19th. During the maintenance window access to the building will be restricted, so anyone arriving after 11:30 may not be granted access into the facility until the end of the maintenance window.
Adhost has been working with Fisher Engineering to develop a plan for restoring redundant, city-fed power to Plaza East. If you would like to see an outline of the plan, please visit http://www.adhost.com/blog.
Thank you for your patience and understanding during this event. If you have any questions or concerns please contact our Technical Support Department at (206) 404-9500 or support@adhost.com.
Regards,
Michael Smith
–
Michael K. Smith – CISSP, GISP
Chief Technical Officer – Adhost Internet LLC mksmith@adhost.com
w: +1 (206) 404-9500 f: +1 (206) 404-9050
PGP: B49A DDF5 8611 27F3 08B9 84BB E61E 38C0 (Key ID: 0×9A96777D)
permalink | digg this | technorati
Adhost Supports KUOW for 5 Years
Posted in Company News
Adhost is proud to celebrate our 5th anniversary as a KUOW sponsor. In case you’re not familiar with KUOW 94.9FM, the radio station serves the Puget Sound region and Western Washington as a provider of National Public Radio (NPR) content and locally-produced public radio programming of the highest quality.
Many Adhost staff and clients are regular KUOW listeners. We believe in KUOW’s mission: to create and serve an informed public, one challenged and invigorated by an understanding and appreciation of events, ideas and cultures.
Listen to our recent spot and underwriter testimonial.
posted by Amy R 2:48 pm July 16, 2009permalink | digg this | technorati
Redundant Generator-based Power Completed
Posted in Company NewsAdhost has successfully completed its announced maintenance window to migrate the electrical infrastructure in the Adhost East Data Center to an N+1 generator-based power system. During the window, all systems operated normally and environmental conditions remained within acceptable limits.
This system should allow us to run indefinitely on generator-based power. This is an important first step in our current three-phase plan to return to our normal configuration of utility power backed by generators. As we develop firmer timelines on the next phase(s), we will announce them to our customers.
If you have any questions or comments please contact our Technical Support Department at (206) 404-9500 or support@adhost.com.
posted by Michael K. Smith 2:45 am July 13, 2009permalink | digg this | technorati
Fisher Plaza East Restoration Timeline
Posted in Company NewsAfter numerous meetings with Fisher Plaza this week, we have a tentative plan to share with our Adhost East Data Center clients. Note that this plan is still very fluid and we are meeting regularly with Fisher, as well as all of the various engineers, electricians, utilities, etc. to improve our electrical sources in the near and long terms. To this end, we are following a number of parallel tracks:
This weekend: Redundant generator-based power
- In order to provide N+1 redundancy, Fisher will add two additional generators and paralleling equipment to the existing temporary generators. This will provide redundancy across the three central risers affected by the power outage. Until the building returns to commercial/street power, this solution will allow the building to perform maintenance on the generators as necessary without impacting service. As of July 9, 2009, we expect this work to be completed by approximately 5AM on Monday, July 13, 2009.
- The City of Seattle has closed John Street next to Fisher Plaza for the additional generators.
- All necessary generators are on-site.
Short-term: Temporary commercial (Seattle City Light) power and generator backup
-
We are working with Seattle City Light to create a temporary connection to their vault that will allow us to switch our main service back over to city power with generator backup.
-
The time frame for this is hoped to be less than 4 weeks and is dependent on the ability to source the proper switch-gear and other related electrical equipment.
-
This temporary solution would follow a parallel track of building a new switch gear room and rebuilding the original switch gear room.
Long-term: Permanent solution
- Fisher will build a new switch gear room outside the current vault and switch gear room on P2. This will allow Fisher to begin construction quickly without waiting to clear the damage equipment from the current switch gear room.
- Fisher will also rebuild the current switch gear room which will allow for redundancy.
- Fisher plans to add new equipment and paralleling equipment to power up the building’s four generators together.
Fisher emphasized that their vendors are locating equipment and designing the new plan as quickly as possible. However, they are expecting several months until full restoration. Once they have a firmer timeline we will share it with you.
posted by Michael K. Smith 5:52 pm July 9, 2009permalink | digg this | technorati
Adhost Revives Twitter Account
Posted in Company NewsLast summer we experimented with a Twitter account for a few weeks but then forgot about it. We learned the hard way last week how important the social media outlets are for getting messages out in a crisis situation like the power outage. In hindsight we should have been posting updates from the beginning of the outage on Twitter and other sites.
So we’re blowing the dust off our Twitter account and will be more proactive about tweeting. We invite you to follow us at: twitter.com/Adhost.
If you’re on the fence about Twitter like we were, we encourage you to read Ian Lurie’s recent blog post: AUUUUGH: Social Media For Crisis Communications. He has some great tips which we are taking to heart.
posted by Amy R 4:26 pm July 8, 2009permalink | digg this | technorati
« Previous Page — Next Page »