Outage knocks out campus' web, Banner, Blackboard services Monday morning

Issue: A little after 11:00 am today (Feb 14) the “Uninterruptable Power Supply” (UPS) in our main computer room at 32nd Street suffered a major failure. The UPS typically does three things for us: 1. “Condition” the incoming power to be without surges and brown-outs which could damage the computers; 2. Provide a failover to batteries to allow a “graceful” shutdown of computers if the power fails; and 3. Provide the same functions should we switch to emergency power from our generator. In this instance the UPS failed catastrophically and all power to our computer servers was shut off. While network connectivity remained up, all access to the machine room failed. This included our main web server, email, Banner, Blackboard, etc.

Resolution: Our Facilities personnel responded immediately to help us diagnose and restore power. Unfortunately during the process of bringing the UPS back into normal mode, the true extent of the damage was revealed. The UPS vendor was called and they are responding to troubleshoot and repair the device. We were able to put the UPS into a bypass position, unfortunately the bypass position does not “condition” the power. Because of the current weather conditions and the very real possibility of power surges coming from the grid, we made a decision to put our machine room power on our back-up generator. The power from the generator is very smooth and gives us a level of confidence while running the UPS in the bypass condition. Power was restored a little after 1 p.m. and most systems were back on-line at 1:20 p.m. By 2 p.m. we had verified that systems were all up and, thankfully, without any failed components due to the power being so rudely interrupted. We will remain on the generator power until we can safely switch back to commercial power.

Future: Early last year a review was completed of our machine room and there were deficiencies noted in our air distribution, our desire for seismic stability of our computer racks, and the fact that we did not have redundant UPS systems. The preferred state is to have two UPS devices that share the load but each can handle the electrical requirements should one of them fail. A capital funding request was put forward to the University and ranked highly in our internal process. That capital budget request is now before the state legislature for possible funding. The need has quite graphically been demonstrated to us.

We will not wait for the legislature to act in reviewing if there are other things we might do to ameliorate this from happening in the future.

Thank you for your patience during the outage. Please let me know if there are any questions I can answer for you.

Regards,
John


John D. Lawson. Ph.D.
Vice Provost for Information Technology/CIO
Western Washington University