No Detail Is Too Insignficant

Short blog today. While sitting a series of large-scale cutovers and work arounds for a client, we had a generator fail while one of 14 systems was in maintenance bypass. The facility is preapring for a retrofit, and we were installing some much needed repairs for the site to hold us through the next year’s worth of upgrades.

The generator failed for the following reason:

1. The generator is installed outside in a cold weather environment. Resultantly, the louvers are automated and are closed when the generator is off.
2. The louvers were not on UPS power – it was on house power.
3. When house power was cut over (without a fast work around), the louvers failed shut (the spec’d requirement).
4. The gen tech was off screwing around, and was not at this post to note that the generator louvers shut while operating. This was noted as an emergency action drill prior to the cuotver.
5. The generator ran until it shut down on coolant overheat.
6. The generator switched open transition to the backup, dropping the UPS load that was on maintenance bypass.
7. The gen tech did not bring coolant with him to the cutover, requring a call to the shop and 45m delay. The work was shifted to the alternate, but less reliable machine installed on site. It also required a 3 hour delay to the 36 hour scehdule for the primary gen restart and retransfer.

Ok, what went wrong (and I know, it’s a lot):

1. Supporting circuity was not traced and incorporated into the workplan.
2. No one knew the failure modes of the generator.
3. The gen tech was not prepared with his primary and secondary parts kit, as briefed and contracted.

We’re 8 hours in, and there will be more to follow.

Tags: data center, data centers, modular power, power distribution, PUE, UPS power