Skip to main content

Lessons Learned: Scada Failover

August 26, 2026

NERC’s recently published lesson learned SCADA Failover details how a single unexamined assumption nearly took a utility offline during scheduled maintenance.

WHAT OCCURRED

During scheduled network switch maintenance at a utility’s telecom infrastructure, a miscommunication between the telecom network team and the applications team resulted in unplanned failovers and intermittent SCADA control loss. The telecom team assumed the applications team’s task had been completed and that servers would be relocated to a standby site; however, the applications task was delayed, leaving the wrong location as active. This caused unexpected failovers lasting 10–15 minutes at a time over approximately one hour, resulting in loss of BPS visibility at a secondary control center, disconnected ICCP links, and affected EMS services. The event lasted 65 minutes total and impacted operations across the utility’s entire footprint.

DURATION & IMPACT

  • 65 minutes total
  • Loss of visibility to secondary control center
  • ICCP links disconnected
  • EMS services affected

ROOT CAUSE

  • Lack of inter-departmental coordination on scheduled maintenance
  • No verification of actual equipment status before work began
  • Missing real-time communication protocol during critical work

KEY TAKEAWAYS & LESSONS LEARNED

    1. Verification Trumps Assumption. Always confirm actual equipment status before critical work begins, regardless of planned schedules.
    2. Coordination Is Infrastructure. When multiple teams touch interconnected systems, communication channels and protocols are as critical as technical redundancy.
    3. Bridge Calls Enable Real-Time Coordination. Scheduled communication among all stakeholders during maintenance reduces conflicts and enables immediate escalation.
    4. Schedule Review Must Span Multiple Time Horizons. Weekly and longer-term schedule reviews catch conflicts that daily planning might miss
    5. Delayed Tasks Must Be Communicated Immediately. Any delay in dependent work must be escalated and communicated before downstream work begins.
    6. Escalation Protocols Must Be Clear and Immediate. When assumptions fail or schedules slip, immediate notification is required, not silent continuation

    For more information on these lessons learned, visit NERC’s Lessons Learned portal or contact GDS Associates for a detailed review of your compliance and oversight practices.