CenturyLink 911 outage was caused by a single network card sending bad packets
How does a single network card emitting bad packets effect other sites?<p>&gt; investigations into the logs, including packet captures, was occurring in tandem, which ultimately identified a suspected card issue in Denver, CO. Field Operations were dispatched to remove the card. Once removed, it did not appear there had been significant improvement; however, the logs were further scrutinized .. to identify that the source packet did originate from this card.<p>&gt; Support shifted focus to the application of strategic polling filters along with the continued efforts to remove the secondary communication channels between select nodes.<p>And then<p>&gt; By 2:30 GMT on December 29, it was confirmed that the impacted IP, Voice, and Ethernet Access services were once again operational. Point-to-point Transport Waves as well as Ethernet Private Lines were still experiencing issues as multiple Optical Carrier Groups (OCG) were still out of service.<p>And finally<p>&gt; The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.<p>Looks like the root cause analysis has a way to go. Addendum says:<p>&gt; The CenturyLink network continued to rebroadcast the invalid packets through the redundant (secondary) communication routes.. These invalid frame packets did not have a source, destination, or expiration and were cleared out of the network via the application of the polling filters and removal of the secondary communication paths between specific nodes. <i></i><i></i>The management card has been sent to the equipment vendor where extensive forensic analysis will occur regarding the underlying cause, how the packets were introduced in this particular manner.<i></i><i></i> The card has not been replaced and will not be until the vendor review is supplied. There is no increased network risk with leaving it unseated. <i></i><i>At this time, there is no indication that there was maintenance work on the card, software, or adjacent equipment.</i><i></i> The CenturyLink network is not at risk of reoccurrence due to the placement of the poling filters and the removal of the secondary communication routes between select nodes.