Please see our postmortem from our network below in regards to the issues experienced on Thursday 11th July . Please note the below report is from our network directly.
Intermittent Loss of connectivity through the Trafford Data Centre which impacted multiple voice services including access to the Inbound and Horizon Portal. The majority of services were restored by 11:40am but we continued see some residual service impact into the afternoon.
Following a partial failure of a line card within one of the IP core routers hosted at the Trafford data centre.
Due to the partial failure this caused a network IP traffic loop condition on the core router causing a very high processing load across the device, Traffic routing through this device would have experienced high latency which would have impacted voice and portal services
The traffic looping itself was identified and resolved at 11:40 following some extensive analysis by both our IP and engineering teams. They cleared down the processing load and the majority of services stabilised with the exception of links directly associated with the line card at fault.
The remaining network links associated with the faulty line card were then rerouted over alternate line cards within the core router, this work largely completed by 14:30. At 19:00 an emergency change control was done to replace the impacted line card and by 21:40 all hardware was replaced and all subsequent monitoring and testing was successful.
Pending a full root cause from our vendors on the actual line card failure, we will be reviewing any design improvements we could implement under these exceptional circumstances and how we can effectively plan and test for this from a business continuity perspective.