It cannot be understated how impactful the Canada-wide outage on the Rogers network was to individuals and businesses. Not only were millions of consumers without their cell phone, home internet, home security, home phone and TV services for the majority of July 8th (and beyond for some) but business outages extended beyond business cellular and network services to payment and emergency services. Regardless of the size of your business, if you relied on Rogers for some facet of service delivery then you and your customers were impacted.
So far, all Rogers has communicated to the public is that the outage was caused by an update to their core network. Without any inside information on the cause of the outage, there are serious questions that Rogers must answer regarding their network resiliency and governance practices as this is the second major outage in less than 18 months for the company.
So far, all Rogers has communicated to the public is that the outage was caused by an update to their core network. Without any inside information on the cause of the outage, there are serious questions that Rogers must answer regarding their network resiliency and governance practices as this is the second major outage in less than 18 months for the company.
- Did the update need to happen during the week? If possible, a core network update should be done over night on the weekend to minimize disruption if things go wrong. If this outage was on Sunday July 10th vs Friday July 8th, it would have been less painful for businesses who don’t operate on the weekend.
- What testing was done in advance on this update? An update as important as this should have gone through rigorous tests in advance to ensure there were no issues.
- What was the fallback plan if the update was unsuccessful? Given the scrambling by Rogers on July 8th, the failure of the update was unexpected and there was no plan to go back to the previous version. There should always be a contingency plan if things go wrong especially for an update as critical as this one.
- How will Rogers address this in their network design in the future so their entire core network isn’t offline in the future when an update goes bad? Core network design is complex but should also include multiple points of redundancy to prevent failures such as this.
- Given the reliance of all businesses on the Internet, consider getting a second Internet connection from a different provider as a backup. In this case, a Bell or Telus backup connection would have allowed your business to continue to function. There is some planning to do and hardware to purchase to make this a seamless transition but is worth investigating.
- If your cell phone is your primary source of communication, consider an IP phone service as a secondary option if cell service is down. This is dependent on having Internet service but can allow you to keep in contact with clients.
- If you still host servers in your environment then consider moving to platforms like Microsoft 365, Microsoft Azure, or Google Workspace which have multiple levels of network redundancy as part of their service offering.
- Ask any outsourced service providers you use to confirm that they are not reliant on a single network provider to prevent them from being inaccessible if a vendor network goes offline.