Outage (all hosted services)
Incident Report for Sol1
Postmortem

What happened?

On 14 November 2021, between 12:13pm and 13:30pm, Sol1 experienced a major outage within its core network hosted at the Equinix SY3 facility.

How were customers impacted?

Due to the nature of the failure, DNS services were impacted, which had a flow-on effect to most other services within our SY3 network including mail, web and other hosted services.

Additionally, for customers relying on DNS nameservers, other external systems may have been affected.

What went wrong during the incident?

A core network switch failed and could not be recovered remotely. Engineers were dispatched to Alexandria immediately to investigate and implement a workaround.

What will Sol1 do to prevent this in the future?

It has been identified the switch that failed is a single point of failure. We have now moved twin network switches into position within the physical network, and a cutover to the new equipment will be performed at an appropriate time.

Posted Nov 15, 2021 - 00:18 UTC

Resolved
Connectivity issue
Posted Nov 14, 2021 - 01:30 UTC