Recent Outage Debrief
This section should describe recent outages, their symptoms, and their recovery. Outages are categorized and organized by the date of their occurrence. If an outage spans categories, it will fall under that which was most severely impacted.
Internet outages are outages where a portion of, or the entirety of, the network is unable to communicate or route to critical services include internet, email, cloud hosted services, or other network services important to business continuity.
Outage Type: Internet
3:40AM - Outage begins with MxToolbox notice
6:34AM - Mike Jose is notified by Andy Ullom
6:35AM - Mike Jose arrives at the datacenter
6:37AM - Mike Jose confirms with Andy Ullom that the outage is external not internal
7:00AM - Andy Ullom arrives at the datacenter
7:15AM - Outbound call to Windstream support to create a ticket on the issue
7:23AM - Inbound call from Windstream technician stating the issue was with upstream device
7:31AM - Outage ends with MxToolbox notice
Outage Duration: ~3 hours 51 minutes
The outage was first noticed as a result of MxToolbox responses. Andy Ullom called Mike Jose at 6:34AM to investigate. Mike Jose arrived at the datacenter at 6:36AM and confirmed that the network was working but not out the network edge. This was performed using pings internally to reach various internal services successfully. Edge was confirmed down by connecting directly to the DMARC device and setting a laptop to the routable IP address (18.104.22.168/30) and attempting to ping the next hop (22.214.171.124/30). This was tested again by using cell phones to attempt to ping the outside IP of 126.96.36.199, which should have been reachable if that end was up, but it was not.