Outage Details
Sportsman Cloud Outage
As you are certainly aware on Tue, June 20th, 2023 from approximately 6 am to 2:30 pm we experienced a service outage. We take these very seriously and understand the enormous impact on your operations. In this case, all our systems were down including Sportsman Web, ActivityReg, and Sportsman Desktop.
While the specific details are still coming in, the problem was a network connectivity error at the Data Center which houses all public-facing Peak Software servers. We put our servers and your data into these facilities for maximum security, performance, and reliability.
The data center upgraded some networking hardware which produced the fallout. Apparently, it generally was successful except for a handful of their customers. Because it was generally working, it took longer to figure out our problem and correct it. Today’s outage represents one the largest outages we’ve seen in our company’s history. The problems today were on the data center side and out of our direct control.
We monitor our networks 24/7 with automated alerts. Upon the failure, all efforts were put into bringing the systems back up. Peak Software is responsible for the actual servers and software however the network connectivity is the responsibility of the data center. Our servers were never down nor did they have any data loss. We maintain redundant hardware to mitigate failures. In addition, we maintain redundant communication lines however the nature of the problem rendered both unusable. Soon after the point of failure we filed a critical issue and were in regular communication with technicians until it got restored. Our email server was also in the blackout so we could not send or receive emails.
Why the data center was upgrading hardware anywhere near a business day is a point of major contention. We have spoken to them directly about this and the event is under investigation. It is their policy to do such upgrades outside business hours. Specifically, Saturday nights are typically the best window for these events where traffic is at its minimum. This is the timeframe Peak Software typically does any maintenance that could involve downtime.
We spoke to a lot of customers today. It is important to report any outages you might experience. While we do monitor our systems and are likely aware of an outage, there is a chance it’s only relative to you so we want to know.
Again, I apologize for the trouble today. We are working with the data center to minimize the likelihood of future outages.
Sincerely
Charlie Warrell
Peak Software Systems
Director of Development / CEO