Scheduled for February 27th, 2025 at 3:00 PM – March 16th, 2025 at 9:07 PM
Affects
Chicago, Illinois Data Center
Under maintenance from 3:00 PM to 9:07 PM
Core Network Infrastructure
Under maintenance from 3:00 PM to 9:07 PM
Updates
Completed
March 16th, 2025 at 9:07 PM
Completed
March 16th, 2025 at 9:07 PM
Our provider has notified us that the new router will be installed soon, therefore a new maintenance window will be posted sometime today or tomorrow.
Update
March 11th, 2025 at 2:49 AM
In progress
March 11th, 2025 at 2:49 AM
Our provider is awaiting a new edge router to arrive at the data center, once it arrives, work will resume on this maintenance. We appreciate everyone's patience with this.
Update
February 28th, 2025 at 1:50 AM
In progress
February 28th, 2025 at 1:50 AM
Maintenance for this has been completed today. We've found the root cause to these Layer 2 issues relating to the router appliance itself. Luckily we have built the network to be redundant, so we will be shifting traffic over to a working router while we work on getting this defective one replaced.
Again this maintenance will continue to take place at 9:00 AM Central time to avoid dropping players in the peak hours of the day. In the mean time some changes have been made to lessen the effects of these spikes.
To clarify, we would've had this done today, however the backup router was still in the process of being setup and was paused while Onsite was being developed. We needed a few prerequisites completed from our upstream before we could shift any traffic onto it. Which is why we are continuing this tomorrow as it went too far into prime time today to justify dropping traffic.
Update
February 27th, 2025 at 8:33 PM
In progress
February 27th, 2025 at 8:33 PM
We are pushing some changes, you may notice a slight interruption.
In progress
February 27th, 2025 at 3:00 PM
In progress
February 27th, 2025 at 3:00 PM
Maintenance is now in progress
Planned
February 27th, 2025 at 3:00 PM
Planned
February 27th, 2025 at 3:00 PM
Hello everyone, our network team has identified the root cause of the frequent short drops that have plagued the network over the past few days. We understand the severity of these drops and have identified the root cause. To permanently resolve these issues, we will be conducting emergency maintenance at 9:00 AM Central Time. During this maintenance, you will experience brief disruptions as mass changes will be done. We apologize in advance for any inconvenience this may pose to you, especially over these past few weeks when we have suffered instability from one of our upstream providers as well as the network maintenance we conducted to upgrade our core late last month. To be frank, the timing could not have been worse.
For transparency, the issue stems from a Layer 2 issue inside our EVPN-VXLAN network, which is affecting our spine switches that process essentially all traffic on our network. As a result, this was felt widespread and not just related to one upstream or one cabinet. This issue appears to have sprung from the preparation of onboarding our on-prem protection suite into our core. We have been working at a fast pace to get that online and operational in hopes of shifting all traffic over to provide stability.
We know the past few weeks have been difficult, and we recognize the importance of a reliable network for our customers and are tirelessly working to improve our offerings.
