Ezrinix - Equinix CH1 Chilling Failure – Incident details

Experiencing partially degraded performance

Equinix CH1 Chilling Failure

Resolved
Degraded performance
Started over 1 year agoLasted 4 days

Affected

Ezrinix

Degraded performance from 3:20 AM to 5:25 AM

Premium VPS

Degraded performance from 3:20 AM to 5:25 AM

CHI-P-VPS-1

Degraded performance from 3:20 AM to 5:25 AM

Budget VPS

Degraded performance from 3:20 AM to 5:25 AM

CHI-B-VPS-1

Degraded performance from 3:20 AM to 5:25 AM

Web Hosting

Degraded performance from 3:20 AM to 5:25 AM

Updates
  • Resolved
    Resolved

    It's late notice, but operations at Equinix CH1 have begun to stabilize thanks to the swift response by their team. As a result, we are beginning to rotate more of our traffic back to our carrier endpoints downtown.

  • Update
    Update

    Update from Equinix: 10 out of 15 portable chillers have been activated. Temperatures have continued to decrease to an average 88F.

  • Update
    Update

    Update from Equinix: 3 more portable chillers have been installed. Twelve more are expected to come online shortly. Temperatures have continued to decrease to an average of 95F.

  • Update
    Update

    Update from Equinix: 4 chillers are currently active, and temperatures are starting to stabilize. More equipment is being brought on-site to assist with cooling.

  • Update
    Update

    Update from Equinix: 5 out of 15 portable chillers have arrived on-site. More are expected to arrive shortly.

  • Monitoring
    Monitoring

    Update from Equinix: 2 out of 8 chillers are now fully operational. A third chiller is operating at a reduced capacity. Temperatures in CH1 are beginning to decrease as more chillers are returning online. Portable chillers are expected to be on-site shortly.

  • Resolved
    Resolved

    Update from Equinix: One chiller is fully operational now, with another one operating at reduced capacity. Technicians are still working to restore the necessary 4-out-of-6 chillers to return the temperatures back to normal conditions. Portable chillers should have been delivered around 9 PM CT and would be setup by technicians as soon as possible.

  • Monitoring
    Monitoring

    Update from Equinix: Technicians continue to troubleshoot the chiller plant failure. IBX Critical Facility Engineers have deployed all available floor fans, opened all available doors and have engaged local rental company to source additional fans and portable spot coolers.

  • Resolved
    Resolved

    Our automation system has rolled over traffic to our on-net carriers due to losing complete connectivity to our endpoint in Equinix CH1. As traffic was moved, you may have experienced a brief disconnect or increased load times as routes readjusted on these alternative carriers. We will continue to monitor and provide updates.

  • Monitoring
    Monitoring

    Equinix CH1 (5th floor), located at 350 E Cermak Rd in Chicago, is currently experiencing multiple chiller plant failures. This has led to increased temperatures in the facility, resulting in several fiber optics failing and the shutdown of connections. Equinix has requested customers power down non-essential loads within the facility in an attempt to decrease temperatures. There are currently technicians on-site from Equinix as well as the chilling unit vendor. As of right now, there is no ETA on when the units will be restored.

    Due to a large portion of our traffic terminating in CH1, we have taken precautions in routing as much traffic away from that site as possible. However, if temperatures continue to rise, there is a possibility that critical network equipment may fail. If that is the case, we will route traffic to our backup carriers on-net in our facility, located at 603 Discovery Dr in West Chicago. Our team is continuing to monitor the situation and will make necessary changes. We will provide updates as we receive them from Equinix.