Currently there is a known disruption in MUC Datacenter

Incident Report for InterNetX

Postmortem

Dear customers of InterNetX GmbH,

On Friday, 16.04.2021, starting at 7:30 a.m, the technical operator of our Munich data center carried out a routine inspection on protective devices of the medium voltage supply, which included maintenance of the transformers and circuit breakers. During a step preceding the actual maintenance, the load swing of the redundant UPS systems, which according to the technical operator had already been carried out successfully several times in the previous week, there was a power interruption of approx. 0.5 seconds at around 7:55 a.m. The period in question was so short that some systems continued to run without interruption. However, numerous systems performed a restart. As a result, several services and systems had to be restarted, in some cases manually. Since Friday morning, our employees have been undertaking everything in their power to restore the availability of all systems/services as quickly as possible and to provide affected customers with appropriate support. The name service was not affected due to redundant distribution across several sites.  

The reason for the interruption of the multiple redundant power supply is currently being analyzed. For this purpose, log files were secured by the manufacturer of the UPS system on Saturday and are currently still being evaluated. As soon as we have more information about the cause, we will notify you. We thank you for your understanding.

We would like to apologize for any inconvenience caused!

Posted Apr 19, 2021 - 20:35 CEST

Resolved

This incident has been resolved.
Posted Apr 16, 2021 - 21:08 CEST

Update

We are continuing to monitor for any further issues.
Posted Apr 16, 2021 - 20:17 CEST

Update

We are continuing to monitor for any further issues.
Posted Apr 16, 2021 - 19:40 CEST

Update

We are continuing to monitor for any further issues.
Posted Apr 16, 2021 - 19:15 CEST

Update

We are continuing to monitor for any further issues.
Posted Apr 16, 2021 - 18:49 CEST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Apr 16, 2021 - 18:23 CEST

Update

We are continuing to work on a fix for this issue.
Posted Apr 16, 2021 - 18:23 CEST

Update

We are continuing to work on a fix for this issue.
Posted Apr 16, 2021 - 18:04 CEST

Update

The "Redirect Service" is back online.
If you are still experiencing issues with your system or service, please feel free to contact support@internetx.com
Posted Apr 16, 2021 - 17:59 CEST

Update

We are continuing to work on a fix for this issue.
Posted Apr 16, 2021 - 14:54 CEST

Update

Most of the services and systems are woking as expected.
If you are still experiencing issues with your system or service, please feel free to contact support@internetx.com
Posted Apr 16, 2021 - 14:06 CEST

Update

We are continuing to work on a fix for this issue.
Posted Apr 16, 2021 - 13:13 CEST

Update

We are continuing to work on a fix for this issue.
Posted Apr 16, 2021 - 13:12 CEST

Update

We are continuing to work on a fix for this issue.
Posted Apr 16, 2021 - 11:27 CEST

Update

We are still working on recovering some services.
Please feel free to contact support@internetx.com at any time if you are experiencing issues with your systems or services.
Posted Apr 16, 2021 - 11:16 CEST

Identified

The issue has been identified and a fix is being implemented.
Posted Apr 16, 2021 - 08:30 CEST

Investigating

Currently, we are experiencing a disruption in our MUC Datacenter. We have taken immediate action to resolve the issue. We apologize for any inconvenience and thank you for your understanding.
Posted Apr 16, 2021 - 08:00 CEST
This incident affected: AutoDNS (User Interface, API), ISAC (User Interface, API), Server (Cloud Server, Bare Metal Server, Kubernetes Cluster, Reseller Cloud), SSL Manager (User Interface, API), Domain Services (Redirector, MailProxy, WhoisProxy), Infrastructure (Data Center MUC), and InterNetX Website.