Service degraded: Mail server issues
Incident Report for Dualog
Postmortem

On January 6th, 2021, at approximately 09:00 UTC, an issue with the mail flow was identified. As a result, email traffic would work intermittently with no particular pattern that could indicate an internal issue, causing a delay in the email delivery. The issue was fixed at 10:00 UTC, and it took around 2 hours to process the backlog of messages generated throughout the incident.

Further investigation showed that the issue was caused by a failure in Microsoft’s cloud infrastructure. The technical responses we have received from their servers have been "Temporarily unable to process your email. Please try again later" and "Mail sent to the wrong Office 365 region".

After highlighting the problem, Microsoft investigated and found a routing issue towards Denmark, which is where Dualog’s primary services are hosted. The result was that email traffic to and from ships, towards an O365 counterpart, would be affected, creating a delay. We applied several workarounds to ensure the messages would flow accordingly. Microsoft finally resolved the issue on January 7th at 14:00 UTC.

Our IT Infrastructure team has taken the necessary precaution to monitor this type of incident and avoid its occurrence in the future. We do apologize for the inconvenience this may have caused to your daily operations.

Posted Jan 08, 2021 - 11:54 UTC

Resolved
All systems back to normal.
Posted Jan 07, 2021 - 00:48 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jan 07, 2021 - 00:31 UTC
Update
The issue has been solved. We are monitoring the systems in case of further problems.
Posted Jan 07, 2021 - 00:31 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jan 07, 2021 - 00:29 UTC
Update
Our backend staff has identified that only customers using Office365 have been affected by the mailserver issue. All mail services now working as intended, while the old mail queued behind the degraded mailserver is slowly being released.
We will keep udating this page until the operation of affected server is normalized.
Posted Jan 06, 2021 - 22:37 UTC
Update
Mail traffic has now been rerouted from the affected mail server towards the other working servers, so new mail messages that are sent through Dualog are bound to be delivered as intended. There is, however, a queue of old messages behind the affected server, that will be cleared once we fix the ongoing issue.
Our backend team is still working on it. We will update this page as the situation evolves.
Posted Jan 06, 2021 - 21:38 UTC
Update
The backend staff is still working to fix the ongoing issues affecting one of our mail servers.
We will update this page as soon as the situation evolves.
Posted Jan 06, 2021 - 21:30 UTC
Identified
We are currently facing some issues with one of our mail servers.
Our backend team has identified the issue and is working on it. We will update this page as soon as the situation evolves.
Posted Jan 06, 2021 - 20:10 UTC
This incident affected: Email and Automatic File Transfer.