On August 29, 2023 at 10:38 PM ET we sent an email to a select group of customers using our Trust Center Updates custom audience feature.
Within a few minutes, several customers alerted us to their inboxes receiving multiples copies of the same TCU email. We began investigating, and we determined that this was a bug with our email notification management system, Courier.
Within 20 minutes we were able to write a script to cancel all emails that were queued up and waiting to be delivered. However, we noticed that the queue continued to be rebuilt, so we contacted Courier support. To mitigate the number of emails being, we continuously ran this script to reduce the likelihood of additional duplicate emails.
Courier support joined us on a Zoom call at 12:01 PM ET and advised that we halt all Courier related emails until the issue was sorted out.
At 1:00PM ET the Courier team notes that they discovered the issue and would have a hotfix available within an hour or two. The root cause of this bug was related to logic that attempted to continuously resent this email due to an issue related to s3 timeouts/socket connections and an improper way of marking the emails as delivered when this happened.
At 3:56 PM ET the Courier team informed us that the fixes were in place, and that emails could safely be delivered again.
At 4:00 PM ET we re-enabled the Courier integration to allow for emails to flow once again. The Courier team began to gather any emails that were not sent during this outage period.
At 7:23 PM ET the Courier team confirms that all emails that were stuck during the outage period were delivered.
Posted Sep 01, 2023 - 08:21 PDT
Resolved
All emails in the backlog have been processed.
Posted Aug 29, 2023 - 16:28 PDT
Update
Emails that were not delivered during this outage are now being sent out. We appreciate your patience as we ramp back up to full operational capacity.
Posted Aug 29, 2023 - 14:06 PDT
Monitoring
A fix has been deployed and we are monitoring. Emails that were backed up during the outage will be processed shortly.
Posted Aug 29, 2023 - 13:01 PDT
Update
Courier has noted another fix is on the way that will require another 60 minutes. We appreciate your patience.
Posted Aug 29, 2023 - 11:49 PDT
Identified
The downstream vendor, Courier, has identified the bug on their end and are working on deploying the fix now. The ETA is 90 minutes. We will be working with them to send out any queued emails that were generated during this outage.
Posted Aug 29, 2023 - 09:54 PDT
Update
We are continuing to investigate this issue.
Posted Aug 29, 2023 - 09:07 PDT
Investigating
We have temporarily stopped email notifications until our email provider has a fix in place.
Posted Aug 29, 2023 - 09:06 PDT
Identified
The issue is with our email infrastructure provider. We are working with them to resolve the issue urgently. Some customers might be receiving duplicate emails.
Posted Aug 29, 2023 - 09:03 PDT
Investigating
We are investigating an issue where certain trust center updates are sending repeatedly to recipients if uploaded via a CSV.