tag:status.safebase.io,2005:/historySafeBase Status - Incident History2024-03-28T22:23:01-07:00SafeBasetag:status.safebase.io,2005:Incident/189685682023-10-30T07:54:19-07:002023-10-30T08:49:35-07:00Fly.io Down<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>07:54</var> PDT</small><br><strong>Resolved</strong> - This has been fixed.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>07:32</var> PDT</small><br><strong>Investigating</strong> - Our Fly.io reverse proxy is currently down due to a Fly.io issue. We are working with them to restore access as soon as possible.<br />This only affects customers whose public Trust Centers are proxied through Fly.</p>tag:status.safebase.io,2005:Incident/187663262023-10-12T14:55:37-07:002023-10-12T14:55:37-07:00Issues with Tray integration middleware.<p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>14:55</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>13:29</var> PDT</small><br><strong>Monitoring</strong> - A hotfix has been released and we are monitoring for continued issues.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>08:01</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with Tray.io, our middleware provider that allows for HubSpot, Salesforce, and Jira integrations. We are engaged with the provider and are working with them to restore service ASAP.</p>tag:status.safebase.io,2005:Incident/187670682023-10-12T09:51:11-07:002023-10-12T09:51:11-07:00Page is slowly response<p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>09:51</var> PDT</small><br><strong>Resolved</strong> - The DDoS attack has been stopped and we have added additional mitigations to reduce the risk of future DDoS attacks.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>09:03</var> PDT</small><br><strong>Identified</strong> - We are experiencing a DDOS attack, we are handling it now.</p>tag:status.safebase.io,2005:Incident/183082982023-08-29T16:28:47-07:002023-09-01T08:21:24-07:00Email notifications sending excessively<p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>16:28</var> PDT</small><br><strong>Resolved</strong> - All emails in the backlog have been processed.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>14:06</var> PDT</small><br><strong>Update</strong> - Emails that were not delivered during this outage are now being sent out. We appreciate your patience as we ramp back up to full operational capacity.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>13:01</var> PDT</small><br><strong>Monitoring</strong> - A fix has been deployed and we are monitoring. Emails that were backed up during the outage will be processed shortly.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>11:49</var> PDT</small><br><strong>Update</strong> - Courier has noted another fix is on the way that will require another 60 minutes. We appreciate your patience.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>09:54</var> PDT</small><br><strong>Identified</strong> - The downstream vendor, Courier, has identified the bug on their end and are working on deploying the fix now. The ETA is 90 minutes. We will be working with them to send out any queued emails that were generated during this outage.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>09:07</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>09:06</var> PDT</small><br><strong>Investigating</strong> - We have temporarily stopped email notifications until our email provider has a fix in place.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>09:03</var> PDT</small><br><strong>Identified</strong> - The issue is with our email infrastructure provider. We are working with them to resolve the issue urgently. Some customers might be receiving duplicate emails.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>08:09</var> PDT</small><br><strong>Investigating</strong> - We are investigating an issue where certain trust center updates are sending repeatedly to recipients if uploaded via a CSV.</p>tag:status.safebase.io,2005:Incident/181633932023-08-16T18:33:02-07:002023-08-16T18:33:02-07:00Intermittent app hanging<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>18:33</var> PDT</small><br><strong>Resolved</strong> - This was determined to be due to a downstream issue from Cloudflare, who have since issued a fix.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>14:00</var> PDT</small><br><strong>Identified</strong> - We have identified the root cause and are working with our infrastructure provider to resolve those issues.<br />Customers experiencing a hangup issue can refresh the browser, or go directly to app.safebase.io again in order to work around this in the meantime.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>09:23</var> PDT</small><br><strong>Investigating</strong> - The issues are still not completely over.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>08:17</var> PDT</small><br><strong>Monitoring</strong> - We have deployed a fix and we're monitoring to make sure the issue isn't happening anymore.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>07:44</var> PDT</small><br><strong>Investigating</strong> - The app sometimes hands up for some users. We are investigating the root cause</p>tag:status.safebase.io,2005:Incident/177731902023-07-06T07:57:29-07:002023-07-06T07:57:29-07:00500 errors on main SafeBase application<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>07:57</var> PDT</small><br><strong>Resolved</strong> - This issue has been resolved and the app should be functioning as normal again.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>16:05</var> PDT</small><br><strong>Monitoring</strong> - We have identified the issue as a DDoS attack against our app. We have blocked the attack and we are monitoring the situation.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>11:10</var> PDT</small><br><strong>Investigating</strong> - We have received reports that users are currently seeing periodic 500 errors for our main application. We are investigating and will provide an update as soon as we can.</p>tag:status.safebase.io,2005:Incident/175635412023-06-14T18:29:10-07:002023-06-14T18:29:10-07:00Emails not sending out<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>18:29</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved and all emails have been sent</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>10:21</var> PDT</small><br><strong>Update</strong> - The issue has been resolved. We are verifying that all emails that had failed get resent properly</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>14:40</var> PDT</small><br><strong>Monitoring</strong> - Emails should be returning to normal. Our engineering team will continue to monitor.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>12:41</var> PDT</small><br><strong>Identified</strong> - We are currently having an issue with our email infrastructure service and emails are not being sent out.</p>tag:status.safebase.io,2005:Incident/174949102023-06-06T17:00:58-07:002023-06-06T17:00:58-07:00Portal Management Page is down<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>17:00</var> PDT</small><br><strong>Resolved</strong> - The issue has been resolved</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>16:23</var> PDT</small><br><strong>Identified</strong> - The Portal management page (/portal) in app.safebase.io is not working for some customers. Public portals are working as normal.<br />We are working on the issue and it should be resolved within an hour.</p>tag:status.safebase.io,2005:Incident/169949842023-05-06T09:00:08-07:002023-05-06T09:00:08-07:00DB Upgrade<p><small>May <var data-var='date'> 6</var>, <var data-var='time'>09:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>08:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>28</var>, <var data-var='time'>08:58</var> PDT</small><br><strong>Scheduled</strong> - We will be upgrading our DB instance to a larger machine to improve our performance and account for our recent growth. The app might be down for a few minutes while the instance restarts.</p>tag:status.safebase.io,2005:Incident/166144072023-03-22T22:11:39-07:002023-03-22T22:11:39-07:00Emails not being sent properly<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>22:11</var> PDT</small><br><strong>Resolved</strong> - The error seems to have just been a slight delay, no user impact has occurred.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>22:00</var> PDT</small><br><strong>Identified</strong> - We have noticed an error causing emails to not be sent in some cases. We are currently investigating.</p>tag:status.safebase.io,2005:Incident/153612672022-12-17T05:58:00-08:002022-12-17T05:58:00-08:00Emails delayed<p><small>Dec <var data-var='date'>17</var>, <var data-var='time'>05:58</var> PST</small><br><strong>Resolved</strong> - Postmark service issue seems to have been resolved and all emails in the queue have been delivered.</p><p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>14:02</var> PST</small><br><strong>Identified</strong> - Postmark, SafeBase's email delivery provider, is having an outage at the moment. This is causing emails to be delayed. Once service is restored, all emails in the queue will be delivered.</p>tag:status.safebase.io,2005:Incident/107111342022-07-28T11:59:57-07:002022-07-28T11:59:57-07:00Authentication Provider (Auth0) is down<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>11:59</var> PDT</small><br><strong>Resolved</strong> - Auth0 has resolved the issue on their end.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>10:38</var> PDT</small><br><strong>Monitoring</strong> - Auth0 is having global downtime, causing our main app login flow to randomly fail. If you are running into this issue, refreshing the page can help.<br /><br />We're monitoring the situation and will update when the service is back up.<br />You can follow Auth0 here: https://status.auth0.com/</p>tag:status.safebase.io,2005:Incident/106253642022-07-18T07:36:12-07:002022-07-18T07:36:12-07:00Emails not sending out<p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>07:36</var> PDT</small><br><strong>Resolved</strong> - This has been resolved.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>07:36</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>18</var>, <var data-var='time'>07:11</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with our email provider and emails not being sent out.</p>tag:status.safebase.io,2005:Incident/87839662021-12-07T17:28:26-08:002021-12-07T17:28:26-08:00Partial Email Outage<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>17:28</var> PST</small><br><strong>Resolved</strong> - AWS outage seems to have been resolved and our third party vendors are functioning properly again</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>09:56</var> PST</small><br><strong>Identified</strong> - Our email provider (Courier) is having a partial outage (https://status.courier.com/) due to an ongoing AWS incident (https://status.aws.amazon.com/). We are monitoring the situation and will update when these systems are back online.</p>tag:status.safebase.io,2005:Incident/85522142021-11-16T10:08:51-08:002021-11-16T10:08:51-08:00App down due to Google Cloud Major Outage<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>10:08</var> PST</small><br><strong>Resolved</strong> - This downtime appears to be resolved.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>10:08</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>09:57</var> PST</small><br><strong>Investigating</strong> - We are currently waiting for updates from Google, which is having a major outage with GCP at the moment.</p>tag:status.safebase.io,2005:Incident/77852912021-08-17T13:31:14-07:002021-08-17T13:31:14-07:00Reverse Proxy Outage<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>13:31</var> PDT</small><br><strong>Resolved</strong> - We have discovered a fix and have deployed it.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>13:23</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.safebase.io,2005:Incident/66192112021-03-26T16:40:15-07:002021-03-26T16:40:15-07:00Safebase main page is not loading<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>16:40</var> PDT</small><br><strong>Resolved</strong> - We have resolved the issue.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>16:26</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.safebase.io,2005:Incident/46876032020-07-22T00:57:00-07:002021-03-25T09:37:34-07:00This is an example incident<p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>00:57</var> PDT</small><br><strong>Resolved</strong> - Empathize with those affected and let them know everything is operating as normal.</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>00:47</var> PDT</small><br><strong>Monitoring</strong> - Let your users know once a fix is in place, and keep communication clear and precise.</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>00:52</var> PDT</small><br><strong>Identified</strong> - As you continue to work through the incident, update your customers frequently.</p><p><small>Jul <var data-var='date'>22</var>, <var data-var='time'>01:22</var> PDT</small><br><strong>Investigating</strong> - When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.</p>