[OUTAGE ALERT] 6/27 - Messages not sending
Latest update - we have rolled a credit refund to all accounts affected. We have restated workflows now. We continue to monitor for any other possible issues.
As of this morning - we received multiple reports of messages not sending.
We have confirmed after speaking with carrier, that the outage started this morning at midnight, due to an unscheduled update by one of the middle stream telecom network companies.
This is currently affecting not only OnlySales, but other platforms, built on SMS routing through Syniverse.
We're looking at alternatives to re-route messages through other networks for today. We're speaking with the CEO/CTO of our carrier, Telgorithm, and continue to monitor.
For the moment, we suspended the workflow service - so that messages don't auto send to a carrier, that cannot at the moment deliver that same message.
Any messages billed today but not delivered as expected will be refunded as credits automatically end of day.
The banner will be dismissed from view, as soon as the issue is resolved. Updates will be added to this post as we get more information.
— Andre Arancibia
Founder/CEO
FAQ's from tickets this morning:
- Were messages caught in this error billed to my account?
Yes - our billing is tied to our carrier. Carrier confirmed the messages were sent, which triggered the billing. But messages cannot complete the loop at the moment and arrive at a number.
- If they were billed to my account - will I be charged again once they're sent?
No, they will not be charged a second time as the issue is resolved and they start to deliver.
- Can I receive credits due to this disruption today?
Yes, end of day, once everything is resolved, we'll process an automated credit refund to those accounts affected by this issue. This credit will come directly out of our company funds, even though we ourselves will be charged for the messages. We understand what the cost of a message sending disruption is to your business, and always strive to make it right - even if, in this case - it was not directly related to us, or our code, and even if we ourselves will be charged for it.
I started this company when working as an insurance agent. I was in your shoes not too long ago. Our platform is built to be as reliable as possible, and as generous with error corrections and refunds, as I would have wished a platform was, when I was using similar software. - Andre
10:34 AM EST: From CEO/CTO of carrier directly. Error has been tracked to third party telecom company. The disruption in message sending, is tracked to have started, at midnight today. Only on messages from Telgorithm carrier (Numbers in our platform tagged as [N])
Once we got this confirmed, we suspended the workflow service, so that messages don't auto send, but don't route.
12:31 PM EST: From CTO of carrier directly. In call currently with gateway provider, trying to find alternatives for routing messages for today. We'll be updated once the call is complete, will update this document once we do.
12:46 PM EST: We now have confirmation that the error should be resolved, within 60 to 90 minutes. When it is fully resolved, we will restart the workflow service (auto sending messages). Messages sent earlier today, before we paused workflows, will send once the fix takes place.
2:40 PM EST: As of now, messages are being delivered. We have restarted workflows sending again. As for messages sent earlier today, some are sending, some might still be in process from what we're hearing. Should resume sending in the next few minutes:
