April 9th – Problem with billing system

8:30 AM UTC. Some of our customers have reported issues with duplicate charges being applied to their accounts.

Our engineering team is working on fixing this. We will be refunding all the incorrect charges applied to the accounts.

If the money was deducted from your Credit card or PayPal account, we will refund the money back to it.

2:45 PM UTC. We have identified an issue where phone number and Premium subscriptions seem to have been deactivated for some of our customers. We are working on restoring these subscriptions as soon as possible.

Due to the increased influx of emails, our responses will be delayed. If you have sent an email to us about these incorrect charges we will let you know as soon as the refunds are processed.

5:54 PM UTC. Most of the phone number subscriptions that had been removed from customer accounts have now been restored and should be working as before. We are working on reassigning the rest of the phone numbers.

April 10th 3:51 PM UTC. All the duplicate charges applied yesterday have now been refunded back to the respective Sonetel prepaid accounts.

April 11th 6:30 AM UTC. We are looking into reports that some customers have not received refunds for the duplicate charges. We are verifying the charges for the last 2 months to ensure no refunds have been missed. All pending refunds will be processed as soon as possible.

7:00 AM UTC. While processing the refunds yesterday an unexpected issue caused some of the refunds to be missed. The problem was discovered today and it is being addressed by the engineering team. Once this is resolved we will process the rest of the refunds as well.

Update:
The issue was caused by a unique combination of a design flaw, a bug and insufficient automatic monitoring in a system that has been working without issues since over 3 years.

Here is an explanation of what went wrong:
Our recurring billing system has a failover function that ensures that it can handle subscription renewals even if the primary renewal function is not working for any reason.

The primary renewal function had stopped working since some time due to a bug, and renewals were therefore handled by the failover function. This was not discovered directly as there was no external automatic monitoring of this specific function.

When the issue was discovered by our team, they restarted the primary function for renewals.
Upon restart, the primary renewal function had a large queue of renewals pending which had built up while the failover function handled renewals.
To avoid duplicate charges, the primary function is designed to check what has been billed by the failover function – but it only checks the last two days. This was a design flaw.

Unfortunately – in this case – since the primary function had been off-line for more than two days, this caused it to charge duplicate renewal charges that already had been charged by the failover function.

So it was thereby a unique combination of events that broke the safety and failover methods we had in place and lead to duplicate renewals being charged in some customer accounts.

The design flaw and the bug have been corrected and additional monitoring of the primary renewal function has been added to ensure that this problem never can occur again.

We sincerely apologize for the inconvenience.