What happened
On Thursday, September 4, between 17:55 and 18:45 UTC (19:55 to 20:45 Paris time), Allo was unable to handle many incoming calls. The apps were also slow or returned errors during this period.
No stored data was lost. Some contacts and companies created during these two windows may not have appeared in search results afterwards. We are re-indexing them.
Why it happened
A specific CRM update added a large number of team members at once. Due to a config defect, dozens of parallel copies of the same import ran at the same time, creating millions of duplicate records over a few hours.
The flood of jobs arrived faster than the queue could process them, and the queue's storage filled up. When that happened, the queue stopped accepting new messages.
Parts of our code publish messages to the queue while holding an open database transaction. With the queue blocked, those requests hung, holding database connections until the database forcibly closed them. Within minutes the database connection pools on every server were exhausted.
What we are doing about it
Underlying defect was fixed, so a CRM connection shared by a team synchronizes once.
Properly manage DB connections to not block them by async work.
Migrating self hosted queue engine to server-less to aim for best-in-class SLA.
We are sorry for the disruption, and especially for the calls that did not get through.
