What happened
On Tuesday, September 9, between 13:12 and 13:40 UTC (15:12 to 15:40 Paris time), the Allo Core was slow and some requests failed. During those 28 minutes, roughly 5% of requests returned errors and many others took several seconds to complete. In the apps this showed up as slow loading of conversations and contacts, delayed presence updates, and some calls or messages taking longer than usual to appear. No data was lost.
Why it happened
Our cloud provider automatically rolls out security updates to the servers that run Allo. That afternoon, a new server image was released and the platform began replacing every server, one at a time. This is normal maintenance and our application did not crash at any point.
The problem was timing. The rollout started during our busiest hour of the day. Each time a server was replaced, the copy of Allo running on it moved to a new server and started cold. A freshly started instance handles requests several times slower than a warm one for its first few minutes. At peak traffic, the remaining warm instances were already close to their database connection limit, so they could not absorb the extra load. Requests queued for a database connection, timed out, and returned errors. This repeated with each server replaced until the rollout finished and the new instances warmed up.
The database itself was healthy throughout. The bottleneck was on our side: too little headroom at peak, and instances that take too long to reach full speed.
What we are doing about it
Moving maintenance out of business hours. So routine updates never land during peak traffic.
Adding capacity at peak. We are increasing the number of running instances during weekday afternoons, so losing or replacing one no longer pushes the others past their limit.
Starting faster. We are shipping changes that cut instance startup time by roughly 40% and warm up the most-used code paths before an instance receives traffic.
We are sorry for the disruption.
