Here’s What Caused Facebook’s Global Outage

Global Facebook Services Outage

Facebook recently had a terrible outage that caused all its services to go offline globally. The incident caught attention as the users flooded Twitter with WhatsApp, Instagram, and Facebook’s inaccessibility complaints.

Many people initially suspected it to be an issue with their ISP, it turned out that the issues appeared at Facebook’s end. Consequently, Facebook, WhatsApp, Instagram, and Oculus remained unavailable for over six hours, causing massive trouble.

It is pertinent to mention that Facebook services hold a necessary stance for business and individual users for their day-to-day communications and business activities. Hence, these outages were frustrating for some users.

Finally, Facebook authorities confirmed the glitches via their official Twitter accounts.

After all the chaos, Facebook has finally revealed what triggered the issue.

As elaborated in a blog post from Santosh Janardhan, Facebook’s VP, Infrastructure, the outages happened due to a technical issue during “configuration changes on backbone routers”.

Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.

This problem affected not only the services but also the tech giant’s internal systems. Hence, it became difficult for the teams to resolve the matter.

According to reports, even Facebook employees couldn’t enter their workplaces as their badges to access the doors stopped working.

What Facebook Disappeared From The Internet

Although Facebook’s explanatory post doesn’t reveal many technical details about why exactly it suddenly disappeared from the internet. What people observed during the outages was that visiting “facebook.com” would return no results. Rather the domain appeared to be ‘for sale’ (due to a third-party mess-up unrelated to Facebook’s technical issue).

Explaining the matter, Johannes B. Ullrich from the SANS Technology Institute highlighted the problem as a DNS error. As elaborated,

The BGP routes pointing traffic to Facebook’s IP address space have been withdrawn. The Internet no longer knows where to find Facebook’s IPs. One symptom is that DNS requests are failing. But this is just the result of Facebook hosting its DNS servers inside its own network. Even with working DNS (for example if you still have cached results), the IPs are currently not reachable.

Service Restoration Underway

Hours after the outages, Facebook officials eventually started restoration efforts (of course, after getting into the data centers, finally). As Janardhan explained in the post, the team is actively working for 100% service restoration. He also ruled out any cyber attacks behind this situation.

We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.

Indeed, many users worldwide can now access their Facebook, Instagram, and WhatsApp accounts again. Anyone still facing issues should rest assured to get the access back shortly.

Let us know your thoughts in the comments.