On Monday, numerous Internet users in the USA faced trouble after Google went down for over an hour. Upon scratching the surface, researchers found a traffic hijacking issue with Google services. While the initial speculations hinted towards an intentional attack by a Nigerian ISP, it later was confirmed to be down to a BGP mishap.
Google Went Down As It Suffered IP Hijack By MainOne
In a recent blog post, Ameet Naik, Technical Marketing Manager at ThousandEyes sheds more light on the incident. According to the report, the firm faced troubles connecting to G Suite, after which they began working to find the problem. Eventually, they noticed Google’s traffic dropping at China Telecom from numerous ThousandEyes vantage points globally.
Upon further research, they noticed an IP hijacking attack. As explained,
“Traffic from Paris to www.google.com resolved to 216.58.204.132. While Google announces many /24 prefixes to cover its IP address range, this address was not covered by a /24 prefix. Instead, it was covered by a /19 prefix. We saw a suspicious announcement for 216.58.192.0/19 appear after about 12:45 pm PST with a convoluted AS path that included TransTelecom (AS 20485) in Russia, China Telecom (AS 4809) in China and MainOne (AS 37282), a small ISP in Nigeria. The traffic paths we saw mirrored the BGP AS Path, except all the traffic slammed into the great firewall, terminating at China Telecom edge router.”
According to Naik, ThousandEyes noticed found more than 180 prefixes impacted by this incident. Nonetheless, BGPmon, who first disclosed the problem in a tweet, reports about 212 prefixes which they also have shared online.
How Did It Happen?
Regarding what could have happened to the services, BGPmom stated,
Appears that Nigerian ISP AS37282 'MainOne Cable Company' leaked many @google prefixes to China telecom, who then advertised it to AS20485 TRANSTELECOM (russia). From there on others appear to have picked this up.
— Cisco BGPmon (@bgpmon) November 12, 2018
Naik explained the matter in a bit more detail.
“Our analysis indicates that the origin of this leak was the BGP peering relationship between MainOne, the Nigerian provider, and China Telecom. MainOne has a peering relationship with Google via IXPN in Lagos and has direct routes to Google, which leaked into China Telecom. These leaked routes propagated from China Telecom, via TransTelecom to NTT and other transit ISPs.”
The incident eventually caused Google Cloud Platform, Google Analytics, and Google Search to go down. The problem began at 13:12 and ended at 14:35 (Pacific Time) on November 12, 2018, lasting for about 74 minutes.
MainOne Confirmed The Problem Arose Erroneously
After the incident, Google also disclosed the problem on their Cloud Platform website. However, they didn’t mention anything about the source behind this trouble. They only stated they would investigate the matter.
However, in a tweet later, MainOne confirmed their role behind this trouble.
We have investigated the advertisement of @Google prefixes through one of our upstream partners. This was an error during a planned network upgrade due to a misconfiguration on our BGP filters. The error was corrected within 74mins & processes put in place to avoid reoccurrence
— MainOne (@Mainoneservice) November 13, 2018
This certainly rules out any deliberate moves behind this incident. Nonetheless, it definitely highlights the weaknesses in the old BGP that dates back to the 1980s.
Let us know your thoughts about the article in the comments section below.
