UPDATED Google Cloud went down hard on Thursday, and took Cloudflare and some of its customers with it.
Google published its first status update 11:46 Pacific Daylight Time (PDT) when it reported over 40 of its locations and 26 services were “experiencing impact due to Identity and Access Management Service Issue.”
Google’s Cloud Console was among the afflicted services, meaning users couldn’t log in to figure out why other impacted services like Cloud Storage and Cloud SQL weren’t working.
At 12:30 PT, Google posted news that the incident was over, other than in the us-central1 region, which runs out of facilities in Iowa.
But at 12:41 the company admitted “customers are still experiencing varying degrees of impact on individual google cloud products.”
Google assured customers it’s figured out what went wrong and that “All the respective engineering teams are actively engaged and working on service recovery.”
Cloudflare also went down on Thursday.
“This is a Google Cloud outage,” a Cloudflare spokesperson told The Register. “A limited number of services at Cloudflare use Google Cloud and were impacted. We expect them to come back shortly. The core Cloudflare services were not impacted.”
Interestingly, Cloudflare posted news of its own problems at 11:19 PDT – suggesting it felt the impact before Google informed customers about its woes.
At 12:12 PDT Cloudflare reported it was “starting to see services recover” but warned, “We still expect to see intermittent errors across the impacted services as systems handle retried and caches are filled.”
The effects of the incident appear to have been immense, as according to Downdetector Azure, Spotify and Discord all had problems at the time of the Google outage. The Register fancies that’s due to a domino effect as Google went down, took Cloudflare with it, and Cloudflare’s customers flamed out.
A spokesperson for the Chocolate Factory’s cloud arm offered the following helpful statement: ““We are currently investigating a service disruption to some Google Cloud services. Please view our public status dashboard for the latest updates.”
This is a developing story and The Register will update it as more information becomes available.®
UPDATED AT UTC 21:15/PDT 14:15
A 14:00 PDT update from Google states: “We have implemented mitigation for the issue in us-central1 and multi-region/us and we are seeing signs of recovery.”
The search monopolist also stated it has “received confirmation from our internal monitoring and customers that the Google Cloud products are also seeing recovery in multiple regions and are also seeing signs of some recovery in us-central1 and mutli-region/us.”
The company expects recovery to complete in “less than an hour”.
We have also removed a reference to AWS from this story, after the cloud giant told us it has not experienced an outage.
