Google Cloud Outage: Engineer Cuts Fiber, Partial G-Cloud Downtime

Incident at Google Cloud

On September 1st, a region of Google Cloud's us-central1-b zone experienced partial outage after the accidental disconnection of all fiber optic cables during routine maintenance.

Causes

• Maintenance procedure that disconnected 100% of fiber routes in 13 minutes.
• Lack of prior alerts that prevented the engineer from detecting the error.

Impact

Resources hosted in the affected area experienced "severe network degradation and resource isolation," with 100% traffic drop rates and packet loss.

Google Response

The team rerouted traffic to "healthy" paths in other parts of the region and physically restored the optical links. Once connectivity was restored, traffic returned to its original path.

Lessons Learned

• Physical redundancy does not replace operational discipline.
• Maintenance procedures must include state validation and automatic notifications.
• The culture of "read the documentation" (RTFM) remains essential.

Source: The Register