Discord Downtime..The who, what and why!

[cbizzle]

Discord Status


Unavailable Guilds
Resolved - We've traced the issue to problems caused during a routine deployment of new code, which after being rolled back resolved this incident. We'll continue to investigate and resolve the issues with the new version of this code.
Nov 16, 17:09 PST
Monitoring - Engineers have rolled back a deployment to some of our core infrastructure which we believe was causing these unavailable guilds. We're continuing to monitor and investigate.
Nov 16, 16:29 PST
Investigating - We're aware of issues causing some users to experience unavailable or offline guilds. We're currently investigating the root cause, and hope to provide an update shortly.
Nov 16, 16:21 PST

[cbizzle]

**UPDATE ON DISCORD SITUATION**


Monitoring - We've tracked the issue down to network issues within Google's networking. We're currently working with Google on a P0 ticket to isolate and resolve these issues, but they do not currently have an ETA.
Nov 17, 17:40 PST
Identified - We're yet again seeing issues connecting to and using Discord. We're again working extremely hard to mitigate and prevent these issues.
Nov 17, 17:22 PST
Monitoring - At this time we believe the majority of service has recovered. We're still investigating the root cause of these issues, but we believe they where caused by networking issues in our provider.
Nov 17, 15:15 PST
Identified - We're currently investigating a major outage which is effecting the vast majority of users. We hope to have an update within the next 5 minutes.
Nov 17, 14:31 PST

[cbizzle]

Update - Our team is still working with Google to identify the root cause of the networking issues. We're actively monitoring the platform to ensure any further networking issues are caught and resolved before they have a cascading effect on the system.
Nov 17, 19:35 PST

[cbizzle]

Major OutageSubscribe

Identified - We're yet again investigating a major outage causing offline guilds and connection issues. We're still working both internally and externally with Google to resolve this issue.
Nov 18, 11:42 PST



Sent from the Bizzle phone.

[cbizzle]

Update - We've restarted some core services to assist in getting users online, and we're simultaneously working on implementing and deploying some code changes that should improve the reconnect process for users. Additionally we're actively communicating with members of Google's SRE team while they diagnose and debug the networking problems we're seeing. Finally, we're hoping to have a full update for users within the next 30 minutes to help explain the severity and frequency of issues they've been seeing this week.
Nov 18, 12:28 PST

[cbizzle]

Monitoring - At this time we believe the majority of service has recovered for users. That said, we'd like to provide a more in-depth update on the issues users have been experiencing over the past few days.

We're currently working with Google on a priority 0 ticket for their Google Cloud Platform (which we use to bring you Discord) related to networking. Over the past day we've observed multiple major network partitions and issues on the nodes of our real time system responsible for keeping your Discord clients up to date. These networking "blips" are causing issues within various layers of our software, and many of the issues we've diagnosed will require development and testing to improve our resiliency (something we will be focusing on).

Unfortunately despite the dialog we've had with Google throughout this process, they currently haven't narrowed it down to a clear root cause. We deem the quality of service our users are getting through this process unacceptable, and have communicated this to Google's support and SRE teams. We're working around the clock to ensure Google properly diagnoses and resolves the issues we're seeing, while also monitoring and supporting our infrastructure in the hopes we can quickly catch and prevent these issues from spreading.

As always, apologies for the interruptions you've experienced and thanks for using Discord in your day to day, We hope you understand how much the performance and reliability of our service matters to us, and we hope you see improvements as we work through these issues with Google.
Nov 18, 12:56 PST