Coordinating GitHub’s database incident recovery

Published by Jason Warner / GitHub

This credits the original publisher. Better Loop membership or a shared assessment is not implied.

The public work

GitHub’s report traces an October 2018 network partition through database failover, inconsistent replicas, recovery and delayed background processing. The timeline explains why responders prioritized data integrity and why early recovery estimates proved unreliable.

What to notice

Validate failover against application assumptions; coordinate recovery and backlog processing around data consistency, and qualify estimates when workload changes undermine extrapolation.

Keep the context

Operator-authored historical report; some reconciliation was still underway when published. Its action plans are not proof of completed remediation or current GitHub architecture. No system changes were made here.

AI use: Not reported in the source.

The incident report does not document AI-assistant use by its authors.

A useful public example is not an assessment of a reader, a publisher or a Better Loop member.

Authored practice suggestion

Try the idea.
Check your own work.

Use material you are allowed to work with. This suggestion is preparation; it does not record a completed task or an improvement.

A prompt to adapt privately

From a local copy of the incident report, reconstruct a dependency-ordered recovery timeline. Identify the evidence behind pausing writes, rebuilding replicas and resuming queued work. Produce a short incident-update template that separates known state, uncertain estimates and the next verification step.

A check to adapt

Timeline ordering matches the report; availability and data integrity are distinguished; recovery estimates state their assumptions and are updated only with supporting evidence.

Open a related practice brief

Private practice needs no account. Your configured model provider may process the evidence you select.

See the full skill-to-sharing flow

Another way into the work.

Teaching plots with Gapminder data

Software Carpentry’s lesson reads Gapminder country data and displays labeled GDP time-series and scatter plots. It shows table transposition, legends and file export, alongside learner exercises and accessibility guidance.

Published bySoftware Carpentry / The Carpentries

Curated public exampleReport

Operations & education

Testing recovery assumptions after GitLab’s database outage

GitLab’s postmortem reconstructs its January 2017 production-database outage, failed backup paths, and recovery from an earlier snapshot. It documents unrecoverable changes and traces follow-up work to monitoring, recovery testing, runbooks and operational ownership.

Published byGitLab