Testing recovery assumptions after GitLab’s database outage

Published by GitLab

This credits the original publisher. Better Loop membership or a shared assessment is not implied.

The public work

GitLab’s postmortem reconstructs its January 2017 production-database outage, failed backup paths, and recovery from an earlier snapshot. It documents unrecoverable changes and traces follow-up work to monitoring, recovery testing, runbooks and operational ownership.

What to notice

A backup job is not proof of recoverability. Validate restoration, alert delivery and ownership, and distinguish failover mechanisms from disaster-recovery paths.

Keep the context

Operator-authored retrospective of one historical incident. Some loss estimates are approximate; proposed improvements are not all verified as completed in this report. No recovery procedure was executed here.

AI use: Not reported in the source.

The incident report does not document AI-assistant use by its authors.

A useful public example is not an assessment of a reader, a publisher or a Better Loop member.

Authored practice suggestion

Try the idea.
Check your own work.

Use material you are allowed to work with. This suggestion is preparation; it does not record a completed task or an improvement.

A prompt to adapt privately

Using only a local copy of this public postmortem, connect each failed recovery path to its documented cause and proposed correction. Draft a recovery-readiness review checklist with owners, observable restore checks and stop conditions. Keep confirmed recovery actions separate from planned improvements.

A check to adapt

Each correction traces to a reported failure; restoration, alerting and ownership have explicit checks; retrospective facts and proposed work are labeled separately.

Open a related practice brief

Private practice needs no account. Your configured model provider may process the evidence you select.

See the full skill-to-sharing flow

Another way into the work.

Teaching plots with Gapminder data

Software Carpentry’s lesson reads Gapminder country data and displays labeled GDP time-series and scatter plots. It shows table transposition, legends and file export, alongside learner exercises and accessibility guidance.

Published bySoftware Carpentry / The Carpentries

Curated public exampleReport

Operations & education

Coordinating GitHub’s database incident recovery

GitHub’s report traces an October 2018 network partition through database failover, inconsistent replicas, recovery and delayed background processing. The timeline explains why responders prioritized data integrity and why early recovery estimates proved unreliable.

Published byJason Warner / GitHub

Curated public exampleReport

Code & software

Separate a code solution from its edit format

Aider published code-editing benchmark results for an architect model that describes a solution and an editor model that converts it into file changes. The report compares paired configurations with solo baselines and discusses latency tradeoffs.

Published byAider