Runbooks

Let the on-call rotation break the runbook before 3 AM does.

The people who carry the pager comment on the live runbook. Agent connect makes the cluster context explicit.

Set it up for your agentHow agent connect works
liverunbooks/ingest-queue-backlog.md

Ingest queue backlog

Owner: ingest on-call. Service: telemetry-ingest. Severity: page.

When robot telemetry lags more than ten minutes, page the ingest on-call and work this file. Do not restart the broker from here. Poses are not replayable from the queue.

When this page fires

  • telemetry.ingest.lag_seconds stays above 600 for five minutes.
  • Fleet console pose updates go stale.
  • ingest-consumer CPU stays under 20% while the queue depth climbs.

Steps

  1. Open the lag board.https://grafana.saltmarsh.dev/d/lag-stage
  2. Confirm the consumer is deploy/ingest-consumer in namespace telemetry. A hot consumer belongs in ingest-consumer-hot.md, not this file.
  3. If the annotation ship.saltmarsh.dev/release landed in the last hour, roll that release back before you scale.
  4. Scale the consumer.kubectl scale deploy/ingest-consumer -n telemetry --replicas=6kubectl --context prod-ingest scale deploy/ingest-consumer -n telemetry --replicas=6
    Confirm kubectl config current-context prints prod-ingest before you scale.
  5. Watch lag for five minutes. If it is still climbing, page #ingest-platform. Do not flush the queue.

Do not

  • Do not scale past 12 replicas. The pose writer saturates first.
  • Do not delete the consumer pod to unstick it. That drops the in-flight batch.

grafana.saltmarsh.dev/d/lag-stage

Hannah Kowalski

This is the staging board. On-call will open lag-stage and think the queue is fine.

Ravi Menon

Production is /d/lag-prod on grafana.saltmarsh.internal.

kubectl scale deploy/ingest-consumer

Ravi Menon

@agent step 4 scales with whatever context the shell has. Name prod-ingest in the command.

agent

Updated step 4 to pass --context prod-ingest, and added a current-context check before the scale.

resolved
Start with @agent to ask the agent
ResolveReply
Fictional example: a Saltmarsh Robotics ingest runbook in Gander, with a service-owner thread and a scripted agent connect reply.

How runbooks get reviewed today

A wiki page is a fine home for a stable runbook. It is a poor home for a draft the agent is still aligning with the cluster you actually run.

Review in chat loses the step number. Review in a PR assumes the runbook is ready to commit. The on-call engineer who will use it at 3 AM may not be the person approving the PR.

Break the ingest runbook on purpose

Owen Briggs, SRE at Saltmarsh Robotics, asks his agent for a runbook for when the robot telemetry queue backs up. He runs gander watch on runbooks/ingest-queue-backlog.md and sends it to Ravi Menon and Hannah Kowalski.

Hannah highlights the lag board link and says it is the staging board. Ravi replies with the production board path. That thread stays between them.

Ravi starts another comment on the scale step: "@agent step 4 scales with whatever context the shell has. Name prod-ingest in the command." Owen's agent names the prod-ingest context in the command, replies, and resolves the thread. Hannah sees the change on the same link. The runbook goes into the on-call repo when the rotation signs off.

Who's in the doc

WriterOn-call engineerService ownerThe author's agent

Writer SRE. Sends the link. The rotation does not install the agent.

On-call engineer Tries to follow the steps without tribal knowledge.

Service owner Checks that the steps match the cluster they run.

The author's agent Can name the context it assumed, when on-call uses agent connect.

Reviewers only need a browser. Talk to each other in normal threads. Use agent connect when you want the draft changed.

Agents that fit runbooks

Claude CodeThe runbook can live beside the service. MCP returns agent connect to the session that wrote it.
CodexCodex drafts the steps from the repo. Agent connect comes back through MCP.
OpenCodeA terminal agent is a natural place to draft operational steps and hear agent connect.
Grok BuildA scheduled job can refresh runbooks in a folder the rotation already watches.

Let the rotation find the missing context in review, on the file the agent can still fix.

See how it works on the homepage