Last updated on

Agent Reliability Engineering

Markdown source: View raw markdown

Night-time failures happen when observability, guardrails, and repair paths are unclear. Drawing on Agent Reliability Engineering, AI Reliability Engineering, AI SRE practices, and the Google SRE workbook, this page distills a pragmatic setup that keeps agents online.

The goal: tight feedback loops, explicit health checks, and fast repair steps that on-call can run without guesswork.

Core reliability stack

  • Telemetry and replay: Instrument traces and events to Langfuse or LangSmith so every failure has context and can be replayed.
  • Channel gateway: Use OpenClaw to front WhatsApp/WebChat channels, enforce auth, and expose health.
  • Runbooks: Keep these probes and repair commands close to the bot so handoffs are fast.

Health checks (no guessing)

Intent: confirm channel connectivity and agent reachability without poking production users.

  • openclaw status — local summary: gateway reachability/mode, update hint, linked channel auth age, sessions, and recent activity.
  • openclaw status --all — full local diagnosis (read-only, color, safe to paste for debugging).
  • openclaw status --deep — also probes the running Gateway (per-channel probes when supported).
  • openclaw health --json — asks the running Gateway for a full health snapshot (WS-only; no direct Baileys socket).
  • Send /status as a standalone message in WhatsApp/WebChat to get a status reply without invoking the agent.
  • Logs: tail /tmp/openclaw/openclaw-*.log | grep -E "web-heartbeat|web-reconnect|web-auto-reply|web-inbound".

Doctor (repair and migration)

  • What it does: repairs stale config/state, checks health, and provides actionable fixes for OpenClaw.
  • Quick start: run openclaw doctor.
  • Headless / automation:
    • openclaw doctor --yes — accept defaults without prompting (including restart/service/sandbox repair when applicable).
    • openclaw doctor --repair — apply repair steps automatically.

When incidents happen

  • Capture trace IDs from Langfuse/LangSmith and correlate with OpenClaw logs before retrying traffic.
  • Prefer replaying a failing trace through the gateway rather than mutating prompts live.
  • If a channel is degraded, publish a status message via /status and keep openclaw status --deep running to watch recovery.