Observe and diagnose
No single signal explains a distributed command. Use current connection state, pending correlation, retained messages, traces, client telemetry, and the product's external evidence together. Each source answers a different question.
Mount diagnostics from the right state source
For a one-process development host, create_dev_inspectors(components) wraps the actual live component set. It cannot inspect in-memory objects in another process.
For a distributed deployment, create_redis_inspectors(redis_client) reads supported shared registry, history, RPC, deduplication, context, and browser-window state. Supply compatible spill storage when operators must hydrate externalized history payloads.
Mount the router behind administrative authentication:
from servercheetah.admin_auth_helpers import resolve_admin_auth
from servercheetah.api.diagnostics import create_diagnostics_router
admin_auth = resolve_admin_auth(bind_host="0.0.0.0", strict=True)
app.include_router(
create_diagnostics_router(
inspectors,
admin_auth=admin_auth,
enable_raw_messages=False,
),
prefix="/api/diag",
)Call /capabilities before relying on a view. A 501 means the composition lacks the required inspector; 404 means the requested record is absent; 403 means administrative access was denied.
Bounded user, user-summary, and action observations include a completeness block. Read it before treating an observed count as exhaustive. Delivery inspection is a separate capability covering admitted work, node listener and reconciliation health, capacity reservations, and payload-free command status; connection or pending-RPC counts are not substitutes for it.
Investigate in journey order
- Find the logical client and current instance. Check presence, owning node, heartbeat, client type, and advertised actions.
- Find the attempt. Confirm the expected runtime instance and socket owner. A replacement instance cannot acknowledge an older bound attempt.
- Inspect delivery health. Check admitted and due work, oldest sampled age, dispatcher listener and reconciliation health, reservations, and the command's payload-free status.
- Inspect pending correlation. A pending RPC means the waiter has not resolved, not that the client never acted.
- Inspect retained history. Look for accepted progress, result, error, event, or state by command and trace identity.
- Follow the trace. Determine where the journey stopped or slowed; do not treat sampled tracing as the retained message record.
- Inspect the external target. For uncertain side effects, only the target system or a product reconciliation record may establish what actually happened.
Redis-backed diagnostic pagination uses opaque cursors. A scan page can be empty while has_more remains true; continue until the response says the scan is finished.
Connect traces without confusing their authority
Server roles emit lifecycle events through ITracer. Client runtimes emit structured events through ITelemetrySink. A no-op implementation is valid, and a client configuration may use a local console sink. OpenTelemetryTracer(exporter=None) is silent rather than an implicit console exporter. The application supplies explicit exporters, a protected HTTP receiver for client telemetry, sampling, redaction, and backend retention. The host that creates an OpenTelemetry tracer shuts it down after all component sets using it have stopped.
A trace explains timing and technical parentage. History records accepted application messages. Product storage records domain truth. Sharing a trace ID makes the views easier to join but does not make them interchangeable.
Alert on meaningful conditions
Useful operational signals include:
- connection churn and unexpected runtime replacement;
- dispatch backlog and resource-quota rejection;
- old pending RPCs, grouped by owning application node;
- ingestion rejection by authentication, size, validation, stale instance, or duplicate;
- Redis and payload-store latency or unavailability;
- history growth, spill failures, and retention pressure;
- worker assignments that outlive their lease or client presence;
- error-code changes after a client or server rollout.
Keep sensitive parameters and returned payloads out of ordinary logs. Prefer stable command, trace, client, tenant, and error identities with controlled access to the full retained evidence.