LAB429/ Cheetah product page ↗

Cheetah / Cheetah documentation

Terminal results and delivery status

Cheetah exposes terminal correlation and delivery evidence separately because they answer different questions. A terminal RPC says that an accepted result or error was correlated to the command. Delivery status says how far the command's outbound attempt can be proved.

Send and wait for one terminal outcome

resolution = await scoped.send_command_and_wait(
    client_id="primary-browser",
    command={"name": "get_page_info", "params": {}},
    options=SendOptions(timeout_ms=30_000),
)

This method copies the options with mode=ResponseMode.rpc, admits the command, and awaits its existing correlation registration. Progress messages do not resolve it. A terminal result returns lightweight RpcResolution; a terminal error raises CommandError.

RpcResolution contains the command ID, success flag, terminal kind, optional error code, history stream key, and trace linkage. It intentionally does not contain the complete handler payload. Full returned messages live in history so that cross-process notification remains small and payload storage follows one retention path.

When cache-aware parser delivery performs its one verified inline retry, the resolution's command_id identifies the retry. Always use the ID from the returned resolution when reading the corresponding terminal payload.

Await an RPC that was registered earlier

send_command() can be called with mode=ResponseMode.rpc and awaited separately:

command_id = await scoped.send_command(
    "primary-browser",
    command,
    SendOptions(mode=ResponseMode.rpc, timeout_ms=30_000),
)

resolution = await scoped.await_result(command_id, timeout_ms=2_000)

await_result() does not create a registration. If no pending RPC exists, including after a registration has already been resolved and removed, it raises ValueError.

The optional timeout_ms on await_result() limits only that local call. None waits until the correlator-owned registration reaches its own result or deadline. Zero performs an immediate check. A local deadline raises RpcAwaitTimeoutError without cancelling the shared future, another waiter, the remote action, or a later history result. If completion races the exact local deadline, the completed result wins.

The registered observation deadline is different. When that deadline expires, the correlator raises RpcTimeoutError and removes its pending work. Neither kind of timeout proves that the remote handler stopped or that a late result cannot be retained.

Runtime binding and restart fencing

An admitted RPC starts unbound while the command waits for its first send attempt. Immediately before that attempt, the dispatcher binds the actual runtime instance_id; the execution observation timeout begins at that point.

If the logical client reconnects through a replacement runtime before any attempt, the unbound command can bind that replacement. If a bound runtime is replaced, its pending RPCs fail with InstanceRestartedError, and a late response from the old runtime cannot resolve a new or differently bound registration. This distinguishes safe movement of unattempted work from unsafe replay after an ambiguous attempt.

Inspect delivery evidence

status = await scoped.get_delivery_status(
    client_id="primary-browser",
    command_id=command_id,
)

The result is DeliveryStatus | None. None means the selected dispatcher has no retained status for that routing and command; it is not proof that the command never existed.

StateStable resolutionWhat is established
queuednone yetdispatcher accepted responsibility; no send attempt is yet recorded
send_startednone yetthe first socket attempt began and the runtime was bound
transport_acceptedsent_unconfirmedthe socket transport accepted the write; client receipt is unproved
client_accepteddelivered_ackthe bound runtime acknowledged a valid command frame
uncertainuncertainsending began, but the transport outcome cannot be proved
expired_before_attemptexpired_before_attemptthe delivery budget ended before a send began
target_unavailabletarget_unavailablethe logical client became unavailable before an attempt

The status can also carry delivery sequence and relevant queued, send-started, resolved, and client-accepted timestamps. Status is payload-free and scoped by the trusted routing. Its retention is bounded by the dispatcher implementation.

A client ACK means that the authenticated, bound runtime accepted the command into its dispatch path. It does not mean local permission was granted, the handler began, or an external side effect completed. Those outcomes return later as result or error evidence, and some real-world effects still require application reconciliation.

Retry decisions

ObservationAutomatic retry posture
admission rejected before durable handoffsafe only after considering why the request was rejected
expired_before_attempt or target_unavailableno send attempt occurred according to retained delivery evidence
client_accepted plus a terminal errordelivery happened; retry depends on action semantics and the error
transport_accepted, uncertain, or local/RPC timeoutdo not infer non-execution; inspect history and application state
DispatchAdmissionUnknownErrorexplicitly unsafe to retry automatically

Idempotency and target-system reconciliation remain product concerns. Cheetah preserves honest evidence boundaries; it cannot turn an arbitrary browser or machine side effect into a distributed transaction.

Continue to Response waiting and history to read the full retained result and progress messages.