Recover from failures
Cheetah recovery reconstructs current coordination from trustworthy sources. It cannot rewind a browser action, restore a destroyed JavaScript heap, or prove that an unobserved external effect did not happen.
Classify the interruption first
Connection loss before an attempt can leave work provably unattempted. It may follow the stable logical client to a replacement runtime if product policy still allows it.
Connection loss after sending begins can leave the outcome unknown. Cheetah does not automatically replay that attempt because the frame may have reached the client.
Runtime replacement establishes a new instance_id for the same logical client. Pending work bound to the earlier instance is fenced so an acknowledgement from the replacement cannot satisfy it.
Waiter timeout ends one caller's observation. Client work or a later retained result may still exist. Do not equate timeout with cancellation or failure.
Application-process loss destroys local futures and request stacks. Returned messages can still enter shared history. Durable product jobs reconcile after restart rather than trying to resurrect Python futures.
Missed notification does not necessarily mean missing data. Notifications wake readers; history carries accepted payloads. Query retained state before declaring loss.
Reconstruct from the right truth
After restart or failover, rebuild each layer from its owner:
- the client reports its current runtime, actions, and platform contexts;
- Redis-backed coordination supplies shared routes, correlation metadata, deduplication, and retained messages according to their persistence configuration;
- the product database supplies durable jobs and business state;
- the external target supplies the final authority for effects outside both systems.
Do not copy stale connection snapshots back into a registry. Let current clients reconnect and report their actual state.
Decide whether another attempt is safe
Use this order:
- Search retained history for a terminal result or error from the original command.
- Inspect the target system using a stable product idempotency or operation key.
- Determine whether the original attempt was fenced before sending or became uncertain after sending started.
- Retry only if the work is idempotent or the product can safely reconcile duplicates.
- Use a new command ID while retaining the relationship to the original product operation.
For destructive or externally visible actions, build reconciliation into the workflow before an incident. Human guesswork after a timeout is not a reliability strategy.
Handle client platform lifecycles
Browser extension service workers can stop while tabs remain open. On restart, the extension must reconstruct providers, handlers, policy, and current context reporting before becoming a usable runtime. Tab or content-script state may have changed independently.
Console and custom clients have different persistence opportunities but the same identity rule: stable client identity is not process memory. Report current truth after restart and do not claim previous leases or actions without evidence.
Use an incident record that joins the layers
Capture the effective tenant/user, logical client, runtime instance, command ID, trace ID, action, target context, dispatch owner, observation deadline, terminal evidence, and product operation key. Record which facts are known, absent, or uncertain.
This prevents a later result from being mistaken for a duplicate incident and makes it possible to decide whether the product needs reconciliation, compensation, or no further action.