Bound capacity and retain evidence
A distributed application accumulates several kinds of state, and each one has a different job and lifetime. Capacity planning fails when all of them are treated as “the queue” or “the database.” Choose limits and retention from the question each record must answer, then verify that every process and backing store applies a compatible policy.
Separate the stores before choosing limits
| State | Purpose | Important limit |
|---|---|---|
| current connection and topology registries | describe clients, runtime instances, contexts, and ownership now | current coordination can expire or be reconstructed; it is not business history |
| dispatcher mailbox and delivery status | retain admitted logical-client work and bounded delivery evidence | it is not a durable product job queue and does not establish external completion |
| RPC correlation and response notification | wake the application process waiting for a terminal observation | futures are process-local; notification is a signal, not the retained payload |
| history | retain accepted progress, result, error, event, telemetry, and state messages | it is not a complete outbound-command or compliance ledger |
| payload and spill storage | keep supported large values outside a command or Redis Stream entry | access, expiry, cleanup, backup, and multi-host reachability are separate responsibilities |
| derived views | save a rebuildable projection of retained messages | the host owns scheduling, reducer cost, migration, and deletion |
| product records | own durable jobs, business state, idempotency, and reconciliation | Cheetah does not create or retire them automatically |
| traces and audit records | explain technical timing or compliance-relevant activity | tracing may be sampled; the supplied audit interface is not wired into ordinary execution |
History and product state often refer to the same command, but they are not interchangeable. History can show what evidence reached Cheetah. The product database decides whether an invoice, workflow, or external operation is complete.
Choose history retention deliberately
The in-memory preset keeps a bounded list per stream and loses it when the process ends. Its current default is 1,000 messages per stream. This is useful for local development and focused examples, not for a recovery or compliance promise.
The Redis and production presets use an IHistoryRetentionPolicy. Their current static defaults are:
| Setting | Default | Meaning |
|---|---|---|
| messages per stream | 100 | older entries are trimmed as newer accepted messages arrive |
| inactive stream TTL | 900 seconds | refreshed on append; an inactive stream eventually expires |
| command-index TTL | 900 seconds | bounds lookup keys used by command ID |
| spill threshold | disabled | large returned payloads remain inline unless spill is explicitly configured |
| managed spill lifetime | 86,400 seconds | immutable expiry assigned to a newly spilled payload when managed spill is enabled |
An application can supply another policy, but the current policy interface resolves by trusted user and applies one decision to that user's streams. It does not yet receive a stream category. Keep business plans and entitlements behind the policy implementation rather than teaching history storage about commercial tiers.
TTL and length answer different questions. A busy stream can lose old messages through trimming long before its TTL. An inactive stream can disappear while it still contains fewer than the maximum. Cursor-based reading is the portable incremental contract; timestamp filtering differs between the in-memory and Redis implementations.
Keep external payload lifetime aligned
Redis history spill moves a supported top-level payload out of the stream entry. It does not make the message durable forever or confidential from the history infrastructure. Every role that may hydrate the message needs compatible access to the spill store.
Managed spill gives the reference and external object one assigned expiry. The built-in file store hides expired objects, performs opportunistic cleanup during continued writes, and exposes an explicit cleanup hook. An idle directory does not clean itself merely because time passed. If a custom store cannot guarantee eventual expiry, composition fails closed unless the host deliberately accepts unmanaged retention.
Explicit history deletion removes selected message identities and attempts related spill cleanup. It does not erase traces, derived projections, product records, exports, or downstream copies. Tenant offboarding and legal deletion therefore need an inventory of every participating store.
Bound work before it consumes shared resources
Use limits at the earliest boundary that can enforce them truthfully:
- the proxy, HTTP framework, and REST role should agree on request size; the REST role defaults to a 1 MiB ingestion limit;
- the App role has no WebSocket command-size limit unless the host selects one, so configure it when client or proxy limits require a predictable boundary;
- pre-admission reconnect waiting is bounded process-wide and by tenant in shared mode, or by effective user in ordinary mode;
- Redis delivery uses fail-closed reservations for a per-scope backlog, currently defaulting to 10,000 admitted or in-flight commands;
- worker placement has separate resource and negotiated-capacity rules; a general server quota does not substitute for platform capacity;
- product APIs and external systems still need request frequency, account, cost, and downstream concurrency controls appropriate to their threat model.
The supplied IRateLimiter, audit logger, and the connection, pending-RPC, and storage methods on IResourceQuota are not automatically invoked by ordinary node execution. The active Redis quota integration is the delivery-backlog reservation. Do not configure an unused implementation and report that the corresponding protection exists. Enforce pre-authentication traffic at the outer edge and add product-aware controls where the framework has no current integration point.
Treat audit as a separate requirement
Tracing helps explain timing and causality and may be sampled. History retains selected application messages. Diagnostics expose current and retained framework state. None is a compliance-grade record of authentication, administrative access, command intent, policy decisions, and product effects.
If the product needs an audit trail, define its required events, principal and actor fields, ordering, durability, redaction, availability behavior, retention, export, and access policy. Wire that trail at the host and product boundaries that actually observe the events. Do not label ordinary logs or the currently unwired IAuditLogger interface as an active audit facility.
Verify the operating policy
Exercise a realistic stream until trimming occurs, let an inactive stream expire, retrieve and then expire a spilled payload, and test explicit deletion across every relevant store. Fill delivery and wait capacity until admission fails with the documented reason, then verify that reservations and waiters drain after success, timeout, disconnect, and shutdown.
Monitor history growth, spill failures, cleanup lag, oldest admitted delivery, capacity rejection, Redis memory, and product-job backlog separately. A single “queue depth” graph cannot explain these different lifetimes.
Observe and diagnose the deployed system