jagreehal avatar

resilience

Add retry, timeout, and circuit breaker patterns at the workflow level. Business functions stay clea

提供方 jagreehal|开源

Resilience Patterns

Critical rules

  • Resilience is composition: step.retry / step.withTimeout at the workflow — never inside business functions.
  • Retry at exactly one layer. Never double-retry (3×3×3 storms).
  • Retry only transient errors. Never retry non-idempotent writes without an idempotency key.
  • Always set per-attempt timeouts and enable jitter in production.
  • Multi-step retries require every step to be idempotent. Use circuit breakers for sticky-down externals.
  • Before configuring policies, read references/patterns.md and references/defaults.md.

Workflow

  1. Keep fn(args, deps) returning Results with no retry loops.
  2. Choose defaults from references/defaults.md for the operation type.
  3. Wrap with step.retry + per-attempt timeout + jitter + retryOn for transient codes only.
  4. For writes, require an idempotency key or do not retry.
  5. Detect timeouts with isStepTimeoutError / getStepTimeoutMeta and log.
  6. Add circuit breakers (opossum/cockatiel) for externals that stay down.

Resources

Validation

  • Retry only at workflow level; single layer
  • retryOn matches only transient errors
  • Non-idempotent writes gated by idempotency key or not retried
  • Per-attempt timeout; jitter + backoff in production
  • Timeout errors detected and logged
  • Underlying function tests unchanged

Constraints

  • Do not put retry inside business functions or stack retries across API/service/client for the same call.
  • Related: result-types, fn-args-deps, api-design, observability.