Observe transactions
Correlate request, job, archive and event IDs; separate queue latency from TSA/storage time.
Observe jobs, storage/TSA dependencies, incidents and post-recovery consistency.
Detect stalled work and preservation risks and recover without duplicate effects.
| Prepare | Receive |
|---|---|
jobId, correlationId, traceId, queue lag, TSA/storage failures and latest verification checkpoint. | Incident record, authorized recovery actions and post-recovery evidence reconciliation. |
Participants: Operations, infrastructure, security and service owners.
Correlate request, job, archive and event IDs; separate queue latency from TSA/storage time.
Distinguish renewal deadlines, trust failures, storage-access loss and backlogs; thresholds are environment-specific.
Freeze relevant changes and reconcile outbox/inbox/checkpoints; replay requires authority and preserved deduplication context.
Check data, evidence, renewal schedules and pending events; assign an accountable reopening decision.
| Current | Guard | Actor | Next | Artifact / recovery |
|---|---|---|---|---|
| Alert [model] | Classify impact and related transactions | Operations | Investigating [model] | Retain traces, jobs, config and recent changes; redact sensitive data. |
| Investigating [model] | Reconcile dependencies, outbox/inbox and checkpoints | Incident owner | Controlled recovery [model] | Replay scoped work with deduplication; no blind mass retries. |
| Recovering [model] | Check data, evidence, schedules and queues | Service owner | Approved reopening [model] | Report residual recovery gaps and accountable follow-up. |
| Situation | Action | Boundary |
|---|---|---|
| TSA outage | Separate dependency failure from invalid requests. | Waiting work must not become completed. |
| Replay after recovery | Deduplicate events and effects. | Do not recount usage. |
| Metadata or evidence loss | Use class-specific recovery plans. | Website configuration does not establish RTO/RPO. |
Commit outbox events with transactions; deduplicate through inbox processing. Replay rights are separate from deletion; alert against renewal deadlines.
After storage recovery, reconcile manifests, jobs and evidence before reopening intake. RTO/RPO apply only when approved for the deployment.
Link incidents to jobs, affected storage/TSA dependencies, recovery decisions and verification before reopening.
| API | Application scope | Documentation |
|---|---|---|
GET /v1/jobs/{jobId} | jobs:read | Job API |
| GET /v1/audit-records | audit:read | Audit and Reporting API |
| GET /v1/usage-reports | billing:read | Audit and Reporting API |
| Element | Definition |
|---|---|
| Input | jobId, correlationId, traceId, queue lag, TSA/storage failures and latest verification checkpoint. |
| Output | Incident record, authorized recovery actions and post-recovery evidence reconciliation. |
