How do you audit and evaluate human overrides of automated memory-compaction remediation?

Auditing and evaluating human overrides of automated memory-compaction remediation means proving who intervened, why the override was necessary, what policy and workflows it affected, whether it improved safety and service reliability, and whether control was returned to automation correctly. An immutable action log answers *what happened*; evaluation examines *whether it was justified and effective*.

The audit must link operator decisions to the exact scheduling-policy revision, compaction jobs, validated memory versions, and downstream agent outcomes—without copying sensitive documents into logs.

Beginner-friendly production incident

A LangGraph API-test-generation system uses requirements, test-generation, security-review, and publisher agents. Celery workers compact context. An automated remediator repeatedly lowers admission when the compaction queue grows, starving background jobs.

An on-call engineer overrides the controller and reserves capacity for critical security reviews. An incident report later claims that the override fixed latency. But the queue improved at the same time an upstream LLM provider recovered from throttling.

Without a structured audit, the team cannot separate the operator's action, provider recovery, and subsequent workflow outcomes. A good evaluation captures the intervention timeline, compares like-for-like workloads, and explicitly reports uncertainty.

Capture evidence → Validate authority → Reconstruct timeline → Measure effects → Review safety → Identify root cause and improvement → Verify closure

Step-by-step numerical example

Suppose an operator activates a tenant-scoped override at 14:00 because critical compaction handoffs are missing a 30-second deadline.

1. Before the override: 18 of 300 eligible handoffs miss the deadline: 18/300 = 6%.
2. During the override: 6 of 300 comparable handoffs miss: 6/300 = 2%.
3. The observed change is 4 percentage points and a relative miss-rate reduction of (6% − 2%)/6% ≈ 66.7%.
4. Suppose 4 background compactions were delayed beyond an agreed maintenance threshold during the override; audit those consequences too. A critical-latency improvement can mask background starvation.
5. A provider-status trace shows a major quota restriction ended five minutes after the intervention. Therefore, the 66.7% *observed association* is not reliable evidence that the human override alone caused the improvement.
6. After a staged handback, 3 of 300 further eligible handoffs miss (1%) and no verified invalid-summary consumption occurs.
7. Record the measurements, cohort definitions, provider timeline, remaining uncertainty, and an action item to test the override using controlled traffic replay.

This example uses small illustrative cohorts. Confidence intervals, workload differences, and sparse serious errors must be considered; zero observed security incidents never proves zero underlying risk.

What belongs in an override audit record?

1. Identity and authorization: Operator identity or scoped service principal, authenticated role, approval ID, and any break-glass justification.
2. Incident context: Incident ID, severity, declared scope, affected tenant/workflows, detection signals, and incident owner.
3. Decision evidence: SLO window and denominator, queue age, rate-limit metrics, compaction validation failures, relevant traces, and competing hypotheses.
4. Before/after policy: Requested change, actual configuration, expected revision, committed new revision, effective time, and propagation acknowledgment.
5. Scope and duration: Tenants, regions, queue groups, affected agent roles, expiry time, review deadlines, and extensions.
6. In-flight work: Celery task IDs and logical compaction job IDs, source watermarks, reserved/executing jobs, and any reconciliation outcomes.
7. Memory provenance: Summary candidate and active-version identifiers, validation status, source-event revision, and authorized event-tail coverage.
8. Outcome: Critical deadline misses, accepted-workflow quality, costs, background starvation, safety violations, and downstream artifact repairs.
9. Handback: Who authorized restoration, validated readiness conditions, policy epoch, canary cohort, and post-restoration observations.
10. Review decisions: Whether the override was appropriate, accidental, too broad, too late, or prolonged; the concrete follow-up actions.

Keep sensitive prompt contents out of general logs. Use cryptographic integrity protections, appropriate retention, and role-based access; an append-only table alone does not guarantee immutability or tamper evidence.

Example audit event

JSON
{
  "event_id": "audit-981",
  "incident_id": "INC-42",
  "override_id": "OVR-17",
  "actor_role": "incident-responder",
  "scope": {"tenant_id": "tenant-7"},
  "action": "PAUSE_OPTIONAL_COMPACTION",
  "reason": "CRITICAL_HANDOFF_DEADLINE_MISSES",
  "previous_policy_revision": 50,
  "new_policy_revision": 51,
  "approval_ref": "approval-42",
  "result": "APPLIED",
  "evidence_refs": ["trace-501", "slo-window-82"]
}

This illustrative record omits the actual timestamp and authenticated actor ID for brevity. A real audit event requires both, plus a reliable source clock and any necessary correlation identifiers. Never treat an arbitrary actor-role string as proof of authorization.

How do you evaluate override quality?

Necessity: Was a verified SLO, safety, quota, or policy-control condition present, or was the override based on one noisy metric?

Proportionality: Did it affect only the necessary tenant, queue, or policy version? A tenant-specific compaction problem usually does not justify turning off global memory validation.

Timeliness: How long passed from the first actionable alert to intervention, policy propagation, and confirmed reduction in downstream harm?

Correctness: Did every consumed summary remain validated, authorized, and complete with required uncompacted events? Were pending OAuth or publication approvals preserved?

Effectiveness: Did critical-agent deadline misses, queue age, or accepted-workflow cost improve compared with matched or controlled baselines?

Side effects: Did background jobs starve, model-provider retries increase, other tenants lose capacity, or valid source revisions become inaccessible?

Reversibility: Could the operator restore the prior policy without deleting queued jobs or corrupting LangGraph checkpoints? Did stale remediation tasks remain fenced?

Handback quality: Was the override released after explicit readiness checks and a monitored canary, rather than simply expiring into an unsafe configuration?

Avoid a one-number “operator score.” Review the actual context and uncertainty; retrospective metrics can reward risky shortcuts if quality is excluded.

Reconstruct the override timeline

Correlate these events by incident and workflow ID:

• Alert produced and acknowledged.
• Operator obtained authorized incident ownership.
• Override requested, approved, and committed with a policy revision.
• Celery admissions changed; reserved and active tasks continued or were reconciled.
• LangGraph handoffs consumed validated summaries with known event watermarks.
• SLO measurements and provider-capacity metrics changed.
• Handback prerequisites passed, canary began, and automation resumed.
• Incident closed after the final verification window.

Use the event's occurrence time and recorded/ingested time where possible: asynchronous telemetry can arrive out of order. Correlation and policy revisions are more reliable than assuming timestamps across separate systems are perfectly synchronized.

Distinguish operator action from correlated improvements

Suppose queue wait falls after the override. Consider alternative explanations:

• Provider request/token quota recovered.
• A large tenant's ingestion burst ended.
• Celery workers autoscaled or restarted.
• Duplicate compaction jobs were independently coalesced.
• A different scheduler policy version was deployed.
• Cached summaries became available before the operator action propagated.

Compare equivalent workload slices and, where safe, use a control cohort or a deterministic traffic replay. Do not experiment with a safety-sensitive override merely to obtain causal proof during a live incident. Record that attribution is uncertain when evidence is incomplete.

Human intervention security and governance

• Authorize server-side: Distinguish read-only monitoring, incident-response, and global-policy administration.
• Enforce policy precedence: Non-bypassable context-integrity checks outrank human overrides. A normal emergency button cannot waive a required security approval.
• Use version fencing: Conditional writes prevent stale automated remediation or delayed operator commands from replacing the active override.
• Require documented reasons: A scoped reason code, incident link, and approval trail reduce undocumented changes.
• Use time-bounded access: Break-glass credentials and exceptions should have review and expiry behavior appropriate to their risk.
• Protect audit evidence: Control access to actor identities, tenant IDs, and potentially sensitive source references.
• Review repeated overrides: Frequent interventions may indicate poor SLO definitions, a noisy controller, insufficient quota, or missing automation—not merely operator error.

LangGraph and Celery integration

In LangGraph, link interrupted or resumed workflow thread/checkpoint IDs to incident and override identifiers. A human's scheduling-policy approval is separate from approving an agent's final artifact. On resumption, recheck source revisions, required approvals, and model-facing context. LangGraph interrupts enable human input with checkpointed state; node code around resumption must be designed for replay-safe side effects.

In Celery, correlate logical compaction job IDs with individual task attempts and event states. Track enqueued, reserved, executing, retried, and succeeded jobs before and after the override. An override may change admission without canceling in-flight calls. Repeated delivery must not promote stale candidates; log the version-checked activation result separately from task success.

Example post-incident review questions

1. Which evidence justified the override, and was the incident scoped correctly?
2. Who authorized and performed the change? Did it meet separation-of-duties requirements?
3. Did the policy actually change, and did workers respect the new policy revision?
4. Which workflows, source revisions, and memory summaries were affected?
5. Did accepted-workflow quality and critical authorization tests remain intact?
6. Was the latency improvement attributable to the intervention or other changes?
7. Were there unintended fairness, spending, or reliability consequences?
8. Did any stale command or duplicate Celery task run after override activation?
9. Did the handback satisfy readiness and safety requirements?
10. What benchmark, alert, control-plane, or runbook improvements will prevent recurrence?

Useful audit and evaluation metrics

Track time to authorized intervention, time to effective policy propagation, time to stable SLO recovery, number and duration of overrides, scope size, unauthorized-change attempts, stale-policy-write rejections, proportion with complete evidence, background-starvation events, policy reversals, operator handback errors, and repeat incidents.

An audit record can prove an action was attempted and committed; it does not automatically prove downstream recovery. Pair policy events with agent execution outcomes and source-integrity checks.

Difference from earlier lessons

• Q105: Auditing general tool calls.
• Q153–154: Human-in-the-loop approvals for agent workflows.
• Q233: Provenance and audit of compacted summary *versions*.
• Q249: Designing safe human intervention and override controls.
• Q250: Returning remediation authority to automation safely.
• This lesson: Retrospectively auditing and evaluating each override—its authorization, blast radius, timeline, causal evidence, business impact, safety, and lessons learned.

In short

An effective human override is not just one that changes a policy flag. It is authorized, scoped, replay-safe, measurable, auditable, and followed by a verified handback. Review both its benefits and harms so the next incident can be handled with better evidence.

Further reading

• LangGraph: Interrupts and human-in-the-loop
• LangGraph: Persistence
• Celery: Monitoring and Management
• Google SRE: Postmortem Culture