How do you prevent policy-version split brain when replacing automated memory-compaction controllers across distributed AI agents?

Policy-version split brain occurs when two generations of a distributed AI-agent remediation controller both believe they may change the same workflow or memory state. During a policy replacement, a retired Celery worker may still run an old task while a new LangGraph-driven controller starts acting. A configuration toggle or queue revocation alone cannot guarantee that stale actions stop. The crucial invariant is only the currently authorized controller generation can commit a memory mutation.

What makes this different from Question 256?

Question 256 decides *whether* to retire or replace a remediation policy and outlines governance and rollout. This lesson explains the concurrency-control protocol that prevents stale workers from applying obsolete decisions while generations overlap. This is not a shared-memory schema migration (Questions 220–224), nor merely duplicate-task prevention.

Production incident with numbers

An AI support system runs 10,000 agent workflows; 600 queued memory-compaction jobs were issued under policy version 7. Operations activate version 8 after discovering that version 7 sometimes drops required handoff facts. Of the 600 old jobs, 80 are already executing, 120 are reserved, and 400 are waiting. Revoking visible task IDs is not sufficient to stop already executing jobs. Suppose 30 version-7 jobs resume after version 8 has taken control; if each blindly commits, up to 30 workflows could receive outdated compaction decisions.

Assign each affected workflow a monotonically increasing fencing generation. A version-7 job carries generation 41, while the new authorized generation is 42. On every attempted write, the transactional storage layer checks both expected_generation = current_generation and the underlying memory revision. All 30 stale jobs with 41 are rejected; the new controller commits only after checking that the canonical memory snapshot is still valid. This numerical example is illustrative, not a measured production result.

A safe replacement protocol

1. Define a scoped authority key: for example tenant_id + workflow_id + memory_stream. Global policy versions alone do not prove exclusive ownership of individual workflows.
2. Version the policy and its authority separately: immutable policy_version identifies business logic; monotonic fence_generation identifies the current writer for a scope. Preserve a signed or access-controlled policy registry with clear operator ownership.
3. Switch authority atomically: in one durable transaction, increment the fence for the scope and record the new policy/version owner. A lease may help liveness, but lease expiry by itself cannot prevent a paused worker from committing late.
4. Enforce fencing at the side-effect boundary: do an atomic compare-and-swap or conditional update against the current fence and memory revision. A check performed only when the task starts is unsafe. For an external vector store or API lacking conditional writes, use a serialized write gateway or another authoritative guard before changing state.
5. Make attempts idempotent: use a stable operation ID for each logical mutation and persist the outcome; a Celery retry or LangGraph replay must not double-compact or duplicate audit events.
6. Drain and reconcile: classify pending, reserved, running, failed and retried jobs; fence stale generations and reconcile checkpoints rather than assuming a revoke command killed tasks. Keep old payloads diagnosable but never authoritative.
7. Verify and roll back safely: inject late completions, worker crashes, clock skew, delayed messages, concurrent retries and partial database/network failures. Confirm stale-commit count is zero and required handoff facts survive. A rollback to old *logic* must still use a newer fencing generation, never reactivate an older token.

How the atomic guard works

A conceptual SQL mutation is:

SQL
UPDATE workflow_memory
SET compacted_payload = :candidate,
    memory_revision = memory_revision + 1
WHERE tenant_id = :tenant
  AND workflow_id = :workflow
  AND fence_generation = :task_generation
  AND memory_revision = :read_revision
RETURNING memory_revision;

Exactly one row updated means the write passed this guard; zero rows means a stale generation, stale revision, or missing scope and must not be interpreted as success. In a real implementation, canonical source facts are checked before committing, authority changes occur in controlled transactions, and relevant authorization/RLS rules are enforced. For multiple side effects, use an atomic outbox or a coordinated writer; this single-row example is not an all-system transaction.

Failure scenarios, trade-offs and prevention

• Only checking an in-memory flag: a disconnected worker retains stale permission. Protect the storage commit.
• Reusing generation numbers after rollback: an old worker may become valid again. Fence values must increase monotonically.
• Relying solely on distributed locks: expired leases or paused processes can still write; fencing handles late holders when the sink enforces it.
• Non-transactional vector database writes: the relational guard and external mutation can diverge. Stage candidates, use versioned keys or a serialized writer, and reconcile failed operations.
• Overly coarse fencing: one tenant-wide fence can block unrelated workflows. Finer scopes improve concurrency but increase metadata and recovery complexity.
• Audit gaps: record task_id, workflow_id, policy_version, fence_generation, memory revision, rejection reason, and the resulting content fingerprint, without exposing sensitive contents.

Operational acceptance tests: inject 100 intentionally delayed old-generation commits after cutover and require 0 accepted, confirm all live-generation writes are either committed once or safely retried, and compare compaction outputs against canonical facts. A finite test does not mathematically prove absence of rare failures; continuously alert on stale-write attempts and any accepted stale writes.

References

• Celery tasks: https://docs.celeryq.dev/en/stable/userguide/tasks.html
• Celery workers and revocation: https://docs.celeryq.dev/en/stable/userguide/workers.html
• LangGraph durable execution: https://docs.langchain.com/oss/python/langgraph/durable-execution
• Martin Kleppmann, *How to do distributed locking*: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
• PostgreSQL UPDATE: https://www.postgresql.org/docs/current/sql-update.html