How do you validate a shared-memory migration before switching all agents to the new schema?

Validate a shared-memory migration before cutover by proving that the new schema preserves meaning, access controls, freshness, and retrieval behavior—not merely that the records can be parsed. The final decision should be a measurable go/no-go gate backed by independent checks.

Beginner-friendly example: API test-generation agents

Three agents share durable endpoint knowledge: a requirements extractor writes it, a test generator reads it, and a security reviewer checks the resulting test cases. The old record stores auth: "OAuth2"; the new record stores auth: {"scheme": "OAuth2", "scopes": ["pay:write"]} plus source provenance.

A migrated record is valid only if the scopes came from an authoritative requirement, the version is current, and the right tenant and agents can access it. Do not invent missing scopes or treat unknown information as approval.

Define acceptance gates → Validate converted records → Compare old/new reads → Test live updates → Exercise agent consumers → Canary → Cut over or stop

Numerical cutover example

Suppose the migration covers 10,000 records:

1. 9,990 records pass structural checks; 10 fail.
2. Of the 9,990 structurally valid records, 30 contain an incorrect source revision or missing authorization evidence.
3. Therefore 9,960 / 10,000 = 99.6% pass both structural and semantic validation.
4. In a 500-query retrieval comparison, 490 return the expected authorized evidence under the new store, an observed 98% pass rate.
5. The release criteria require 100% correct authorization and tenant isolation, at least 99.9% verified record correctness, and at least 99% retrieval-check success.

Result: NO-GO. The record-correctness rate is below 99.9%, the retrieval rate is below 99%, and the security-sensitive cases require zero known violations. Thresholds are illustrative; production criteria must match business and safety requirements. Do not hide failures with an overall average.

What validation should be performed?

1. Completeness: Reconcile expected and migrated record IDs, counts, and missing/deleted records by tenant and source. Account for writes made after the initial snapshot.
2. Structural correctness: Check schema versions, types, required fields, constraints, and references; quarantine unsupported or corrupt records.
3. Semantic equivalence: Compare canonical business meaning. For example, OAuth2 without known scopes must not silently become OAuth2 with an invented scope.
4. Provenance and freshness: Verify source document version, record revision, timestamps, and last-applied change event so stale converted data cannot override new updates.
5. Access control: Test cross-tenant isolation, least-privilege agent reads, and sensitive-data handling against both positive and negative authorization cases.
6. Retrieval quality: Test representative agent questions and expected evidence, including keyword and vector-search behavior, metadata filters, missing records, and cross-tenant negatives.
7. Concurrent updates: Inject a write while migration/backfill runs; ensure revision checks or a change-log catch-up preserve the newest authoritative value.
8. Integration behavior: Run requirements, test-generation, and security-review agents with the migrated memory; confirm validated output and safe handling of incomplete context.
9. Operational performance: Measure p95 retrieval latency, errors, index lag, migration backlog, and resource utilization against release criteria.
10. Recovery readiness: Demonstrate backups, rollback/routing procedures, and an owner-approved response to failed gates.

How do you compare old and new memory safely?

Use shadow reads: serve production requests from the authoritative old path while separately querying the new memory store without letting shadow results trigger publication, payments, notifications, or other external actions. Compare normalized evidence, provenance, security filters, and response quality.

Raw document equality may be the wrong test if storage formats legitimately changed; compare a canonical representation and independently evaluate retrieval outcomes. Do not expose other tenants' data in diff logs.

What happens to new writes during validation?

A backfill can be perfect at the snapshot time but still be stale at cutover. Keep one authoritative write path, capture updates durably (for example through a transactional outbox or change-data-capture mechanism), apply them idempotently to the new representation, and measure replication/index lag.

Before switching readers, confirm the new store has caught up to the agreed cutover watermark. Use revision-aware writes and read routing so late old-format events cannot overwrite newer records.

Canary and go/no-go checklist

• Run a small cohort of agents or workflows against the new memory schema.
• Compare accepted outputs, missed requirements, authorization failures, latency, and cost against the old path on comparable work.
• Require zero known cross-tenant leaks or unsafe approvals; treat any such finding as an immediate stop.
• Confirm unresolved exceptions have explicit owners and have not been silently declared migrated.
• Switch traffic progressively only after independent checks pass, then continue monitoring and keep a defined rollback route.

A small passing canary is not proof of universal correctness; expand coverage and account for statistical uncertainty.

LangGraph and Celery implementation

In LangGraph, test resumed and fresh workflows separately. A valid checkpoint may reference a memory record whose schema or revision changed. Ensure nodes revalidate required evidence rather than assuming a checkpoint guarantees memory freshness.

With Celery, run idempotent migration and verification tasks using stable record IDs, durable progress markers, and revision checks. Tasks may retry or arrive out of order; verification should detect stale, duplicated, or skipped updates. Keep legacy-compatible consumers available until queued old-version messages have drained.

Difference from earlier lessons

• Q220 explains how to execute shared-memory schema migration.
• Q221 explains how to maintain consistency while old and new memory coexist.
• This lesson defines the independent evidence and release gates required to declare the migrated system ready for agent cutover.

In short

Cut over only after record reconciliation, semantic and security validation, retrieval checks, live-write catch-up, agent integration tests, and a passing canary. A completed backfill is not the same as a verified migration.