How do you prevent duplicate task execution across multiple agents?

Prevent duplicate task execution across agents by giving each logical task one identity and coordinating who is allowed to execute it. When several agents or Celery workers receive the same job, an atomic claim stops them from starting competing attempts. This is a *coordination* problem; idempotent side effects are an additional safeguard, not a replacement for coordination.

Beginner-friendly example: API test generation

Suppose two workers both receive a request to generate tests for POST /payments because the queue redelivers a message after a timeout.

Without coordination, both agents may spend tokens generating essentially the same tests and race to publish their suites.

With coordination, the job has an ID such as run-42:payments:test-generation. Both workers attempt to claim it in a shared database; only one successfully moves it from PENDING to RUNNING. The other observes that the job is already claimed and does not start competing generation.

Job received → Atomic ownership claim → Execute → Persist result → Mark complete

How should task ownership work?

1. Create a stable task ID: Identify one *logical* task independently of delivery attempt, agent, or queue message ID.
2. Persist task status: Track states such as PENDING, RUNNING, SUCCEEDED, and FAILED, plus an owner and claim expiry.
3. Claim atomically: Use a conditional database update or transaction so only one worker wins ownership.
4. Recheck before external effects: Ensure the worker still holds valid ownership when writing results.
5. Record the outcome: Store an artifact reference and completion state; duplicate deliveries should read the existing result.
6. Recover abandoned tasks: After verified lease expiry, allow a controlled new claim, with safeguards against the original worker continuing.

Simple SQL claim example

Assume the tasks table already contains a row for the task:

SQL
UPDATE agent_tasks
SET status = 'RUNNING', owner_id = 'worker-a'
WHERE task_id = 'run-42:payments:test-generation'
  AND status = 'PENDING'
RETURNING task_id;

One returned row means the claim succeeded. Zero rows means another worker probably claimed or completed it; inspect task state rather than executing anyway. Concurrent updates to the same PostgreSQL row are serialized appropriately for this conditional claim.

This minimal example does not implement recovery from crashed workers. Real implementations add lease expiry and a monotonically increasing fencing token to reject writes from an earlier owner.

What if the worker crashes?

A lease helps make abandoned work available again, but a lease timeout cannot prove that the original worker has stopped. A slow worker may regain connectivity and attempt to write after the task was reassigned.

Give each new owner an increasing fencing token, and require the result store or downstream service to reject operations from older tokens. If a downstream system cannot validate fencing tokens, use reconciliation and idempotency protections for its side effects.

Celery and LangGraph considerations

Celery tasks can be retried or redelivered depending on configuration; a task ID alone is not a distributed execution lock. Put the atomic claim in a shared transactional store, not each worker's memory.

In LangGraph, graph execution state and task-ownership state serve different purposes. Checkpoints preserve workflow progress; a task claim coordinates competing workers. Coordinate both with the same run/task identity.

What about repeated but legitimate requests?

Two users may intentionally ask for similar test suites. Do not deduplicate solely on prompt text or endpoint name. Include the tenant, request/run ID, operation version, and task scope in the logical task identity so distinct authorized jobs stay distinct.

In short

Use stable task identities, atomic claims, leases, and fencing to control execution ownership across agents. Add idempotent writes as defense in depth, because distributed systems cannot always guarantee that an earlier worker has stopped.