What is an asynchronous prediction API and when should it be used?

An asynchronous prediction API accepts a prediction request, returns a tracking ID quickly, and processes the actual ML inference later through a queue and worker. The caller can check job status or receive a notification when the result becomes available. It is useful when prediction takes longer than a normal HTTP request should remain open, payloads are large, traffic is bursty, or the caller does not need an immediate answer.

This is request-driven asynchronous inference, not synchronous online inference (Question 115) or scheduled bulk batch scoring (Question 116). One client request typically creates one job or one explicitly scoped job containing multiple items; the job is processed independently of the initial HTTP connection.

Beginner-friendly practical example

A logistics company offers an API that analyzes a customer-uploaded delivery-route document and predicts which routes may be delayed. Each uploaded file requires document parsing, feature extraction and model inference and can take 45 seconds.

Keeping an ordinary HTTP request open for 45 seconds risks browser, gateway or load-balancer timeouts and ties up request-serving resources. Instead:

1. The client submits the document or an authorized upload reference with a unique idempotency key.
2. The API validates authentication, authorization, size, file type and processing eligibility.
3. The service durably records a PENDING job and enqueues work.
4. It returns HTTP 202 Accepted, a job ID and status URL.
5. A worker claims the job, loads the approved model and processes the input.
6. The worker stores an output artifact and marks the job SUCCEEDED or FAILED.
7. The client polls status or receives an authenticated webhook, then obtains the result.

The initial HTTP response says accepted, not completed. MDN documents that HTTP 202 is noncommittal: the later operation can still fail. Do not return a success-style prediction body before inference actually finishes.

API request and response contract

A representative request:

HTTP
POST /v1/prediction-jobs HTTP/1.1
Authorization: Bearer <token>
Idempotency-Key: route-doc-9876-v1
Content-Type: application/json

{
  "input_uri": "s3://example-customer-upload/routes/9876.pdf",
  "model_version": "route-delay/17"
}

After validation and *durable job acceptance*, respond:

HTTP
HTTP/1.1 202 Accepted
Location: /v1/prediction-jobs/job-01HXYZ
Retry-After: 5
Content-Type: application/json

{
  "job_id": "job-01HXYZ",
  "status": "PENDING",
  "status_url": "/v1/prediction-jobs/job-01HXYZ",
  "result_url": null
}

The placeholder token and URI are illustrative. The Location reference and Retry-After polling hint are useful API conventions, but 202 alone does not guarantee the server actually queued the task successfully. Use a transaction/outbox or reliable enqueue procedure so accepted jobs are not lost between recording the job and publishing to the queue.

A status endpoint might return:

HTTP
GET /v1/prediction-jobs/job-01HXYZ

HTTP/1.1 200 OK
Content-Type: application/json

{
  "job_id": "job-01HXYZ",
  "status": "RUNNING",
  "created_at": "2026-11-08T09:00:00Z",
  "attempt": 1,
  "result_url": null
}

When completed, the same endpoint returns SUCCEEDED with an authorized result reference, or a redirect to the result resource. A FAILED job must expose an appropriate safe error code; avoid exposing stack traces, signed storage credentials or sensitive model internals.

Architecture and handoffs

Caller
  │ POST /prediction-jobs
  ▼
API gateway + auth + validation
  │
  ├─ Durable job database: ID / tenant / status / model / input
  │
  └─ Reliable queue / outbox event
                   │
                   ▼
             Worker pool
                   │
             Parse + featurize
                   │
             Run pinned model
                   │
             Save result artifact
                   │
             Commit final status
                   │
      ┌────────────┴────────────┐
      ▼                         ▼
GET /jobs/{id}             Optional signed webhook
      │                         │
      └──── Client retrieves authorized result ────

Important boundaries:

• API nodes should be stateless; job status must survive API restarts.
• The queue contains a job ID or trusted payload reference, not raw secrets or necessarily the entire uploaded file.
• Every job is bound to a tenant/owner and immutable model version, authorized input and execution policy.
• The worker stores result bytes durably before publishing SUCCEEDED.
• The client reads results using authorization independent of whether it knows the job ID.
• Store an output expiry and deletion policy; async results should not remain accessible forever by default.

Numerical capacity and queueing example

Assume a document inference worker takes 45 seconds per request, on average, and can process one at a time.

• One worker's ideal service capacity: 60 / 45 = 1.33 jobs/minute.
• With 6 workers: 6 × 60 / 45 = 8 jobs/minute.
• Suppose average arrivals are 6 requests/minute: utilization is approximately 6 / 8 = 75%.
• A sudden burst of 60 extra jobs arrives while all workers are busy. With the simplified assumption of no continuing arrivals, the burst requires about 60 / 8 = 7.5 minutes of aggregate worker service. The last burst jobs may wait much of this interval before starting.

This is an idealized workload calculation, not a precise queue-latency prediction. Real wait times depend on job-duration variability, scheduling, worker startups, priority and continued arrivals. At arrival rates above 8 jobs/minute, the backlog grows unless capacity is increased or admission is restricted.

For an example service-level objective (SLO), choose:

• API acknowledgment: p95 under 500 ms for accepted requests.
• Job completion: 95% of eligible document jobs finish within 5 minutes, measured from acceptance.
• Result availability: Completed results retrievable within 10 seconds of final-state commit.

The example SLO can fail during the 60-job burst even though API acknowledgment remains fast. An async API does not eliminate processing latency—it removes it from the initial HTTP response. Be explicit about both acknowledgment latency and end-to-end completion time.

Polling versus webhooks

Polling:

• Caller periodically requests GET /prediction-jobs/{id}.
• Easy for mobile/web clients, works across firewalls and supports reconnects.
• Can produce many empty status checks and increased API load.
• Use Retry-After, exponential backoff, jitter and a maximum polling duration.

Webhooks:

• Server sends an event to a registered callback when the job completes or fails.
• Convenient for server-to-server integrations; reduces polling.
• Callbacks can be delayed, delivered more than once or fail.
• Authenticate webhook signatures, validate destination ownership, restrict callback destinations to prevent SSRF, retry deliveries with backoff, and let consumers deduplicate notification IDs.

Best practice: expose a durable status endpoint even when webhooks are enabled. A webhook is notification, not the sole source of truth. Microsoft’s async request-reply pattern documents polling through a status resource after an HTTP 202 acknowledgment.

Status state machine

PENDING → RUNNING → SUCCEEDED
   │         │
   │         ├─ transient failure → RETRYING → RUNNING
   │         ├─ terminal failure  → FAILED
   │         ├─ deadline exceeded → TIMED_OUT
   │         └─ cancel accepted   → CANCELLED
   └─ expired/cancelled → CANCELLED

Persist status transitions with timestamps, attempt numbers and reasons. A worker must not overwrite CANCELLED with SUCCEEDED if the cancellation won the race; use conditional state transitions or fencing tokens. Terminal outcomes should not be silently reset, and retries should remain auditable.

A result resource is available only for a SUCCEEDED job with verified durable output. A failed job is not equivalent to an HTTP 500 for the original POST, because that POST already returned 202.

Idempotency and retries

Suppose the client submits a job, the server accepts it, but the acknowledgment is lost. The client retries POST.

Without idempotency, the system might process the document twice and charge twice.

• Accept an Idempotency-Key scoped to tenant, endpoint and a defined retention period.
• Bind it to a canonical payload digest; if the same key is used with a different request, return a documented conflict rather than silently reusing an unrelated job.
• Return the same job ID for an equivalent retry.
• Use a transactional uniqueness constraint for key ownership, not an in-memory dictionary.
• Treat queue delivery as at-least-once unless exactly-once semantics are demonstrably provided end-to-end.
• Make worker claims lease-based and writes idempotent; duplicate workers should not publish conflicting results.
• Separate queue message delivery attempt from logical prediction job ID and model inference attempt.

For transient storage/network errors, retry with bounded exponential backoff and jitter. For invalid inputs, unsupported model versions, revoked permissions, or deterministic parsing errors, fail with a meaningful final error rather than retrying endlessly.

Use a dead-letter queue or failure store for jobs exhausted after the configured retry limit, while preserving authorization and privacy constraints.

Timeouts, cancellation and backpressure

Use separate limits:

• API deadline: Maximum time to validate and durably accept a job.
• Queue wait deadline: How long the job may remain pending.
• Execution timeout: Maximum worker run time.
• End-to-end job deadline: Maximum elapsed time from acceptance to final result.
• Result retention time: How long completed output can be downloaded.

A client disconnect must not automatically cancel a durably accepted job unless that is an explicit contract. Support a cancellation endpoint if needed; cancellation may be best-effort and must handle races with completion.

When arrivals exceed capacity:

• Cap per-tenant and global queued jobs.
• Apply quotas and fair scheduling; avoid one customer monopolizing GPU workers.
• Return 429 Too Many Requests or 503 Service Unavailable with a documented retry hint before acceptance when overloaded, rather than issuing a 202 for work the system cannot retain.
• Scale workers based on queue depth, oldest-job age and service time.
• Reject oversized payloads and protect against retry storms.

Merely adding workers does not always help when a shared model server, database or storage API is the real bottleneck.

Example Python API contract (illustrative)

PYTHON
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel

app = FastAPI()

class PredictionRequest(BaseModel):
    input_uri: str
    model_version: str

@app.post("/prediction-jobs", status_code=202)
def submit_job(body: PredictionRequest, idempotency_key: str = Header()):
    # 1. Authenticate the principal and authorize the referenced input.
    # 2. Check payload, job quotas and the model-version allowlist.
    # 3. In one durable transaction, insert/find a unique job and
    #    insert its outbox event; a dispatcher publishes it to the queue.
    job_id = durable_create_or_find_job(body, idempotency_key)
    return {
        "job_id": job_id,
        "status": "PENDING",
        "status_url": f"/prediction-jobs/{job_id}",
    }

This illustrates the endpoint, not executable production code: durable_create_or_find_job and authorization are intentionally unspecified, and a production response should also set a Location header. Never enqueue first and return 202 if a subsequent database failure would leave an untracked job. Job APIs must enforce tenant isolation for status/results and ownership of external input references.

Security and result handling

• Authorize submission, polling, cancellation, callbacks and result retrieval independently.
• Use unguessable job IDs but never treat possession of an ID as sufficient authorization.
• Validate input URIs, prevent arbitrary server-side URL fetches, and use short-lived access where appropriate.
• Encrypt payloads and prediction artifacts, avoid secrets in queue messages, and limit who can invoke each model.
• Publish only validated outputs; persist a checksum, content type, ownership and expiry.
• Use tenant-scoped storage paths or access control; expired signed URLs should be regenerated only after authorization.
• Apply deletion and data-retention policies to input and output artifacts.

For user-provided documents, logs should record sizes, statuses and trace IDs without copying sensitive document contents.

Monitoring and operations

Track both fast acknowledgment and slow execution:

• POST accept latency, rejects, auth failures and idempotency conflicts.
• Queue depth, age of oldest queued job, enqueue-to-start waiting time.
• Active worker count, utilization, throughput and autoscaling delay.
• End-to-end p50/p95/p99 completion time and deadline misses.
• Success, failure, retry, timeout and cancellation rates by model version.
• Worker errors, poison messages and dead-letter queue volume.
• Webhook delivery failures, repeat deliveries and polling load.
• Result-storage failures, orphaned outputs and expired artifacts.
• Per-tenant fairness and cost per successful completed prediction.

Monitor SUCCEEDED only when the result has been committed and is actually readable. Correlate every log with the same job ID, tenant scope, model version and tracing ID.

When to use—and not use—async inference

Use async for long-running document, image or video analysis, large payloads, bursty prediction traffic, expensive models that can wait in a queue, or workflows whose user can return later.

Avoid async when a transaction requires an immediate prediction before proceeding (for example, a hard real-time checkout decision), when the client cannot track completion, or when the overhead of queues and status storage is unjustified for a tiny low-latency model.

Distinguish the modes:

• Synchronous online: One request stays open until a prediction response or timeout.
• Asynchronous request-driven: One request creates a job; caller checks later or gets notified.
• Scheduled batch: A pipeline scores a known dataset on a schedule or source-data event, without requiring an individual API job per input record.
• Streaming: Event-processing architecture with continuous ingestion, not just a queue of finite one-off requests (Question 118).

Amazon SageMaker Asynchronous Inference demonstrates a managed approach: the caller supplies an S3 input reference, the service queues inference and writes results to S3, with optional SNS notification. This is an example of a platform-specific design; generic HTTP services can instead use their own queues, databases and workers.

Common mistakes

• Returning HTTP 202 and treating it as proof that inference succeeded.
• Acknowledging jobs before durable status and enqueue state exists.
• Keeping job status only in API process memory.
• Using raw job IDs as access tokens.
• Re-running duplicate client submissions without idempotency checks.
• Ignoring queue waiting time when promising a job-completion SLA.
• Allowing infinite retries of invalid or permanently failing requests.
• Sending callback webhooks to arbitrary user-provided internal network addresses.
• Marking a job completed before the output artifact is durable.
• Using one unbounded queue where a noisy tenant can starve everyone else.
• Confusing asynchronous request handling with automatic batch dataset processing.

In short: Design asynchronous inference as a durable job protocol: validate and accept, return 202 and a job ID, process through bounded workers, persist status and results, handle duplicate submissions and failures safely, and measure the full time from acceptance to result availability.