MLOpsModel Deployment & CI/CD Automation

How do you design dynamic batching, deadline-aware scheduling, and fair concurrency for production ML inference?

Dynamic batching combines compatible concurrent inference requests into efficient compute batches; deadline-aware scheduling decides which work may safely wait; fairness controls stop one tenant, priority, or unusually large request from monopolizing the model. The production objective is not maximum GPU utilization. It is maximum valid, on-time, authorized predictions per unit cost while respecting tenant boundaries and business deadlines.

Curriculum boundary: Question 126 planned fleet capacity/autoscaling, Question 127 covered overload admission and backpressure, Question 133 optimized total inference cost, Question 138 covered numerical parity across runtimes, and Question 139 managed model loading/warmup and memory safety. **Question 140 addresses the *per-model execution scheduler after replicas are ready*: dynamic batch formation, queue-delay budgets, size-aware grouping, priority and tenant fairness, sequence affinity, cancellation, and per-request output mapping.** This is not a repetition of cluster capacity or generic API rejection behavior.

Beginner-friendly production example

A delivery-risk model on a GPU processes individual checkout requests in 12 ms each at batch size one. Processing eight together takes 30 ms instead of 96 ms sequentially, giving substantially higher throughput. An engineer sets a 100 ms batching wait in pursuit of large batches.

But the application has a 100 ms total customer deadline, including feature retrieval, preprocessing, inference and response networking. A request that waits 100 ms *before* model execution is already too late. Likewise, if one tenant sends 90% of traffic, a single shared FIFO queue may delay everyone else even when sufficient GPU compute exists.

A safe scheduler therefore measures *end-to-end deadline remaining*, not only its own compute efficiency, and imposes tenant-appropriate queue service policies.

Building blocks

  • Batch formation: Combine inputs supported by the same model, exact model revision, compatible shape/layout, preprocessing and privacy requirements; preserve an individual request-to-result mapping.
  • Maximum batch size: Limits work in one model execution; for variable-length models also bound total tokens/pixels/nodes and expected activation memory, not just request count.
  • Batch wait cap: Optional time to accumulate compatible arrivals; waiting consumes the request's deadline.
  • Queue timeout: A separate upper bound for queued work; on expiry, return a documented timeout/rejection rather than executing a result no one can use.
  • Priority and fairness: Differentiate approved urgency classes, protect each tenant's ability to make progress, and avoid starvation of lower-priority clients.
  • Concurrency cap: Limit in-flight batches/model instances so the GPU, CPU preprocessing and feature dependencies retain safe operating margins.
  • Stateful sequence routing: Keep requests belonging to one model session/sequence on the correct model instance and order, using the runtime's supported sequence-scheduling mechanism.

Source distinction: NVIDIA Triton's dynamic batcher is intended for stateless models. Its *sequence batcher* handles stateful models requiring ordered requests and instance affinity. Triton exposes queue-delay, priority and timeout settings. Ray Serve exposes maximum batch size, maximum wait and (in current documentation) custom effective batch-size functions for heterogeneous requests. Their specific semantics differ; avoid copying settings between products without checking actual runtime behavior.

Numerical example 1: throughput versus queue wait

Suppose a GPU can process:

  • One prediction in 12 ms: ideal saturated compute throughput roughly 1 / 0.012 = 83.3 predictions/sec.
  • Eight predictions in 30 ms: ideal saturated compute throughput roughly 8 / 0.030 = 266.7 predictions/sec.
  • Batch-eight compute throughput is 3.2× batch-one throughput, under these hypothetical measurements.

However, assume the first request waits 20 ms to form a batch, plus:

  • Feature fetch 25 ms.
  • Preprocessing 10 ms.
  • GPU batch execution 30 ms.
  • Postprocessing/network 10 ms.

Its end-to-end latency is approximately 25 + 10 + 20 + 30 + 10 = 95 ms if stages are sequential.

With a 100 ms deadline, just 5 ms remains for unexpected variation. If the batch wait were 40 ms instead, latency becomes 115 ms, breaching the deadline even though GPU throughput is excellent. Tail latencies and concurrent resource contention can make outcomes worse.

Lesson: Benchmark batch throughput and per-request p95/p99, but gate on actual fraction of requests completing before the customer's deadline.

Numerical example 2: arrival rate and expected batch fill

Requests arrive at a hypothetical average of 100/sec. An illustrative batching timer waits up to 20 ms after the first request.

  • Expected *additional* arrivals under a stationary Poisson approximation: 100 × 0.020 = 2.
  • Including the triggering first request, approximate expected batch population is 3, before accounting for maximum size, compatibility groups, timer behavior and scheduling.
  • This is not a guarantee that every batch contains three requests: arrivals vary, and shape/tenant/version constraints can fragment groups.
  • If traffic falls to 10/sec, expected extra arrivals in the same window are 10 × 0.020 = 0.2, so the scheduler will frequently execute single requests after a needless wait.

For latency-critical APIs, choose a modest delay or issue available batches promptly. Do not set the delay based only on peak-hour traffic; test low-load and bursty periods.

Numerical example 3: deadline-aware scheduling

A request has an end-to-end 150 ms deadline. By the time it reaches the batching queue:

  • 35 ms has already elapsed in authentication and feature retrieval.
  • An estimated 40 ms will be needed for GPU execution.
  • An estimated 15 ms for postprocessing and response transit.
  • Reserve a 10 ms safety margin for jitter.

Maximum remaining queue budget is: 150 − 35 − 40 − 15 − 10 = 50 ms.

A queue policy allowing 80 ms is unsafe for this request even if other customers have more generous deadlines. Pass an absolute deadline or derived remaining budget through the service; include clock-skew/monotonic-time considerations for cross-process deadlines.

If the remaining safe budget is zero or negative, reject/defer using the approved contract rather than wasting compute on a result that cannot meet the user's deadline. This does not replace Question 127's admission-control design; it gives the per-model scheduler the actual timing information needed to make a correct batching decision.

Numerical example 4: tenant fairness

Two authorized tenants share a batchable model:

  • Tenant A submits 900 predictions/sec.
  • Tenant B submits 100 predictions/sec.
  • The shared GPU fleet can sustain 800 useful predictions/sec at the accepted batch/latency operating point.

Demand totals 1,000/sec, which exceeds the sustainable rate. The scheduler cannot guarantee every request completes on time.

If an explicit weighted fair-sharing policy reserves 25% of the service capacity for B when it has demand:

  • B has a nominal allocation of 0.25 × 800 = 200/sec, but needs only 100/sec.
  • The unused 100/sec can be borrowed by A when policy permits.
  • A could then receive 700/sec and B 100/sec, meeting B's demand while rejecting/defering approximately 200/sec of A's excess load.
  • These figures assume predictable service cost per request; unequal batch sizes, tokens, priorities, bursts and GPU memory can change effective capacity.

The implementation should enforce admission/queuing quotas and measure per-tenant outcomes. Fairness is not guaranteed by placing tenant labels on requests. Priority queues can also starve low priority traffic unless service shares, aging or bounded priority mechanisms are defined. The precise weighting is business-specific and must not permit cross-tenant information exposure.

Numerical example 5: variable-length batches and GPU memory

A transformer inference server supports at most 4,096 tokens in one approved batch. Its arrivals include sequences of lengths 1,500, 1,200, 900 and 800 tokens.

  • The first three total 1,500 + 1,200 + 900 = 3,600 tokens and fit.
  • Adding the fourth would total 4,400, exceeding the 4,096 token policy.
  • A request-count-only cap of 4 would incorrectly admit all four.
  • A resource-aware packer forms a batch of the first three and routes the remaining request to another batch or waits within its own safe deadline.

Token count is not a complete memory model: padding strategy, attention mechanism, KV cache, activation tensors and runtime-specific implementation affect actual VRAM. Measure peak device memory under representative shapes and maintain the headroom policies established in Question 139.

Practical scheduler design

1. Declare the business SLO and input contract

  • Record end-to-end deadline, allowed fallback/rejection semantics, supported model versions, batchable shapes, maximum tokens/work and important decision types.
  • Separate strict interactive requests from latency-tolerant asynchronous work where possible.

2. Benchmark GPU batch profiles

  • Measure batch sizes 1, 2, 4, 8, 16 and relevant irregular/shape-specific cases on actual hardware.
  • Record compute duration, peak memory, throughput and numerical/decision parity. Some accelerators achieve their best performance at particular sizes, but these are measured properties, not guarantees.

3. Define scheduling groups

  • Group only requests that share compatible model digest, preprocessing contract, tensor dimensions and approved isolation policy.
  • Support safe padding/bucketing where needed and avoid mixing stateful sessions in a stateless batcher.
  • Return each output to the correct authenticated request even when execution ordering differs from arrival ordering.

4. Bound batch wait and queue capacity

  • Set maximum wait from the remaining end-to-end deadline after accounting for all stages.
  • Enforce finite queue length, per-request timeout and cancellation semantics.
  • Watch for queue size hiding debt: a queue that grows indefinitely can report high GPU utilization while users receive worthless late responses.

5. Allocate scarce execution fairly

  • Establish classes or per-tenant shares with documented priorities, burst rules and starvation protection.
  • Do not rely on an arbitrary client-supplied priority=high value. Derive scheduling authority from trusted server-side identity and policy (Question 134).

6. Handle cancellations and retries

  • Mark client-disconnected or expired queued requests as cancelled and remove where supported.
  • If a request has already entered an accelerator batch, cancellation may not interrupt the whole batch; account for wasted work and avoid returning the cancelled result as a new business action.
  • Coordinate idempotency and tracing: an application retry can otherwise generate duplicate execution and bias throughput metrics.

7. Validate stateful and variable-length cases

  • For conversational/session models, use supported sequence scheduling with correlation IDs, start/end markers and instance affinity rather than pooling independent steps arbitrarily.
  • For graph, sequence or multimodal workloads, budget based on actual work units and memory—not just batch item count.

8. Stage and roll out

  • Use representative low-, normal-, and peak-load scenarios, mixed tenants, variable shapes, sudden cancellation and model warmup events.
  • Compare candidate versus stable using the same actual eligible traffic mix and correctness criteria. Keep an immutable rollback configuration.

A small executable-style Ray Serve example

PYTHON
from typing import List from ray import serve @serve.deployment class RiskPredictor: @serve.batch(max_batch_size=8, batch_wait_timeout_s=0.005) async def predict_batch(self, inputs: List[dict]) -> List[dict]: # Educational placeholder for a vectorized, side-effect-free model call. # Real code must validate tenant, schema, deadlines, model version and shapes. return [{"risk_score": 0.3} for _ in inputs] async def __call__(self, request: dict): return await self.predict_batch(request)

The decorator shows Ray Serve's batching API, not a secure deployable service. Validate result count and request-output alignment, use an actual model and implement authorization outside/inside the trusted boundary as appropriate. A five-millisecond batch-wait cap is only an illustrative choice; profile under real load. Ray's current documentation also supports max_concurrent_batches and custom batch_size_fn for work-based batching; verify your installed Ray version.

NVIDIA Triton dynamic batching example

PROTOBUF
# Educational model-config fragment; other required config fields omitted. max_batch_size: 8 dynamic_batching { preferred_batch_size: [4, 8] max_queue_delay_microseconds: 5000 default_queue_policy { max_queue_size: 128 } }

This is not a complete Triton `config.pbtxt`. Its behavior also depends on backend/model support, instance-group configuration, server version and specific queue-policy settings. Triton's dynamic batching is for stateless models. Stateful models require the appropriate sequence batcher; inspect model readiness, ordering and per-model metrics before claiming correctness.

Failure scenario 1: head-of-line blocking

Symptoms: A small urgent request waits behind a batch of large inputs. GPU utilization remains high while short-request p99 and error-budget burn increase.

Diagnosis: Inspect per-request input work units, queue age, scheduling-class mix and actual compute time. Distinguish slow batching assembly from GPU processing, and identify whether large inputs monopolize the same executor.

Correction: Use compatible size buckets, explicit deadline-aware queues, controlled priority or physically separate latency classes where business risk warrants it. Bound large-request work and capacity. Avoid unconditional prioritization that starves ordinary users.

Prevention: Load tests with deliberately mixed tiny and huge requests, and per-class p95/p99 and completion-rate acceptance gates.

Failure scenario 2: high throughput, broken p99

Symptoms: Batch-eight throughput rises, but most low-traffic requests wait for max_batch_size or a long timer.

Diagnosis: Plot actual batch-size histogram by traffic period, queue delay, execute duration and end-to-end deadline breaches. Confirm the scheduler flushes incomplete batches as configured.

Correction: Reduce waiting, issue partial batches earlier, and separate interactive from offline traffic. Do not infer successful performance from throughput measured with continuously saturated traffic only.

Prevention: Low-RPS, burst, and cooldown test profiles plus good-event SLO gates.

Failure scenario 3: a noisy tenant starves others

Symptoms: Tenant A's increased traffic consumes GPU queue slots; Tenant B's small workload times out.

Diagnosis: Compare tenant-scoped eligible demand, admitted versus completed requests, queue wait, rejection, actual cost units and authorized priority. Avoid placing tenant IDs in high-cardinality global metric labels without cardinality controls.

Correction: Apply trusted fairness weights, per-tenant queuing/admission limits and controlled spare-capacity borrowing. Protect shared feature-store capacity as well as GPU time.

Prevention: Mixed-tenant experiments verifying both aggregate SLO and minimum service for protected tenant classes.

Failure scenario 4: batch output is mapped to the wrong customer

Symptoms: Model returns correctly shaped predictions, but results are associated with the wrong request IDs after reordering or timeout.

Diagnosis: Trace batch assembly indices, request identity, model execution order and output scattering. Test cancellation/race conditions, not just whole-batch output length.

Correction: Bind immutable per-request context to batch indices and verify one-to-one output mapping. Respect security boundaries from Question 134; never share outputs or cache entries across tenants.

Prevention: Randomized concurrency, cancellations and reordered-result contract tests with synthetic tenant data.

Failure scenario 5: stateful model mixed as stateless batch

Symptoms: Predictions depend on another user's previous session state or sequence steps arrive out of order.

Diagnosis: Check session correlation IDs, scheduler type, instance affinity, start/end markers and concurrent steps. A generic stateless dynamic batcher is not proof of sequence safety.

Correction: Use a state-aware sequence scheduler or dedicated state storage with explicit ordered processing and tenant isolation. Reject malformed sessions instead of guessing.

Prevention: Concurrent session tests with deliberate interleaving, replay and instance loss.

Failure scenario 6: oversized batch triggers OOM

Symptoms: Increasing max_batch_size causes sporadic GPU OOM when long sequences coincide, although average requests fit.

Diagnosis: Correlate peak VRAM to total effective work units, activation/padding growth, simultaneous model instances and input shape. Compare memory allocated and reserved where supported.

Correction: Enforce work-based batch limits and memory headroom; reduce execution concurrency or bucket shapes. Retest actual worst-case loads and rollback safety.

Prevention: Worst-case shape tests, mixed-version rollout overlap tests and OOM-negative fixtures, building on Question 139.

Failure scenario 7: expired jobs keep consuming GPU

Symptoms: Customer deadlines have passed, but a deep internal queue continues running old requests; retries create more work and prolong the incident.

Diagnosis: Compare request arrival/deadline with actual batch start, cancellation status, queue age and retry duplicates.

Correction: Enforce expiration when queueing, before execution and before any authoritative business side effect; coordinate client retry policy and idempotency. Do not claim an accelerator operation can always be preempted mid-batch.

Prevention: Cancellation storms and queue-saturation tests, plus a gauge of expired work executing after its business deadline.

Monitoring and release criteria

  • Per-request: Eligible arrival time, deadline, queue wait, batch start/end, model version, result identity, total latency, invalid/timeout status and mode.
  • Scheduler: Batch-size and total-work histograms, batch formation wait, queued count/age, priority-class wait, per-tenant accepted/completed/expired counts, cancelled and wasted compute.
  • Runtime: GPU utilization and VRAM peaks, model execution duration, concurrent batches, warmup/reload state and OOM errors.
  • Correctness: Batch-to-request output mapping, schema validity, numerical/decision parity, session-order invariants, feature freshness and forbidden cross-tenant output.
  • SLO: Fraction of valid on-time responses by relevant traffic cohort, business priority and release; aggregate throughput alone does not prove fairness.
  • Economics: Cost per useful prediction and capacity consumed by late/cancelled requests.

Suggested gate: reject a scheduler change if it violates critical output isolation, sequence integrity, deadlines, memory caps or approved minimum-tenant service. Accept only after paired staging tests and controlled rollout demonstrate a legitimate efficiency gain without harming required user outcomes.

Engineering trade-offs and limitations

  • Larger batches: Higher tensor efficiency but greater wait, memory peaks and potential tail latency.
  • Tiny/no batching delay: Better responsiveness at low RPS but often lower GPU utilization.
  • Strict deadline scheduling: Avoids spending compute on unusable results but can reduce batch fill and requires reliable deadline propagation.
  • Size bucketing: Reduces padding and head-of-line blocking but fragments queues and can leave rare shapes waiting.
  • Priorities: Protect critical work but risk starvation; define quotas or aging and prevent untrusted clients from self-assigning priority.
  • Tenant partitioning: Stronger fairness/isolation but sometimes wastes idle capacity; controlled borrowing improves utilization.
  • Work-unit caps: Better protection than item count for variable-length requests but depend on runtime-specific memory/compute estimates.
  • Stateful sequence routing: Preserves context integrity but constrains scheduling freedom and failover.
  • Aggressive cancellation: Saves some queued work but cannot always stop GPU batches already in flight.

Limitations: Neither queueing simulations nor a single benchmark proves performance under actual heavy-tail input distributions, changing customer mix or correlated failures. Scheduler semantics are implementation/version-specific. Revalidate every time batch policy, model graph, accelerator, tokenizer, feature preprocessing, or serving runtime changes.

Common mistakes

  • Optimizing only requests/sec without measuring total on-time valid customer decisions.
  • Equating max_batch_size=8 with every live batch being size eight.
  • Using a long batching timer with a stricter end-to-end request deadline.
  • Ignoring model preprocessing, feature-fetch and network time when budgeting queue wait.
  • Treating all requests as equal cost when token/pixel counts vary dramatically.
  • Allowing a single high-volume tenant to occupy every queue slot.
  • Letting clients set privileged scheduling priority without authorization.
  • Mixing stateful sequences in a stateless batching path.
  • Forgetting that output reordering and cancellation can misassociate customer predictions.
  • Keeping expired, disconnected requests in a queue indefinitely.
  • Increasing concurrency without checking GPU activation peaks and model coexistence.
  • Assessing fairness from global averages while a small protected cohort fails.

Production checklist

  • Versioned model and prediction contract; approved batching compatibility and session semantics.
  • Per-request deadline carried through the real queue and postprocessing path.
  • Measured batch-size-versus-latency/memory profile on target hardware.
  • Queue, batch and work-unit limits; cancellation and retry policy; protected critical decisions.
  • Tenant/priority fairness plan grounded in trusted identity and business policy.
  • Explicit stateful sequence treatment and one-to-one result mapping.
  • Low-, normal-, bursty-, and peak-load tests with variable shapes and tenant contention.
  • Monitoring for queue age, batch efficiency, p99 deadline misses and useful-prediction cost.
  • Safe rollback to the prior scheduler/runtime policy if invariants regress.

Takeaway: Dynamic batching is a scheduling problem, not merely a performance switch. Build batches only when they fit both the model's resource limits and every request's correctness/deadline contract. Protect tenants and sessions, measure actual user outcomes, and trade a little accelerator efficiency for predictably safe decisions when necessary.

#dynamic batching ML inference deadline scheduling#NVIDIA Triton batcher queue priority fairness#Ray Serve request batching latency throughput#production ML inference tenant fair scheduling#GPU inference batch queue head of line blocking#mlops#model-deployment#dynamic-batching#inference-scheduling#fairness

Learning Sequence

MLOps Curriculum

100% Completed

Explore Related Questions

Deepen your understanding across related MLOps topics.