How do you decide whether a model should run as a batch job or online service?

Choose batch inference when predictions can be computed ahead of time for many records; choose online inference when a client needs a response during a live request. The decision depends on response-time SLA, traffic shape, input freshness, throughput, cost and failure handling—not the algorithm alone.

This begins Model Deployment & CI/CD Automation. Question 114 connected experiments to the model registry. This lesson assumes an approved model exists and asks how predictions should be delivered.

Beginner-friendly example: delivery ETA

A food-delivery company uses the same model for two purposes:

• Live checkout: A customer requests a delivery estimate using current driver location and traffic. The service aims for p95 end-to-end response latency below 200 milliseconds. Choose online inference.
• Next-day courier planning: Operations wants predicted durations for 1,000,000 hypothetical deliveries by 07:00 the following day. Choose scheduled batch inference.

Batch and online predictions might share model weights but have different service contracts.

Batch inference: scheduled scoring

Typical flow:

Data snapshot → Validate → Batch prediction job
      → Committed prediction files/table → Downstream consumers

• Start hourly, nightly, weekly or when a committed input snapshot appears.
• Process large datasets in partitions; parallelize where supported.
• Write predictions with stable entity IDs, input snapshot, model version and scoring time.
• Publish outputs only after required partitions finish and validation succeeds.
• Measure job-completion SLA, output count, backlog and cost per record.
• Stop compute after scoring when infrastructure permits.

Best for: Precomputed risk scores, nightly recommendations, campaign targeting, planning dashboards and bulk document classification. Batch is not intrinsically slow per record; it optimizes large-scale processing rather than interactive response latency.

Online inference: serve an API

Typical flow:

App request → Authentication → Current feature lookup
      → Model inference → Response

• Expose a secure endpoint such as POST /predict.
• Keep model ready in memory or use suitable managed/serverless infrastructure.
• Budget for network, authentication, preprocessing, feature retrieval, model execution and serialization.
• Scale replicas based on peak load, latency distributions and saturation.
• Enforce request deadlines, authorization, input validation, rate limits and controlled rollback.
• Measure p50/p95/p99 response latency, timeouts, RPS and error rate.

Best for: Checkout ETA, live fraud decisions, instant recommendations and interactive personalization.

Numerical example: throughput and latency

Suppose checkout traffic peaks at 120 requests/second. A tested server replica handles 40 requests/second while meeting the required latency under representative conditions.

• Absolute theoretical minimum: ceil(120 / 40) = 3 replicas.
• With 70% target steady utilization, planned effective capacity is 40 × 0.70 = 28 RPS each.
• Planning estimate: ceil(120 / 28) = 5 replicas.

Five replicas are a starting capacity estimate, not proof of a 200 ms SLA. Test request bursts, concurrent feature reads, autoscaling lag and p95/p99 behavior.

For the nighttime planning batch, suppose:

• Input: 1,000,000 records.
• Four workers, each scoring 5,000 records/minute.
• Ideal aggregate throughput: 4 × 5,000 = 20,000 records/minute.
• Ideal scoring time: 1,000,000 / 20,000 = 50 minutes.
• Startup, validation and output publication: 15 minutes.
• Estimated batch duration: 65 minutes.

To finish by 07:00, the theoretical latest start is 05:55, without any margin for retries, variability or input delays. A real production schedule should include a buffer.

Different SLAs: The online service must satisfy *per-request response time*; the batch job must finish its *whole dataset* before a deadline.

Estimate infrastructure costs

The following rates are illustrative assumptions, not current cloud prices.

• Online: Two always-on instances at $0.40 per instance-hour, in a 30-day month: 2 × 24 × 30 × $0.40 = $576/month.
• Batch: Four workers at $0.40 per worker-hour, active 65 minutes daily: 4 × (65/60) × 30 × $0.40 = $52/month.

The batch estimate is cheaper for this particular nightly workload, but both exclude storage, network, feature lookups, replicas during peak hours, logs, startup and operational costs. Online serverless deployments can reduce idle expense but introduce cold-start and resource constraints. Repeated huge batches can be more expensive than efficiently utilized always-on serving.

Compare total cost per successful prediction, deadline success, and latency compliance, rather than instance prices alone.

Prediction freshness

• Online inference can include live signals, *if feature lookup is current and fast enough*.
• Batch results may remain in use until the next completed run.
• A nightly 01:00 job can leave predictions nearly 24 hours old before the next refresh, and older if the next job fails.
• A freshly scored output may still be stale if the *input features* were old.

Record the feature observation time, model version and prediction generation time. Enforce consumer-specific maximum permitted prediction age.

Monitoring and reliability

Batch monitoring:

• Completed versus expected input/output partitions.
• Records attempted, succeeded and failed.
• Queueing time, throughput, retry counts and completion-time SLA.
• Output integrity and version lineage.
• Scheduled-start/finish lag and cost per scored record.

Online monitoring:

• RPS, concurrency, p50/p95/p99 latency and timeout rates.
• Server saturation, autoscaling events and availability.
• Feature freshness, model version and input/prediction drift.
• Correctness after delayed labels and serving costs.

Batch recovery: Retry transient processing failures, resume eligible partitions with idempotency keys and publish only complete committed outputs. Prior outputs may be reused only when the consumer permits their age.

Online recovery: Use bounded retries, timeouts, health checks, capacity controls, rate limiting and documented fallbacks. A response timeout does not necessarily prove no side effect occurred; keep non-idempotent downstream actions safe.

Hybrid and asynchronous alternatives

A hybrid approach can precompute slowly changing restaurant baseline preparation times nightly, then combine them with live driver and traffic features during the online ETA request. This reduces online work while preserving request-time personalization. Track version and freshness of both the precomputed and live inputs.

Asynchronous inference is distinct from scheduled batch scoring: a caller submits a heavy or long-running request, gets a request/job ID, and retrieves its output later. AWS SageMaker distinguishes real-time, serverless, asynchronous and batch-transform inference options. Async APIs are covered in Question 117; production batch pipeline integration is covered in Question 116.

Decision checklist

1. Must the user wait for the prediction now? Strong signal for online.
2. Can the result be precomputed from stable inputs? Strong signal for batch.
3. What is the SLA? p95/p99 response latency for online; end-to-end job completion for batch.
4. How fast do inputs change? Define a maximum permissible prediction age.
5. How many requests or records? Estimate peak RPS, batch volume and concurrency.
6. What infrastructure is required? Include memory, CPU/GPU, model loading and feature retrieval.
7. What does full lifecycle cost look like? Include idle capacity and repeated jobs.
8. What happens on a missed deadline or outage? Specify fallback and retry policies.
9. Could a hybrid or asynchronous approach satisfy the requirements better?

Common mistakes

• Paying for always-on endpoints when scoring is only needed once daily.
• Assuming batch is suitable for a user-facing response requiring immediate live inputs.
• Measuring only model execution time instead of complete API latency.
• Sizing online capacity from average RPS and ignoring traffic peaks.
• Publishing partial batch outputs without a completion marker.
• Calling a prediction fresh while its feature data is outdated.
• Assuming all serverless endpoints meet strict latency without cold starts.
• Using expired batch predictions silently instead of applying a freshness policy.
• Failing to record model and dataset versions for prediction audit.

In short: Choose batch for precomputable, bulk predictions with a completion deadline; online for interactive, low-latency inference; and hybrid/async when latency, workload and freshness requirements call for a combination.