How do online, batch, and streaming serving architectures differ operationally?

Online, batch and streaming inference can use the same trained model, but they require different operating procedures after deployment. Online teams keep request-serving endpoints healthy, batch teams guarantee complete prediction datasets by deadlines, and streaming teams keep continuous event processing correct and sufficiently fresh.

The earlier lessons covered *choosing* a serving mode (Question 115), batch execution mechanics (116), request-driven async APIs (117), and streaming fundamentals (118). This lesson is about day-2 ownership: operating, upgrading, scaling, recovering and auditing each mode over time.

Practical example: one delivery-risk model, three operations teams

A food-delivery company owns a registered risk model late-risk/27. Three consumers use it:

• Online checkout service: Calls an endpoint to obtain a delivery-risk estimate before confirming an order.
• Nightly batch planning: Scores 1 million potential deliveries for the next day and publishes a complete dataset.
• Continuous driver-event monitoring: Updates risk from incoming GPS and order-status events and generates live operational alerts.

Although the model weights might be shared, serving contracts cannot be treated as interchangeable. A healthy batch job does not guarantee a responsive API; an API with 99.9% uptime does not guarantee streaming predictions are current; an active stream processor does not prove daily bulk output completeness.

Operational comparison by responsibility (pointers, not a table)

Online serving

• Primary owner: API/model-serving on-call team, coordinated with application, platform and feature-store owners.
• Input arrival: Live requests with bursty traffic and an interactive request/response contract.
• Runtime topology: Load balancer or API gateway, serving replicas, feature dependencies and observability.
• Primary SLO: Availability, error rate and end-to-end p95/p99 request latency.
• State: Usually stateless serving replicas plus external feature/cache state; model weights remain loaded.
• Scaling: Replicas, concurrency, GPU utilization, admission limits, warm pools and autoscaling lag.
• Release: Rolling, blue-green or canary deployment with rapid traffic rollback.
• Recovery: Fail over to healthy replicas or a prior serving revision, with bounded request retries.
• Cost: Often includes provisioned or warm idle capacity plus peak headroom.

Batch serving

• Primary owner: Data platform/pipeline team, partnering with model and downstream analytics owners.
• Input arrival: Committed, versioned input partitions on a business schedule or data-ready event.
• Runtime topology: Job scheduler, ephemeral scoring workers, output storage and consumer handoff.
• Primary SLO: Correct, complete predictions published by a specified deadline.
• State: Input snapshot ID, logical interval, partition receipts, output manifest and task progress.
• Scaling: Worker count, partition size, executor resource quota and backlog of pending windows.
• Release: Pin a model version per logical run; promote a new model on a chosen future run.
• Recovery: Retry incomplete partitions and publish a new atomic output revision; do not expose partial files.
• Cost: Compute during scoring windows, storage and data movement, plus retries.

Streaming serving

• Primary owner: Stream-processing/platform team with model-serving, broker and downstream alerting owners.
• Input arrival: Unbounded event streams, often partitioned by an event key.
• Runtime topology: Message broker, stream processor, keyed state/checkpoints, model execution and event sink.
• Primary SLO: Event-to-prediction freshness, throughput, lag, checkpoint success and output correctness.
• State: Source offsets, keyed/window state, timers, model version and sink delivery state.
• Scaling: Consumer parallelism, broker partitions, keyed-state resharding and inference capacity.
• Release: Controlled upgrade of processing logic, state schemas and pinned model epochs, sometimes with savepoints.
• Recovery: Restore coherent stream state/source positions and replay, then reconcile duplicate external effects.
• Cost: Continuously operating broker/processors, state storage, checkpoint I/O and serving resources.

Each system should have a clearly named service owner, runbook, alert severity, change approver, dependency owner and recovery objective. Shared model ownership does not remove infrastructure-specific responsibilities.

Numerical example: three different operating SLOs

Assume:

• Online checkout peaks at 200 requests per second. Required p95 response latency ≤ 250 ms.
• Batch planning must publish 1,000,000 scores by 06:00 UTC.
• Streaming ingests 2,000 events/sec with p95 event-to-prediction delay ≤ 3 seconds.

These are three different types of promises.

Online capacity: Suppose a replica sustains 50 RPS while meeting its tested latency. At a targeted 70% utilization, planning capacity is 50 × 0.70 = 35 RPS/replica. Needed replicas = ceil(200/35) = 6. Six is a capacity estimate, not proof of p95 latency under bursts.

Batch capacity: Suppose four workers together score 20,000 rows/minute. Compute time is 1,000,000 / 20,000 = 50 minutes. With 15 minutes of startup, validation and output publication, estimated completion is 65 minutes. Starting at 04:45 gives an estimated finish at 05:50, only 10 minutes of deadline margin, so retries or skew may breach the SLO.

Streaming capacity: Suppose processors sustain 2,500 events/sec while arrivals are 2,000/sec. Utilization is 2,000/2,500 = 80%. If an outage reduces sustained processing to 1,500/sec for 6 minutes, backlog increases by (2,000 - 1,500) × 360 = 180,000 events. At restored capacity 2,500/sec, with arrivals still 2,000/sec, net catch-up is 500/sec and draining takes 180,000/500 = 360 seconds = 6 minutes in an idealized model. The service may be *running* but late, so its freshness SLO can fail throughout catch-up.

Operational lesson: Online errors happen at a request boundary; batch failures delay an entire publication; streaming failures can create a growing lag that persists long after processes restart.

Day-2 release management differs

Online rollout

• Build a validated immutable serving image plus model/feature contract.
• Start new replicas, load and health-check model dependencies.
• Shift a small share of live traffic to the new version.
• Compare errors, p95/p99 latency, resource consumption and prediction behavior.
• Increase traffic gradually or restore the previous deployment.
• Treat a traffic rollback as a serving action, not as deletion of the new registry version.

Batch rollout

• Select a specific registry version at the beginning of a logical scoring interval.
• Keep the version fixed across all scoring partitions.
• Validate counts, output schema and prediction distribution.
• Commit a result manifest only after complete verification.
• If a model bug is discovered, re-score affected windows as a new output revision and promote it deliberately.

Streaming rollout

• Pin model identity and feature/state schema in the running job.
• Plan operator-state compatibility and source-offset continuity.
• For stateful job changes, use compatible savepoint or checkpoint migration where supported; test restores before production.
• Define a model-version transition boundary and include version in outputs.
• Validate no event gaps or unintended duplicate side effects during cutover.

A streaming job rolling update that changes keyed-state structure may require a migration rather than a simple replica replacement. A batch job with two models mixed within one output snapshot is inconsistent even if every partition succeeded. An online deployment with new model weights but incompatible live feature schema can fail during a canary despite passing offline evaluation.

Operational consistency: what does “correct” mean?

For online: Correctness at request time means a versioned request is validated, current-enough features are used, and its response belongs to the intended model. Retrying a request may produce another response if inputs change; define idempotency where externally visible side effects exist.

For batch: Correctness for the logical interval means all eligible records are scored with the pinned model and input snapshot, with no unexplained duplicate/missing keys and one committed output revision.

For streaming: Correctness over an event timeline requires event-time handling, state consistency, source positions and sink semantics. Exactly-once internal checkpoint restoration does not imply an email, HTTP callback or other external action happened exactly once. Deduplicate or transactionally coordinate side effects.

Model version consistency applies to all three, but the lifetime differs: individual request, scoring run, or processing epoch.

Scaling and cost: pay for different bottlenecks

• Online: Sustained traffic often justifies always-ready replicas; strict tail-latency requirements may require spare capacity. Serverless options trade idle cost for potential cold-start limits.
• Batch: Add workers or partitioning to meet a deadline; after job completion compute may deallocate. High parallelism can still be limited by storage reads or skew.
• Streaming: More partitions and task slots can raise throughput, but large keyed state, checkpoint barriers, a hot key or a single remote model endpoint may limit scaling.

Illustrative prices, *not provider quotes*:

• Two 24/7 serving instances at $0.50/hour over 30 days: 2 × 24 × 30 × 0.50 = $720 in compute.
• Four batch workers at $0.50/hour for 2 hours/day over 30 days: 4 × 2 × 30 × 0.50 = $120 in compute.
• Three continuously running streaming workers at $0.50/hour: 3 × 24 × 30 × 0.50 = $1,080 in worker compute.

These scenarios have different capacity, workloads and supporting costs. They do not prove that one mode is always cheaper. Count broker, network, object storage, state, logging, failure retries, model inference calls and operational labor before comparing true total cost per business outcome.

Azure ML documentation makes an important platform distinction: online endpoint resources are commonly provisioned with the deployment, while batch deployment compute can be allocated for the job and released afterward. Actual bills depend on service configuration and cloud provider. See official Azure and SageMaker references.

Failure recovery and escalation runbooks

Online incident: p99 latency spikes

1. Check feature dependencies, request queues, saturation, latency by model version and recent rollout.
2. Reduce traffic to unhealthy/new versions; scale or shed overload safely.
3. Preserve request IDs and the serving version needed for investigation.
4. Verify restored p95/p99 and error rates before incident closure.

Batch incident: scores not published by deadline

1. Identify the logical window and source/model identities.
2. Inspect failed partitions, upstream readiness, worker failures and output reconciliation.
3. Retry only incomplete safe partitions or start a controlled new revision.
4. Publish once all requirements pass, record whether the SLA was missed, and notify consumers of availability or fallback status.

Streaming incident: output lag growing

1. Compare source arrival rate, input/consumer lag, processing rate, watermark lag and sink errors.
2. Find the bottleneck: partition skew, model calls, checkpoint failure or downstream backpressure.
3. Restore from a valid checkpoint/savepoint when needed and replay safely.
4. Track catch-up until output freshness recovers; verify duplicates or missed events do not violate contracts.

Avoid a universal “restart the service” playbook. Restarting online replicas may briefly reduce capacity; restarting a batch job may duplicate outputs; restarting streaming can replay events and repeat side effects.

Monitoring and incident-response dashboards

Online dashboard: request count, error rate, p50/p95/p99, timeouts, saturation, model versions, warmup and feature-store latency. Alert on user-visible SLO violations, not CPU spikes alone.

Batch dashboard: expected/complete partitions, scoring run ID, logical date, rows scored, quarantine rates, publication deadline, last committed output revision and failed retries. Alert when a required output will not meet its deadline.

Streaming dashboard: input/processed events, source lag, watermark lag, event-to-output latency, backpressure ratio, keyed-state size, checkpoint age/duration, sink commits and restart count. Alert on stale predictions even if workers report healthy.

Common model-quality dashboard: prediction distribution, feature completeness, data/feature drift, delayed-label performance, version provenance and segment performance. Labels may arrive on different schedules across serving modes; interpret model quality with appropriate time alignment.

Hybrid operational ownership

Real systems often combine modes:

• Batch scores the daily risk of every planned order.
• A streaming processor updates route context as GPS events arrive.
• An online endpoint combines precomputed risk with fresh information at checkout.

To keep this coherent:

• Establish a shared model/feature schema and version catalog.
• Record the source timestamp and freshness budget of each precomputed or streamed feature.
• Define which layer is the system of record for each prediction or alert.
• Prevent the same business action from being triggered independently by batch and stream without an explicit deduplication rule.
• Set clear ownership for cross-mode incidents and correlated tracing IDs.
• Run parity checks on comparable records to detect offline/online feature skew while acknowledging intentional freshness differences.

For example, the online service should not silently assume that today's batch risk score is valid if the newest batch failed. It must enforce a maximum age and choose a documented fallback.

A practical operating contract

For each deployed mode, maintain a short service contract containing:

1. Owner and consumer: Who publishes predictions, who uses them and who is on call.
2. Version and input: Approved model identity, feature schema, event or snapshot version and release boundary.
3. Primary SLO: Request latency, batch publication deadline or event-to-prediction freshness.
4. Capacity: Peak RPS, batch records/time or streaming events/sec and resource headroom.
5. Failure semantics: Retry scope, idempotency, output visibility and rollback/replay strategy.
6. Data correctness: Freshness, completeness, duplicate handling, version parity and audit linkage.
7. Monitoring: Dashboards, alerts, runbooks and incident review criteria.
8. Cost: Compute, storage, state, data movement, retries and cost per useful prediction.
9. Change policy: Canary, future-run promotion or stateful stream migration.

Do not assume a platform's autoscaling or health check alone enforces this contract.

Common operational mistakes

• Applying one latency SLO to batch publication and streaming event freshness.
• Alerting on “workers down” while missing a running-but-stale stream.
• Promoting a model alias midway through a batch run.
• Rolling out a new stateful stream processor without restore compatibility tests.
• Treating internal exactly-once state guarantees as automatic exactly-once external notifications.
• Measuring batch job completion without verifying the consumer-visible committed output.
• Equating online uptime with successful model responses and valid feature values.
• Assuming CPU/GPU replicas alone solve hot-key or remote-service bottlenecks.
• Calculating cost from worker runtime but ignoring broker, state, idle capacity and storage.
• Failing to agree on who owns an incident when batch, stream and online outputs are combined.

In short: Operate online inference as a low-latency service, batch inference as a deadline-bound data product, and streaming inference as a continuously stateful event-processing system. Keep model identity and business correctness consistent, but tailor release management, incident recovery, monitoring and cost controls to each operating mode.