What is a streaming inference architecture?

A streaming inference architecture continuously consumes events, prepares features, applies a deployed machine-learning model, and publishes predictions with low end-to-end delay. Unlike a request-driven asynchronous API, the system operates on an ongoing event stream rather than creating a separately polled job for each client request. Unlike scheduled batch scoring, it does not wait for a complete daily dataset before processing.

Question 117 covered HTTP 202, request IDs and queued *individual prediction jobs*. Questions 115–116 covered serving-mode selection and batch job integration. This lesson focuses on unbounded streams, event-time correctness, state, throughput, scaling, backpressure and fault recovery.

Beginner-friendly example: real-time delivery-delay alerts

A food-delivery company continuously receives GPS updates, delivery-status events and traffic alerts. It needs to estimate whether active orders will be late and notify operations while there is still time to intervene.

A typical event looks like:

JSON
{
  "event_id": "gps-78221",
  "delivery_id": "order-482",
  "event_time": "2026-11-08T10:00:12Z",
  "driver_lat": 12.9716,
  "driver_lon": 77.5946
}

The stream processor consumes the event, joins it with the latest *valid* order state, derives features such as recent travel speed, invokes a pinned delay-risk model, and emits an alert when P(late) > 0.80. A downstream consumer then updates a dashboard or triggers an appropriate alert. Avoid storing unnecessary location data in logs, and enforce retention and access restrictions.

Core flow:

Driver GPS / order status / traffic producers
                    |
              Kafka topics
                    |
       Deserialize + validate + deduplicate
                    |
      Key by delivery_id / route / region
                    |
   Enrich + maintain windowed feature state
                    |
           Load/pin model version
                    |
              Predict risk
                    |
       Emit predictions / risk alerts
                    |
        Kafka topic / transactional sink
                    |
       Dashboard / alerting / warehouse

Kafka is a durable event-log broker; Apache Flink or Kafka Streams can transform, join and aggregate incoming events; inference may execute inside the stream processor or call a separately scaled model-serving service.

Stateless and stateful inference

Stateless inference: Every event contains all necessary features. Example: classify each incoming text event using its content and language. The prediction itself needs no memory of previous events, although deduplication and output handling may still be stateful.

Stateful inference: A prediction depends on multiple events for the same entity. Example: compute driver speed over the last five minutes, delivery distance remaining or the number of traffic alerts over the last ten minutes. A keyed state store maintains these aggregates and joins them to the new event.

Be explicit about what is stateful:

• Feature state: Sliding-window counts, recent positions, event-time joins and last valid status.
• Model state: Loaded model weights and model version, sometimes updated through a controlled model-release mechanism.
• Delivery state: Input offsets, processed event IDs, committed output versions and checkpoint metadata.

Don't confuse stateful stream processing with recurrent or stateful ML models. A conventional gradient-boosted model can consume features produced by a stateful stream processor.

Event time versus processing time

Event time is when the underlying business event occurred. Processing time is when a system actually handles it. Network interruptions and device buffering can make them different.

Suppose GPS event A happened at 10:00:05 but arrived at 10:00:20. Event B happened at 10:00:15 but arrived at 10:00:16.

Arrival order: B, A. Event-time order: A, B.

If a five-minute speed feature relies on event order, processing the later-arriving A as though it happened after B could produce incorrect state. Keyed event-time windows, timers, version-aware updates and late-event handling help preserve historical semantics.

Watermarks are an estimate of event-time progress. A watermark of 10:05:00 indicates the processing system expects no more on-time events at or before that timestamp. It is not an absolute proof that late records will never arrive. Configure a permitted lateness and correction policy.

For a window [10:00,10:05), a watermark passing 10:05 may cause a window result to be emitted. A delayed 10:02 event arriving afterward can be:

• Dropped with an audited late-event count.
• Routed to a side output for investigation.
• Included in a corrected/update result when allowed lateness and sink semantics support it.

An event-time window aligns with the historical event timestamps, while a processing-time window depends on when the worker received or processed records. This matters for feature reproducibility and avoiding leakage.

Window types and inference features

Common patterns:

• Tumbling window: Fixed non-overlapping five-minute buckets; simple aggregation.
• Sliding window: Five-minute window recalculated every minute; smoother signals with more state/compute.
• Session window: Group driver activity until an inactivity gap occurs.
• Per-key running state: Last known position or active delivery status, often using event-time ordering.

For delivery-delay inference, a sliding window could count recent traffic events, average route speeds and aggregate stops. Use time-to-live for state associated with finished deliveries and event-time-aware cleanup rules so state does not grow indefinitely.

Numerical throughput and capacity example

Suppose the platform receives 3,000 GPS events per second at peak. A stream-processing/inference task handles 600 events per second at the required p95 latency under representative testing.

• Theoretical tasks = ceil(3000/600) = 5.
• Target 75% utilization to absorb spikes: effective throughput per task = 600 × 0.75 = 450 events/sec.
• Planned parallel tasks = ceil(3000/450) = 7.

Seven tasks have a nominal capacity of 7 × 600 = 4,200 events/sec, but a 75%-utilization planning capacity of 3,150 events/sec. These are planning estimates, not latency guarantees: hot delivery IDs, network I/O, joins, checkpoint overhead and model-serving calls can create bottlenecks. Test skew and tail latency.

Backlog example: Suppose an incident limits processing to 2,000 events/sec for 10 minutes while arrivals remain 3,000 events/sec.

• Deficit = 3,000 - 2,000 = 1,000 events/sec.
• Backlog growth = 1,000 × 600 = 600,000 events.

After capacity returns, suppose processing can sustain 4,000 events/sec, while new arrivals remain at 3,000/sec. Net drain = 1,000 events/sec, so clearing 600,000 events requires 600,000 / 1,000 = 600 seconds = 10 minutes, ignoring changing traffic, queue overhead and other disturbances.

The system can appear healthy in terms of worker uptime while producing ten-minute-old predictions, which defeats the business goal. Measure event-time-to-prediction latency and consumer lag, not just raw inference execution time.

End-to-end latency budgeting

A realistic latency SLO measures prediction_published_at - event_time, with care for device clock skew and timestamp quality. It consists of:

• Device/broker transport and ingestion delay.
• Queue/broker lag.
• Deserialization, validation and feature enrichment.
• Waiting for event-time windows or late-data tolerance.
• Model execution (local or remote).
• Sink commit and downstream delivery.

Suppose the target is p95 below 2 seconds for alert-producing GPS events. A design that waits five minutes for a window to close cannot meet this target for all events. Instead compute rolling incremental features and emit provisional predictions continuously, correcting them later if required.

Keep separate objectives for event processing latency, watermark lag, inference duration, sink publication latency and end-to-end freshness.

Deploying the model inside or outside the stream engine

Embedded inference: A model is loaded in each processing task. This avoids network calls, but uses task memory and makes model deployment/rescaling more tightly coupled to Flink jobs. Good for compact CPU-friendly models.

External inference service: Stream tasks call a dedicated serving tier. This lets GPU models scale independently, but adds latency, request failures, service quotas and backpressure. Use bounded concurrency, backoff and asynchronous I/O when supported, without dropping event IDs or violating ordering policies.

Microbatch inference: Collect a small bounded set of events for efficient tensor operations, then publish results. Better accelerator utilization may add buffering latency.

For all approaches, pin the model version for a defined processing epoch and include it in predictions. A rolling model upgrade can otherwise mix versions across parallel tasks. Use a controlled barrier, epoch marker or version-stamped routing policy and record transitions.

Backpressure and elasticity

Backpressure happens when downstream processing is slower than upstream production. In a bounded stream-processing pipeline this propagates to earlier operators or increases broker backlog.

Likely causes:

• A slow model server or GPU saturation.
• Expensive window joins or oversized per-key state.
• An overloaded database/output sink.
• Hot partitions, network congestion or checkpoint delays.

Operational responses:

• Scale tasks and partitions when the bottleneck supports it.
• Increase serving capacity or use bounded inference batching.
• Reduce expensive remote feature lookups with permitted caching.
• Restrict concurrency to prevent overwhelming the model server.
• Apply producer rate controls, load shedding or degrade gracefully only under an explicit business policy.
• Track oldest-event lag and stale-prediction counts.

More workers don't help if a single Kafka partition, hot key or external model endpoint imposes a hard throughput bottleneck. Plan sufficient broker partitions and correct keyed-state partitioning.

Fault tolerance and checkpoints

A stream processor such as Flink coordinates checkpoints of:

• Source positions or offsets.
• Keyed operator state and event-time timers.
• In-flight data where supported by the checkpoint mode.
• Sink transaction state or identifiers needed for consistent output.

On worker failure, the job restores the last completed checkpoint and replays events from the recorded source positions. For example, with a 60-second checkpoint interval, a worker that fails 45 seconds after a checkpoint may need to replay approximately 45 seconds of events. Recovery time additionally depends on checkpoint restore and processing speed; the true replay span can be larger depending on completed checkpoint timing.

Streaming checkpoints protect processor state and input position, not necessarily the training optimizer state from Question 108. A reliable inference stream must also retain the pinned model artifact/version and feature-enrichment dependencies needed to process replayed events consistently.

At-least-once versus exactly-once: the important boundary

At-least-once delivery: Input events may be replayed after failure. Without deduplication, a prediction or notification might be emitted twice.

Exactly-once state consistency: A stream engine can restore internal state as though replayed records had been processed once at a logical checkpoint boundary.

End-to-end exactly-once effects: Also requires compatible sink transactions or idempotent writes and correct coordination with the source. An engine-level guarantee does not mean every external HTTP request, email, database insert or business action is automatically executed once.

For example, Kafka transactional producers and exactly-once-capable processors can support consistent consume-process-produce pipelines to Kafka with the correct configuration. But a stream worker that calls an arbitrary SMS API twice after checkpoint replay can still send duplicate messages.

For externally visible risk alerts, use a stable event or decision ID such as delivery_id + source_event_id + model_version + rule_version. Store/deduplicate published decisions, or use a transactional outbox to separate inference from the one-time business action. Define whether an event can legitimately lead to multiple revised predictions.

Checkpoint and sink trade-offs

More frequent checkpoints can reduce replay work, but increase coordination, storage and I/O overhead. Exactly-once transactional sinks may delay visible output until the transaction commits. Unaligned checkpoints can sometimes ease recovery under backpressure but add checkpoint complexity and storage of in-flight data.

For a stream with large state, rescaling and restoring may take minutes even if single-event inference normally takes milliseconds. Measure recovery time objective, checkpoint duration, restore failures, state size and data lag after recovery.

Avoid assuming checkpoint interval alone determines maximum lost work: a failed/incomplete checkpoint and slow barrier alignment change the actual recovery position.

Practical pseudocode (framework-independent)

PYTHON
def on_event(event, keyed_state, model, output):
    # Broker offset and state are managed by the stream runtime.
    verify_event_schema(event)
    if already_processed(event.event_id, keyed_state):
        return

    update_event_time_features(event, keyed_state)
    features = read_current_features(event.delivery_id, keyed_state)

    prediction = model.predict(features)
    output.emit({
        "event_id": event.event_id,
        "delivery_id": event.delivery_id,
        "model_version": model.version,
        "risk_score": float(prediction),
        "event_time": event.event_time,
    })
    mark_processed(event.event_id, keyed_state)

This is illustrative pseudocode, not a directly executable Flink application. An actual Flink/Kafka Streams implementation uses framework-managed keyed state, timers, watermark strategies, appropriate checkpoint configuration and an output sink that satisfies the chosen delivery semantics. The ordering of emit and mark_processed in ordinary application code cannot alone guarantee exactly-once effects after failures.

What to monitor

• Input: Events/sec, broker partition lag, oldest-unprocessed event, schema errors and rejected events.
• Freshness: Event-time lag, watermarks and p50/p95/p99 event-to-prediction latency.
• Operators: Busy/backpressured/idle time, queue depth, hot-key skew, task restarts and resource use.
• State: Key count, state bytes, state TTL expiry, restore duration and checkpoint completion/failure.
• Model: Pinned version, prediction distributions, feature missingness, inference latency, errors and drift after labels arrive.
• Output: Sink publish failures, duplicates, dropped or late events, transactional commit delay and downstream consumption lag.
• SLO: Percentage of valid events with predictions published within deadline, and accuracy/coverage once ground truth is known.

Alert when the stream is running but stale: healthy containers do not imply useful real-time predictions.

Trade-offs and common mistakes

• Stateless processing: Simpler and cheaper but cannot represent recent behavior without external state.
• Stateful feature windows: Better temporal signals, at the cost of memory, checkpoint overhead and late-event semantics.
• Embedded models: Lower network latency, but model loading and GPU needs couple inference to processor lifecycle.
• Remote model serving: Independent scaling and model specialization, but harder to meet strict latency and failure budgets.
• Higher parallelism: More throughput until partition/key skew or external services become the bottleneck.
• Long lateness window: More complete event-time results, but delayed final decisions or more correction events.
• Exactly-once transactional pipelines: Stronger consistency with additional complexity and possible output delays.
• Blind retries of external side effects: Can duplicate alerts even when stream state itself recovers correctly.

In short: Streaming inference is a continuously running event-to-prediction pipeline. Design the broker, stateful feature processing, event-time windows, model execution and output sink as one reliability contract. Scale for peak throughput, measure event-to-prediction freshness, recover from checkpoints, and treat exactly-once promises as end-to-end claims that must be verified at every output boundary.