What is the difference between CPU, GPU, and accelerator scheduling for ML workloads?

Scheduling means deciding which machine should run an ML job based on the resources that job needs and what is currently available.

CPU, GPU, and other accelerators have different scheduling rules. A scheduler that only checks free CPU and memory may still be unable to place a job that requires a particular GPU or TPU.

1. CPU scheduling: flexible and shareable

CPUs are general-purpose processors. They work well for data cleaning, feature processing, orchestration, and many smaller ML models.

In Kubernetes, a job can request a fraction of a CPU. For example:

• cpu: "500m" means a request for 0.5 CPU.
• cpu: "2" means a request for 2 CPUs.

Kubernetes uses CPU *requests* when choosing a suitable node. CPU *limits*, if configured, cap the CPU time a container can use. Depending on the node's current workload and limits, multiple CPU-based jobs can share available compute.

Example: Two preprocessing jobs that each request 500m account for a total request of 1 CPU.

2. GPU scheduling: device availability matters

GPUs accelerate parallel numerical work such as deep-learning training and some forms of inference. But requesting 1 GPU is different from requesting 1 CPU.

Kubernetes typically discovers GPUs through a device plugin, which advertises a schedulable resource such as nvidia.com/gpu. The job asks for a discrete number of GPU resources.

A simplified container resource specification is:

YAML
resources:
  requests:
    cpu: "2"
    memory: "8Gi"
  limits:
    nvidia.com/gpu: 1

Here the workload asks for 2 CPUs, 8 GiB of RAM, and 1 GPU. With the standard Kubernetes GPU resource model, GPU limits also establish the GPU request. The GPU plugin, compatible drivers, and suitable GPU nodes must already be configured.

GPUs are normally allocated as whole advertised devices, even if the program does not fully use the device's compute power or memory. GPU memory requirements and framework compatibility must also be checked; getting a GPU allocation does not guarantee that a large model will fit in memory.

3. A numerical scheduling example

Suppose a cluster has one worker node with:

• 8 allocatable CPUs
• 32 GiB allocatable RAM
• 2 advertised GPUs

A training job requests 2 CPUs, 8 GiB RAM, and 1 GPU.

• Job A: Runs and receives one GPU.
• Job B: Runs and receives the other GPU.
• Job C: Also needs one GPU but cannot be placed on this node.

After A and B, the node still has 4 CPUs and 16 GiB RAM unrequested, yet Job C remains Pending because no GPU resource is available.

A CPU-only preprocessing job needing 0.5 CPU and 2 GiB RAM may still fit, subject to other scheduling constraints.

This is why monitoring *only CPU utilization* can be misleading in an ML cluster.

4. Can multiple ML jobs share one GPU?

Sometimes, but GPU sharing requires explicit support and configuration.

• Time-slicing: Supported NVIDIA configurations allow several workloads to take turns using one physical GPU. They share the underlying device; time-sliced allocations do not provide separate GPU memory or fault isolation.
• Multi-Instance GPU (MIG): On supported NVIDIA GPUs, a physical GPU can be divided into hardware-isolated instances with defined resources. The scheduler can allocate the advertised instances to different jobs.

These approaches are not identical, and their resource names and capabilities depend on the device plugin and platform. Do not assume that requesting `nvidia.com/gpu: 0.5` works: ordinary Kubernetes extended resources are not requested as fractional devices.

5. What about TPUs and other accelerators?

A TPU is a specialized accelerator often used for large tensor computations. In supported Google Kubernetes Engine configurations, TPU resources can be exposed as google.com/tpu.

TPU placement may require a specific chip count, compatible slice type, and topology. A multi-chip training job might need a coordinated set of devices rather than just any single available accelerator.

Other accelerator families can have their own device plugins, supported shapes, memory constraints, and scheduling requirements.

6. How should an MLOps team choose a scheduling approach?

• CPU jobs: Use requests for modest data-processing and control tasks. Avoid reserving expensive accelerators when the workload cannot benefit from them.
• GPU jobs: Request the correct GPU type and count; verify model memory needs, drivers, and scheduling availability. Consider sharing only when isolation and performance requirements allow it.
• TPU or specialized accelerator jobs: Match framework support, chip topology, and the platform's capacity model.
• Mixed pipelines: Run CPU-heavy ingestion and preprocessing separately from GPU-heavy training or inference where practical.
• Production planning: Monitor job queue time, CPU and accelerator utilization, device memory, throughput, and cost. Node autoscaling or reservations may help when jobs wait for scarce accelerators.

Scheduling is not the same as executing kernels on the GPU. The cluster scheduler chooses *where a workload runs*; the accelerator runtime schedules the lower-level computations once the workload starts.

In short: CPUs are generally allocated in flexible, shareable quantities, while GPUs and specialized accelerators require compatible device resources and often have stricter placement constraints. Good MLOps scheduling matches the workload to the right hardware without leaving expensive devices idle or jobs stuck Pending.