Sharing a graphics card does not always pay off

Sharing one graphics card makes sense when individual workloads leave the accelerator idle and their combined peak fits within the available memory and latency target. The price of the card proves nothing by itself: savings appear only after you measure simultaneous load, licenses, and operating effort.

I have seen plenty of projects where a team celebrated high average utilization after enabling shared access, then disabled it a week later because p99 jumped, out-of-memory errors appeared, and nobody could tell which container had ruined the night for its neighbors. A good sharing design defines boundaries in advance. A bad one merely lets more processes start on one device.

Find the unused capacity first

Sharing is justified when the graphics card is idle inside the workload cycle, not merely according to a schedule. An inference server may wait for requests, an interactive notebook often occupies memory while using little compute, and a video-processing job alternates between decoding, computation, and writing the result. These profiles complement one another. Two large-model training jobs that already keep compute units and memory busy usually just interfere with each other.

Average GPU Utilization does not describe all free capacity. It does not tell you whether enough video memory remains, whether a workload is limited by memory bandwidth, whether the encoder is busy, or how long the request queues are. Look at a time series with an interval that matches the service: seconds and tail latency matter for an API, while minutes and batch completion time may matter more for batch processing.

Before designing anything, record four values for each workload: maximum video memory in use after warm-up, compute-unit utilization, sensitivity to latency, and acceptable restart behavior. Add a chart of work arrival by hour. If two workloads peak at the same time, add their peaks rather than their averages. If the peaks are separated and the queue can wait, a shared accelerator begins to look sensible.

A simple test of reason applies. Shared mode must produce more useful work per card over the accounting period without breaching the response-time objective or the batch deadline. Raising utilization from 30 to 85 percent is not a success if p99 exceeds the user agreement or the overnight job no longer finishes by morning.

Test a complete work cycle, not a single day. A weekly profile often reveals backup, retraining, index building, or reporting work that coincides with the daytime peak only once a week. That rare overlap still determines required capacity if queueing or reduced priority is unacceptable. If background work can stop and resume without losing progress, record that as an explicit scheduler rule. Hoping that engineers will notice contention manually is not resource management.

Sharing mechanisms solve different problems

CUDA streams, MPS, time slicing, vGPU, and MIG are not alternative settings for the same feature. They create different boundaries for memory, scheduling, failures, and administration.

The simplest option works inside one application: a process launches operations in several CUDA streams while the framework combines requests into batches. This is often the best route for one code owner. The application sees common memory and manages concurrency itself, so it does not pay for virtual machines and separate guest drivers. A CUDA stream does not, however, isolate a fault, memory, or bandwidth from another stream.

NVIDIA Multi-Process Service, or MPS, lets multiple CUDA processes submit work together through an MPS server. NVIDIA's guide describes two practical benefits: kernels and copies from different processes can overlap, and a common set of scheduling resources reduces context switching. This helps trusted batch jobs, MPI processes, and small services owned by one department. MPS supports Linux and QNX, while monitoring tools often attribute usage to the MPS server, which makes per-client accounting harder. It is a cooperation mechanism, not a security boundary between tenants.

Time slicing gives processes, containers, or virtual GPUs access in turns. NVIDIA vGPU schedules time slices: while one virtual machine runs, the others wait. A short slice improves responsiveness but causes more switches; a long slice helps throughput but increases waiting. In Kubernetes, the replicas setting in the NVIDIA device plugin advertises several logical resources over one card. The GPU Operator documentation explicitly warns that these replicas do not get memory or fault isolation, and requesting two replicas does not guarantee twice the compute share.

MIG divides a supported card spatially. An instance receives dedicated compute resources and isolated paths through L2 cache, memory controllers, and DRAM address buses. Separate instances can run in parallel with more predictable latency. A MIG-backed vGPU presents such an instance to a virtual machine; some newer hardware and hypervisor combinations can also time-slice an instance. That extra density is not always useful because neighbors and a queue reappear inside the slice.

Another distinction often disappears in discussion: a quota, isolation, and a performance guarantee are not equivalent. A fixed framebuffer capacity limits a virtual machine's memory without necessarily reserving a constant share of every engine. A logical replica helps the scheduler grant access but does not by itself constrain a process inside the container. A hardware slice provides a stronger resource boundary, yet the physical card, host, and part of the software stack remain shared. In the design document, name the exact property a workload needs instead of using the general word "sharing."

Losses do not come only from context switching

The mechanism's own overhead is often smaller than the damage caused by competition for a shared resource. Two workloads may share a card with almost no loss if one occasionally launches short kernels while the other leaves the device underfilled. The same workloads will collide when both begin reading large tensors from memory or copying data over PCIe at once.

Check four kinds of contention. Compute units determine how many kernels actually execute in parallel. Video-memory bandwidth limits models and analytical operations that read heavily relative to the amount of computation. PCIe and NUMA affect data delivery, especially when CPU processes and the card sit on different sockets. Dedicated decode and encode engines can become the bottleneck in a media pipeline even when GPU Utilization looks modest.

Time slicing adds a queue. If one workload holds the device with a long-running kernel, a neighbor's short request may wait for the next slice or preemption point. Average latency therefore hides the worst effect. For an online service, compare p50, p95, and p99, along with the number of requests that exceed their timeout. For a batch job, measure the full completion time and the variation between runs.

MIG reduces interference between instances, but it does not make a small slice behave like the same fraction of a whole card in every program. A profile assigns discrete compute and memory resources, and an application may scale poorly to that geometry. Kernel compilation, batch size, library workspace, and CPU exchange remain application properties. You cannot divide the result from a whole-device run arithmetically and call it a forecast.

Count useful aggregate throughput instead of looking only at each workload's penalty. Suppose two services on separate cards process 100 and 40 requests per second. Together on one card they handle 92 and 36 at acceptable latency, for a total of 128 instead of 140. You freed a card at the cost of about nine percent of the combined work. That may be worthwhile. If the first service holds 100 while the second falls to 10 because of memory pressure or queueing, high hardware utilization conceals a poor decision.

Video memory creates a hard boundary

Workloads do not share abstract "GPU percentages". They share a fixed memory capacity, where averages are especially dangerous. A model occupies weights, cache, temporary buffers, and library workspaces. After the first request, usage can grow during warm-up, algorithm selection, or KV-cache expansion. A reading taken immediately after startup is almost always too optimistic.

With time slicing, processes usually see one physical memory pool without a guaranteed quota. The scheduler can distribute logical replicas, but it does not turn 48 GB into four independent 12 GB pools. One container can consume the free space, leaving a neighbor with an out-of-memory error. A container's CPU-side memory limit does not solve this problem.

A vGPU generally gives a virtual machine a profile with a fixed framebuffer. MIG also gives an instance the amount of memory defined by its profile. That protects a neighbor's capacity but requires the sizes to be selected in advance. If a model needs 22 GB after warm-up, two 20 GB slices will not help, even if the full card has enough memory. One large and one small profile can sometimes work better than a symmetric partition.

For sizing, use the maximum from a sustained run and add headroom for a larger batch, a new model version, and service overhead. No universal table can supply the right headroom. Derive it experimentally: run a typical peak, then increase batch size or concurrency to the planned limit. Test the out-of-memory behavior separately. A controlled failure of one job may be acceptable; a common process crash, a stuck GPU, or a mass pod restart changes the risk category.

Memory brings an awkward tradeoff. Large profiles reduce density, while small ones cause fragmentation: capacity remains free on the card, but no permitted profile fits the new workload. Repartitioning MIG may require stopping work on the device. The profile catalog should therefore follow actual workload classes instead of an attractive division into equal parts.

Support depends on the card and the entire software stack

The phrase "supports GPU virtualization" is not enough for a purchase. You need to verify the exact model, form factor, driver version, guest OS, hypervisor, license edition, and required profile.

In NVIDIA's current guide, the MIG list begins with the Ampere architecture, but it does not include every card from that generation. Widely used accelerators listed there include the A100 and A30, followed by the H100, H200, and supported Blackwell server models. Instance counts differ: the guide lists up to seven for the A100 and several large accelerators, up to four for the A30, and fewer for some Blackwell workstation models. An architecture name cannot replace the compatibility table for a particular driver release.

A consumer card running two CUDA processes at once has not become a MIG card or an enterprise vGPU. It may support concurrent contexts without hardware slices, supported vGPU profiles, or the required licensing terms. Attempts to build a guaranteed multi-user service on that basis often end with a homegrown dispatcher and a long list of exceptions.

NVIDIA vGPU has a matrix of supported GPUs, hypervisors, guest operating systems, and profiles, and the software is licensed. The terms differ for virtual workstations, compute profiles, and deployment methods. Include license costs, the license server, and support in the calculation before comparing them with the price of a second card.

AMD's SR-IOV mechanism creates Virtual Functions that a hypervisor assigns to virtual machines, but support is likewise tied to server accelerators, the GPU Virtualization driver, and validated combinations of operating system and hypervisor. ROCm documentation lists supported configurations separately. You cannot transfer a conclusion from one Instinct series to every AMD card.

Before ordering, ask the supplier for a confirmed matrix for your exact specification and save it with the project version. Then verify it against the manufacturer's guides. The phrase "supports GPU sharing" without a profile, version, and mode provides no operational guarantee.

Recheck compatibility after every major driver, hypervisor, or guest OS update. A working combination may not retain the same profiles, migration support, or monitoring behavior after an upgrade. Create a small acceptance suite: start every profile, apply simultaneous load, collect metrics, restart a guest, and recover from an error. Run it on one node first, then release the update to the pool. This turns a matrix entry into a tested property of your system rather than a promise in a table.

Four workload types produce savings

The best candidate for sharing has predictable memory use, leaves some compute idle, and tolerates small timing variations. Four scenarios work most often in production.

The first is several small inference models with uneven request traffic. They rarely peak at the same time, and batching requests leaves gaps for neighbors. Processes or MPS may be enough for trusted services with one owner. Separate MIG profiles or vGPU make more sense for different teams and tighter latency requirements.

The second is development environments, research notebooks, and teaching labs. A user holds a context and memory much longer than they actively compute. Time slicing increases availability if the platform ends abandoned sessions and does not promise everyone constant performance. Fair access and queueing matter more here than the smallest penalty for one job.

The third is virtual desktops for graphics or engineering applications. vGPU can give each virtual machine a framebuffer profile and let the hypervisor manage access. The economics depend on licenses, required frame rates, displays, and application certification. Office 3D viewing and demanding interactive rendering are not the same user class.

The fourth is complementary pipelines. Video decoding, preprocessing, inference, and postprocessing stress different parts of the system at different times. Co-location reduces idle time if you measure decode engines, memory, PCIe, and CPU rather than compute units alone.

There can also be an organizational saving: one standardized platform is easier to stock and support than a collection of unrelated workstations. A shared node has a larger failure radius, however. If its outage blocks several services, spare capacity and a migration procedure are mandatory, and their cost belongs to the sharing project.

Some workloads should keep a whole card

Do not share an accelerator when one workload already sustains high compute or memory utilization and its completion time has a cost. Large-model training, scientific computation with heavy transfers, and large batch rendering often gain nothing from a neighbor except timing variation. A scheduler does not create more capacity.

Workloads with a strict p99 are also poor neighbors for time slicing. Even a small background task can arrive at the wrong moment. MIG helps reserve resources, but the small profile must sustain the service peak on its own. If that requires almost the entire card, there is nothing meaningful left to share.

Untrusted tenants need a stronger boundary than MPS or ordinary containers with time-sliced access. NVIDIA GPU Operator documentation explicitly contrasts time slicing with MIG's hardware isolation of memory and faults. For separate legal entities, sensitive data, or strict separation of duties, use supported virtualization and hardware boundaries, then assess the threat model separately. The word "container" does not prove GPU isolation.

Do not mix a production service with an experiment that may compile an unexpected kernel, occupy all memory, or trigger a device reset. A separate MIG instance reduces interference but does not remove shared server components, power, the driver, or maintenance windows. Critical services must survive the loss of the whole physical card.

The popular recommendation to "enable eight replicas because average utilization is low" is wrong. A Kubernetes replica count tells the scheduler how many consumers it may admit, but it does not reserve one eighth of compute or memory for each. Start with the number of profiles that a load test proves can run safely at once, not the density you would like to claim.

A pilot must reproduce contention

A test with two quiet demonstration jobs proves nothing. The pilot must simultaneously reproduce the production peak, the background batch, memory growth, and one participant's failure.

First, establish a baseline for each workload on the whole card. Record throughput, p95 and p99, total batch time, maximum video memory, power consumption, and errors. Then repeat the same set together without changing model versions or input size. Otherwise, you are comparing different experiments.

For quick diagnostics on NVIDIA, the real nvidia-smi dmon stream is useful. The utility guide says that the u group reports utilization for SM, memory, encoder, decoder, JPEG, and OFA, while the m group reports framebuffer and BAR1:

nvidia-smi dmon -i 0 -s pucvm -d 1

Expect one line per interval with a GPU identifier and columns for power, temperature, utilization, clocks, and memory. It is not a kernel tracer and cannot assign blame perfectly, but it quickly shows which events coincided. For continuous monitoring, DCGM Exporter publishes metrics including DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, DCGM_FI_PROF_DRAM_ACTIVE, PCIe counters, and XID errors. The DCGM guide warns that field availability depends on the GPU, driver, and MIG mode.

If you are testing time slicing in Kubernetes, make the shared resource visible in its name. This fragment follows the NVIDIA device plugin format and does not create memory isolation:

version: v1
sharing:
  timeSlicing:
    renameByDefault: true
    failRequestsGreaterThanOne: true
    resources:
      - name: nvidia.com/gpu
        replicas: 2

renameByDefault helps distinguish a shared resource, while failRequestsGreaterThanOne blocks the false expectation that requesting several replicas provides proportionally more power. After applying it, inspect the node's labels, Capacity, and Allocatable values, then run the conflicting profiles.

A failure test is mandatory. Drive one participant into a controlled memory error, terminate its process, restart the container, and inspect the neighbors. Record what monitoring shows and how long recovery takes. If the team cannot associate degradation with an owner and release the device quickly, high density will turn into overnight manual investigations.

Operating economics should decide

Do not compare the price of one card with the price of two. Compare the annual cost of two workable designs. For a shared accelerator, add the card and server, vGPU licenses where applicable, integration work, monitoring, failure reserve, repartitioning windows, and on-call engineering time. For the separate design, include extra cards, slots, power, cooling, network capacity, and idle time on each card.

The calculation needs three pilot results: how much useful work the shared node performs, how many service-objective breaches it creates, and how many support hours it needs. Convert breaches and labor into money using your own internal rates. A precise model with honestly stated unknowns is more useful than a spreadsheet that prices hardware to the last tenge while treating engineering work as free.

Write down an admission rule. For example, co-location is allowed when every workload fits its assigned memory with headroom, the combined run preserves target p99 and the batch deadline, one participant's failure does not require restarting all of them, and annual savings remain positive after licenses and spare capacity. Such a rule survives a card replacement better than a list of favorite profiles.

Assign an owner for capacity. That person should approve new workload classes, retain baseline results, and decide which workload gets constrained during contention. Without this role, every team optimizes its own completion time while the shared node gradually receives more promises than it can keep. A simple record for each profile is useful: owner, model and version, maximum memory, priority, permitted hours, p99 target or batch deadline, preemption behavior, and an on-call contact.

Review the decision when the model or request traffic changes. A successful configuration does not become a permanent server property because a longer context, a new input size, or another algorithm can consume the former headroom. A monthly report comparing the observed peak with the admitted limit is usually enough for a stable system. A fast-growing service needs an automatic warning before memory fills or tail latency breaches its target. If keeping the same density requires weekly batch reductions and schedule moves, the operating effort has already consumed the savings.

Do not forget the cost of maintenance downtime. An update on a dedicated card affects one workload, while an update on a shared card requires a window from several owners at once. Coordination, checks after restart, and a possible rollback take time even when the driver installation itself is short. Include the number of planned updates and the cost of a common window in the model. If services follow different calendars or belong to separate departments, coordination can cost more than an idle second card. This expense rarely appears in a hardware specification, but it regularly appears in the change log.

When designing AI and data-center infrastructure, GSE can match a server platform, NVIDIA or AMD accelerators, and a support design to measured workload profiles. The selected mode must still rest on the customer's pilot because hardware compatibility does not prove savings.

If the measurements show no clear gain, leave the whole card to one workload. A simple design with visible headroom is often cheaper than a dense configuration that engineers must constantly explain, retune, and rescue.