48 GB or two 24 GB GPUs for inference?

The choice depends on how the model will run, not on the sum of the gigabytes printed on the boxes. One 48 GB card gives the model a single continuous memory budget and requires no communication between accelerators. Two 24 GB cards provide either two independent service replicas or one distributed instance that pays for model partitioning and synchronization.

At the same price, I usually choose 48 GB for one large model, a long context, and predictable latency. Two 24 GB cards win when every production model fits in 24 GB and the job calls for more independent requests, different models, or separate failure domains. If a model does not fit in 24 GB but does fit in 48 GB, the pair remains viable, but it now pays for every generated token with communication between devices.

Why 24 plus 24 does not become one memory pool

Each graphics card has its own physical VRAM array and address space. CUDA can give one card direct access to another card's memory through peer-to-peer access, while unified virtual addressing simplifies pointer handling. This does not create a shared 48 GB buffer where any model operator can place a tensor of arbitrary size.

The difference appears during loading. If one layer or temporary work buffer needs more free memory than one card has, spare gigabytes on the neighboring card do not rescue the process by themselves. The framework must know how to split that layer, place different layers on different devices, or offload some data to system RAM. Each method has its own constraints.

The NVIDIA CUDA Programming Guide describes multi-GPU work as separate management of devices, contexts, compute distribution, and result exchange. It also says peer access depends on the PCIe or NVLink topology and must be checked for a specific pair of devices. The practical conclusion is blunt: you may count aggregate capacity, but you must plan around the tightest shard.

Suppose quantized weights occupy 38 GB. One 48 GB card has roughly 10 GB left after loading, before runtime overhead. A good split across two cards puts about 19 GB of weights on each and leaves about 5 GB per card. A poor 22 GB plus 16 GB split leaves only about 2 GB on the first card. The process will hit OOM on that card while the second still reports free memory.

The phrase "48 GB in aggregate" therefore describes the ceiling of a distributed configuration, but it does not promise the behavior of one 48 GB card. These are different resources with the same number on the invoice.

Model weights consume only the first part of the budget

Inference memory planning must include weights, the KV cache, temporary tensors, kernel workspaces, and the runtime's own reserve. A bad purchase often starts by multiplying the parameter count by the data type size and declaring that "the model fits with room to spare." Production requests consume that room quickly.

A rough lower estimate for weights looks like this:

BF16 или FP16: параметры × 2 байта
INT8:          параметры × 1 байт + масштабы и метаданные
4 бита:        параметры × 0,5 байта + масштабы, группы и служебные данные

For a 32 billion parameter model, bare weights require about 64 GB in BF16, 32 GB with 8-bit storage, and 16 GB with 4-bit storage. The last two numbers are not total consumption figures. Quantization formats store scales and sometimes zero points, some layers may remain at higher precision, and the loader creates buffers. Checkpoint file size also does not equal peak VRAM during startup.

The second large consumer is the KV cache. An autoregressive model stores attention keys and values for tokens it has already processed, so it does not recompute the whole context for every next token. The Transformers documentation explicitly warns that long context can make the KV cache a significant part of memory. It grows with the number and length of concurrent sequences. Model architecture, the number of KV heads, and the cache data type change the cost per token, so there is no universal "gigabytes per 8K" figure.

Short-lived peaks matter too. Attention, dequantization, or a different kernel choice may request a workspace just when the monitor shows an almost full card. A dependable configuration should not live at 100 percent. For the first estimate, I leave 10-20 percent for the runtime and peaks, then replace that margin with a measured value. This is an engineering allowance, not a physical constant.

By default, vLLM determines KV-cache memory after loading the model and prints its token capacity together with an estimate of maximum concurrency. Those two lines are more useful than one nvidia-smi snapshot: they show how many real sequences the server can sustain at the configured max_model_len.

Which model classes fit each option

One 48 GB card and two 24 GB cards can hold similar classes of weights, but they leave different cache headroom and impose different launch conditions. The table below is for initial screening, not a compatibility promise for a specific checkpoint.

I have intentionally left model names out of the table. Two models with the same advertised parameter count can use memory differently because of grouped-query attention, vocabulary size, tied embeddings, local attention, mixture-of-experts, and the quantization implementation. A MoE model's total parameter count is especially easy to misread: only some experts are active for each token, but all deployed expert weights still need storage somewhere.

The 70B class at 4 bits shows the choice most clearly. Bare arithmetic promises roughly 35 GB of weights. The actual format can push the footprint noticeably higher, while context and concurrent requests demand the remaining space. One 48 GB card gives you one shared remainder. On a pair of 24 GB cards, each shard must fit alongside its part of the KV cache and temporary buffers.

For a 32B model at 4 bits, one 24 GB card often looks sufficient, yet service capacity may be weak because little space remains for long conversations. Two cards can then run as two short-context replicas or as one distributed instance with a larger aggregate cache. The queue profile, not the model name, determines the better choice.

The partitioning method changes the outcome

For one model to use two cards, the runtime must apply tensor parallelism, pipeline parallelism, or simple layer placement across devices. These modes solve different problems and create different traffic.

With tensor parallelism, matrices inside a layer are divided between cards. Both cards work on the same token, then exchange partial results through collective operations. Weight memory and part of the cache are distributed, but synchronization happens many times during one pass. A fast interconnect is especially useful here.

With pipeline parallelism, one card stores the early layers and the other stores later layers. Activations cross the boundary between stages. Communication may be less frequent and smaller, but an uneven layer split leaves one card full while the other sits idle. For a single autoregressive request, a pipeline also does not turn two cards into twice the compute: its stages depend on each other.

A basic device_map="auto" in Transformers can spread weights across available devices and continue onto CPU or disk when space runs out. This is useful for getting a model to start during an evaluation, but it does not prove good production inference. Moving layers through system memory can raise latency sharply even after OOM disappears.

The vLLM documentation gives a sensible rule: if a model fits on one GPU, distributed inference is usually unnecessary; if it does not fit but can fit in one multi-GPU node, use tensor parallelism. The same documentation suggests considering pipeline parallelism when NVLink is absent and the model divides unevenly, because it can reduce communication overhead. I agree with the direction, but I do not accept it as a substitute for a test: the exact architecture and batch size can easily change the winner.

Data parallelism solves a different problem. Each card stores a full copy of the model and handles requests independently. Memory does not combine at all, but with enough queued work, aggregate throughput approaches the sum of two replicas. For a model that comfortably fits in 24 GB, this is often the best use of the second card.

The interconnect determines the two-card penalty

Card-to-card communication speed depends on direct peer-to-peer access, the PCIe bridges along the path, and whether the specific pair supports NVLink. The names of two identical GPUs say nothing about the topology of the finished server. Slots may hang off different CPU sockets, BIOS settings may change their modes, and virtualization may block direct access.

Run the checks on the assembled machine:

nvidia-smi topo -m
nvidia-smi topo -p2p r
nvidia-smi topo -p2p w

The topo -m matrix shows GPU0 and GPU1 as rows and columns, together with the path type between them. In NVIDIA's documentation, PIX means a path through at most one PCIe bridge, PHB passes through a PCIe host bridge, SYS also crosses an inter-socket link, and NV# indicates a bonded set of NVLinks. The topo -p2p commands separately report direct read and write capability.

        GPU0  GPU1  CPU Affinity
GPU0     X    PHB   0-15
GPU1    PHB    X    0-15

This output does not measure actual bandwidth. It describes the route. Follow it with an NCCL or nvbandwidth test on the same operating system, driver, container, and IOMMU settings that production will use. NVIDIA DCGM also includes a PCIe test that checks P2P, errors, replays, and measures GPU-to-GPU and host communication.

Over PCIe, tensor parallelism can be perfectly acceptable for large batches where computation overlaps some communication. During interactive generation with a small batch, synchronization on every token is more visible in latency. NVLink reduces the penalty, but it does not make distributed execution free. If the cards lack direct P2P altogether, traffic may pass through CPU memory, which makes one 48 GB card even more attractive.

Latency and throughput call for different purchases

One 48 GB card usually provides better single-request latency when the compared accelerators have similar compute performance and local memory bandwidth. The entire model pass stays on one device, there are no collective operations, and the scheduler can use the remaining VRAM more freely.

Two 24 GB cards can deliver more requests per second in two ways. If the model fits on each card, two replicas handle different requests with no card-to-card traffic. If the model is partitioned, total compute capacity is higher, but the gain depends on batch size and communication speed. For one user, tensor parallelism sometimes speeds up a large model, sometimes barely changes speed, and can slow it down on a weak topology.

Keep three metrics separate:

• TTFT, time to first token, depends heavily on input length and prefill;
• TPOT, time per output token, describes interactive decode speed;
• total throughput in tokens or requests per second at the target concurrency.

A purchase for chat with a strict latency limit rarely matches a purchase for overnight batch processing. For chat, p95 TTFT and TPOT under the real queue matter. Batch work can tolerate more latency for one job if two cards hold a larger batch and complete the entire set sooner.

Failure behavior also matters. If either card fails in a tensor-parallel pair, the whole model instance stops. Two independent 24 GB replicas allow you to remove one from service and retain part of the capacity. One 48 GB card is simpler, but it is one accelerator and one instance. Cover that risk at the node level instead of pretending that two cards inside one process already provide redundancy.

One 48 GB card beats a complicated pair

Choose 48 GB if the production checkpoint and required context fit there but do not fit in 24 GB. This is the cleanest case: you avoid sharding, card-to-card communication, and a separate class of distributed runtime failures.

One card also makes more sense under these conditions:

• the interactive service runs at batch size 1 or low concurrency;
• you need long context and one large reserve for the KV cache;
• the framework poorly supports multi-GPU execution for the selected architecture or quantization;
• the server has a weak PCIe topology or direct P2P is unavailable;
• the team wants simpler model updates, OOM profiling, and environment reproduction.

The last point is often underestimated. A distributed failure may depend on the NCCL version, device order, container permissions, an incompatible kernel, tensor shape, or memory imbalance. On one card, the search area is smaller. For a small team, that affects ownership cost as much as a few percent of performance.

Having 48 GB also gives you freedom to change quantization for quality. A model that barely fits in 24 GB at an aggressive 4 bits may fit in 48 GB at 8 bits or with more layers in BF16. Better quality is not guaranteed for every model and task, so test it on your own evaluation set, but hardware headroom makes that test possible.

The argument against one large card is the cost of idle capacity. If the service is nearly empty at night, a second independent 24 GB card could run another model, embeddings, recognition, or a batch job. You can co-locate these jobs on 48 GB too, but competing processes share one accelerator and may disturb the main service's latency.

Two 24 GB cards pay off with the right queue

A 24 GB pair wins when each model already fits on one card. There is no reason to combine memory: launch two replicas and let the load balancer send them independent requests. For 7-8B in BF16, 13-14B in INT8, and many 30-32B models at 4 bits, this is practical once cache headroom has been checked.

Two cards are also more convenient for multiple models. One can serve generation while the other holds an embedding model or a separate specialized LLM. Switching workloads does not require unloading tens of gigabytes of weights before every request type. Memory isolation also makes behavior easier to understand, although shared CPU, RAM, and PCIe can still become bottlenecks.

There is a financial caveat. Equal purchase price does not mean equal node cost. Two cards consume more PCIe slots, need enough PCIe lanes, may require a more capable CPU, and demand more power cabling and cooling. The power supply must handle both cards' peaks rather than their average draw. The chassis must move air past adjacent accelerators without thermal starvation.

Licensing and support may also count GPUs, sockets, or instances. I do not infer those charges from memory capacity; request them for the chosen stack. On the other hand, two common 24 GB cards can sometimes be replaced individually more easily than one specialized 48 GB card. That depends on actual availability and delivery terms, so a price list without lead times settles little.

A distributed 70B model at 4 bits remains a sound reason to buy two 24 GB cards when a single 48 GB option is unavailable or when the pair is clearly faster at your batch size. But the pair cannot be treated as equivalent to one card. First check memory balance across ranks, then TTFT, TPOT, throughput, node power, and long-context behavior.

The test run must reproduce production

Make the purchase decision with the same checkpoint, engine, quantization type, and request set on both configurations. Generating 20 tokens from a short prompt hides the KV cache, queue behavior, and thermal limits.

I use this test sequence:

1. Load the exact model build and record the driver, CUDA, engine, and quantizer versions.
2. Run short, median, and maximum contexts at target concurrency, recording p50 and p95 for TTFT and TPOT.
3. Increase concurrent requests to the target level while watching KV-cache capacity, preemption, and OOM separately on each card.
4. Repeat a long run to expose throttling, node power, and queue stability.
5. For two cards, repeat with tensor parallelism, pipeline parallelism, and two replicas wherever each mode applies.

When starting vLLM, save the initial lines in this form:

GPU KV cache size: 643,232 tokens
Maximum concurrency for 40,960 tokens per request: 15.70x

This is the output shape shown in the vLLM documentation, not a promise for your machine. Compare estimated concurrency with observed concurrency: long and short requests mix, and the scheduler consumes cache blocks differently from a perfectly uniform test. If you use a static cache, remember that Transformers reserves the maximum in advance and may spend memory on masked positions. If you offload the cache to CPU, its documentation warns of lower throughput due to transfers.

Record memory for every rank rather than an average. In a distributed run, one card reaching OOM is enough to fail a request or the whole process. Test cold loading too: format conversion, CUDA Graph capture, and kernel compilation can create a peak that never appears on the steady-state chart.

GSE.kz designs and integrates AI and data center infrastructure with a vendor-neutral approach, so a configuration can follow the model, queue, and delivery requirements instead of one accelerator brand. This work needs results from the same test on every candidate, or the integrator will also be choosing by nominal gigabytes.

The operating mode decides at an equal price

If one model instance needs more than 24 GB but fits comfortably in 48 GB with its cache, buy one 48 GB card. It removes the communication penalty and leaves one continuous memory reserve. For interactive inference, this is my default choice.

If the model fits in 24 GB and the queue is large enough, two cards are better as two replicas. You get extra throughput without tensor parallelism and can continue serving some traffic when one replica stops. If you need different models at the same time, the case for the pair becomes stronger.

If a 70B model at 4 bits needs the aggregate 48 GB, the specification cannot provide the answer. On one card, check the actual KV-cache headroom. On two, check shard balance and topology, then compare p95 latency and throughput. The winning configuration sustains your maximum context and queue without OOM, throttling, or CPU transfers.

There is a useful boundary for revisiting the decision. If the cache space left after loading weights is smaller than the target queue requires, move to the next memory class even when a single-request test passes. Reducing max_model_len or request count to get through a demonstration only conceals the shortage. Buy the server for peak production load, not for the most convenient prompt. Likewise, do not assume two cards will accelerate the model: when GPUs spend most of their time waiting for a collective operation, extra compute does not become useful tokens.

Do not buy memory sized exactly to the weights file. Fix the acceptable quantization type, context length, concurrent request count, and latency limit first, then assign VRAM. Memory on two cards combines only to the extent that the selected engine can make it combine. One 48 GB card provides that capacity without the qualification.