Buy virtualization nodes for failure, not forecasts

Buying every node for five years of growth on day one is usually expensive, but postponing the resilience itself is dangerous. I separate those decisions: the cluster must survive the agreed failure from its first day, while capacity for future load can arrive later if the architecture, licenses, and procurement plan already allow for expansion.

You cannot derive the node count from a single vCPU total. You need measured CPU and memory use, the storage profile, the size of the largest virtual machine, delivery time, license cost, and the acceptable period of degraded operation. Once those inputs exist, "three or four?" becomes a financial model that anyone can check instead of an argument between two administrators.

Buy resilience now and growth as demand appears

The first purchase must include enough nodes for the cluster to carry the current peak after one of them fails, without breaking the agreed limits. Capacity for organic future growth can stay in phase two. If three nodes carry today's load only when all three are healthy, you do not have a cluster with reserve. You have three dependent servers.

First, define the failure you promise to survive. In a conventional cluster in one server room, it is often the loss of one compute node, or N+1. Across two racks with independent power, the design event may be the loss of an entire rack. Across sites, the event is a site failure, and one more server does not solve it. Each scenario removes a different combination of processors, memory, network ports, and data copies at the same time.

Do not mix failure reserve with growth reserve. The first answers whether the system will keep working after a failure today. The second says when expansion becomes necessary as the business grows. If both disappear inside a vague "30 percent spare" figure, nobody knows which part new workloads may consume. A year later the growth allowance is gone, yet the administrator still treats it as emergency capacity.

I use three separate figures:

• failure capacity that planned workloads cannot consume;
• operating headroom for peaks, migrations, and background work;
• growth capacity with a date or a measurable purchase trigger.

Two nodes with an external witness can form a valid cluster in some stacks, but that does not make them a comfortable base for growth. During maintenance on one node, the other carries everything, and the next fault stops the service. The Proxmox VE Administration Guide says directly that reliable quorum for high availability needs at least three nodes. Microsoft permits two-node failover clusters but recommends a quorum witness, which provides the deciding vote and prevents both sides of a partition from operating independently. The two statements do not conflict. Quorum decides which side may run, while capacity sizing decides whether it has enough hardware to run on.

N+1 does not cover planned maintenance and one more failure at the same time if the full service must continue through both events. That commitment needs N+2 or an explicitly accepted window of higher risk. I do not buy a second spare automatically. First, I check whether maintenance can move to low-load hours, noncritical jobs can pause, and the node can return quickly. But if an update takes a day, the site is remote, and downtime is prohibited, the calculation must remove two nodes and repeat every placement check. State separately whether the SLA applies to normal operation, one failure, and maintenance mode. Otherwise the supplier sizes N+1, operations expects N+2, and the team discovers the difference during its first firmware update.

Size the load after a failure, not during normal operation

Run the calculation on the nodes that remain after the selected failure. Average utilization across every host on an ordinary Tuesday tells you little. You need an observed peak over a representative period, recorded separately for CPU, memory actually in use, storage operations and latency, network traffic, and hypervisor overhead.

For a uniform cluster that must survive one node failure, the basic check looks like this:

CPU_after_failure = (nodes - 1) × physical_cores_per_node × target_cpu_utilization
RAM_after_failure = (nodes - 1) × (installed_ram - host_reserve) × target_ram_utilization
fit = workload_peak_cpu <= CPU_after_failure
   and workload_peak_ram <= RAM_after_failure

Do not copy target_cpu_utilization from somebody else's slide deck. Choose the limit at which your applications still meet their latency target and batch window during an incident. One cluster may need 65 percent, while another can use 80. The figure must come from a load test or metric history, not a universal rule.

Do not put the sum of assigned vCPU into workload_peak_cpu. A VM with 16 vCPU may use two physical cores for most of the day, while another with four vCPU keeps all four busy. The vCPU total matters for scheduler and license-minimum checks, but physical capacity must be compared with actual processor time and latency requirements. If you have no metrics yet, run a pilot and use a more conservative limit, clearly marking it as an assumption.

Teams make the opposite mistake with memory: they expect the same high oversubscription ratio as CPU. Active memory, guest OS reserve, huge pages, NUMA, and guarantees for critical VMs do not reduce to installed capacity. If the platform uses dynamic memory, take the peak actual allocation plus proven headroom. Do not count deduplication or compression as emergency capacity until you have measured it on your own data.

Check storage in two planes. The first covers usable capacity after replication, a disk or node failure, and the free space required for recovery. The second covers performance during a rebuild. A cluster can have enough terabytes yet exceed its latency limit while surviving disks serve VMs and restore data copies at the same time. The same applies to networking: two empty switch ports do not help if one path failure forces migration, client traffic, and replication through one saturated link.

Collect metrics at an interval that can see short peaks. A five-minute average may hide a one-minute CPU queue or latency burst that users already notice. An interval that is too fine creates noise and can make you buy a server for one isolated outlier. Keep the raw series, calculate several aggregation intervals, and match peaks to the job log. One month is not enough for a seasonal business: capture year close, admissions, mass payments, or whichever period sets your maximum. If you cannot wait for that season, record the unproven part as a separate risk and schedule an early review instead of presenting incomplete data as a precise forecast.

One spare node does not reserve every resource

The N+1 label describes a node count, but it guarantees nothing about a specific workload placement. Four conditions must hold together after a failure: aggregate capacity, room for the largest VM, placement rules, and access to data. If any one fails, the arithmetic reserve is useless.

Consider a cluster where one node has 1 TB of memory and the other three have 512 GB each. If two 420 GB VMs run on the large node, none of the smaller nodes can accept even one with the same reserve. Total free memory across the cluster may look sufficient, but a VM cannot assemble its RAM from three hosts. The fix is not another percentage of spare capacity. Use equal-sized nodes, reduce the VM, or define an explicit recovery profile for it.

Anti-affinity rules also consume capacity. You cannot place two domain controllers, database replicas, or application instances on one surviving node after an incident simply because it has free memory. Check the placement after failure, not only the resource sum. A useful test temporarily prevents the scheduler from using one host and shows where every protected VM actually lands.

In a hyperconverged system, compute reserve is tied even more tightly to storage. Removing a node takes away CPU, RAM, disks, and a path to some of the data at once. Redundancy repair then starts and raises load on the survivors. The N+1 calculation must include the state where "one node is gone and data is rebuilding," rather than only the quiet ten minutes after the VMs restart.

There is also a failure domain that the server specification does not show. Four nodes on one power distribution panel do not survive that panel's failure. Two network adapters connected to one switch do not provide independent paths. Before adding a fifth server, inspect power, racks, switches, storage controllers, the quorum witness, and backup placement. Sometimes a second path deserves the money more than another processor.

Finally, define the operating mode after failure. "The service keeps working" might mean full peak performance, temporary limits on background jobs, or only the critical service tier. I accept managed degradation when the service owner has approved the exact jobs that can stop and the maximum duration. A silent expectation that users will somehow tolerate the slowdown is not a capacity plan.

Four nodes are necessary when three work only while healthy

A numerical example quickly separates emergency reserve from an attractive dashboard. Suppose measurements show a workload peak of 58 equivalent physical cores and 960 GB of memory actually allocated. This is a hypothetical design example, not a standard workload profile. Each node has 32 physical cores and 512 GB of RAM; reserve 32 GB for the host, limit post-failure CPU utilization to 70 percent, and limit host memory use to 80 percent of the remainder.

One available node provides 22.4 planning cores and 384 GB of working memory. After a failure, a three-node cluster has two survivors: 44.8 cores and 768 GB. It fails both the CPU and RAM checks. A four-node cluster leaves three survivors after the same failure: 67.2 cores and 1,152 GB. It passes with 9.2 cores and 192 GB of headroom.

The comparison gives three short results:

• CPU: after failure, three nodes leave 44.8 cores and four leave 67.2 cores against a requirement of 58;
• RAM: after failure, three nodes leave 768 GB and four leave 1,152 GB against a requirement of 960 GB;
• result: the three-node design fails, while the four-node design passes both limits.

This does not mean you can fill the fourth node with a new system until the dashboard shows no free space. In normal operation, four nodes spread the load and lower the displayed percentages, but one node's capacity remains reserved for failure. A new VM is safe only while a repeated post-failure calculation preserves the agreed headroom.

Now check the largest VM. If it needs 448 GB of guaranteed memory, it fits on a host with 480 GB after the host reserve, but leaves only 32 GB for neighbors. The placement plan must show where the other VMs from the failed host will go. You may need 768 GB in every node even though the aggregate calculation with 512 GB formally passes. That is why the totals calculation comes before a placement simulation but never replaces it.

The peak needs a definition too. Do not add the maximum from December, quarter close, and the nightly backup if they cannot physically overlap. Do not use the 95th percentile if the missing five percent contains a payroll run that cannot wait. Put the time series on one scale, mark required processing windows, and choose the combination that can really occur after a failure.

Before approving the result, I run a controlled test: evacuate one node, disable automatic VM return, start a representative peak, and watch latency. The test does not prove behavior during a power cut or data corruption, but it catches the most common error: capacity exists in the spreadsheet, while placement rules or the network keep the cluster from using it.

The cost of waiting extends beyond the server price

Compare the present cost of unused capacity with the cost of a later purchase and the risk of the period when capacity is already insufficient. The fourth node's quote is visible immediately. An urgent expansion has less visible parts: repeated design work, another procurement procedure, a changed configuration, delivery, installation, licenses, migration, and possible limits on service growth.

For each phase, I calculate two totals:

buy_now = hardware_now + licenses_now + support_now + power_and_space
buy_later = hardware_later + licenses_later + integration_later
          + expected_shortage_cost + compatibility_risk

expected_shortage_cost =
    probability_capacity_exhausted_before_delivery
    × business_cost_of_restrictions_or_downtime

The last line does not ask for a probability accurate to two decimal places. Build three scenarios: expected growth, high growth, and delayed delivery. Each needs an observable trigger, a duration, and a monetary consequence. If the high case only postpones a test environment, its risk is small. If it blocks a new branch or leaves the cluster without emergency reserve for two months, the saving from deferral disappears quickly.

Idle depreciation is not as simple as an accounting table suggests. An early node already helps during maintenance, speeds data recovery, and allows failure testing. Yet an expensive processor bought for demand three years away ages and incurs support costs before it carries that demand. Split the cost of high availability from the cost of future performance. Pay the first part now in most cases, and assess the second from the date it will actually be used.

Use one currency and one accounting boundary. For a project in Kazakhstan, it is practical to convert every quote to tenge with separate lines for VAT, currency risk, and support indexation. Do not hide the exchange rate inside the server price: software licenses and support may renew on another schedule. Finance needs to see which part of the result depends on the exchange rate and which part comes from the node count.

An urgent later purchase rarely matches the first node byte for byte. After 18 months, the supplier may substitute the processor, network card, or drive. Even a compatible model requires new firmware, a microcode check, and another performance test. Include engineering hours and a change window. If procurement rules require a new tender, calendar risk may matter more than an expected hardware discount.

The useful output is a decision boundary, not one return-on-investment figure. For example: buy node four now because N+1 needs it; postpone node five until projected post-failure load reaches 75 percent of available CPU capacity or 80 percent of memory, allowing for delivery time. The threshold must leave enough time to finish procurement before the reserve runs out.

If the comparison spans several years, discount future payments to one date, but do not let the discount rate hide operating risk. Discounting lowers today's value of a node purchased in two years, yet it does not reduce the damage from insufficient capacity next quarter. Show committed payments, probabilistic losses, and the cost of options you can cancel as separate figures. A pre-agreed price, framework contract, or reserved manufacturing slot has value only for its stated period and configuration. Record the quote expiry and what happens after it. Otherwise the financial model keeps using a price the supplier no longer has to honor.

Phased procurement works only with planned compatibility

A postponed node is cheaper today only when it can join without a rebuild tomorrow. The phase-one specification must define the acceptable future node profile, network ports, licenses, rack space, power, and processor compatibility rules. "We will add the same one" becomes obsolete before the next budget arrives.

For CPUs, define in advance the common instruction set that VMs may use. Different processor generations can often share a cluster through a compatibility mode, but some or all VMs then lose access to newer instructions. If an application is bought for a specific acceleration feature, or if the database is licensed by core, a newer processor changes both performance and cost. Check the hypervisor and server-maker compatibility matrices before fixing the model, not during installation.

For memory, keep channel population symmetrical and leave a clear upgrade path. A node with half-empty slots can look flexible, but future modules of the required capacity and rank may not work with those already installed. Sometimes installing the target amount in all four nodes now costs less than replacing every module later. Make that choice from the platform specification, not from total DIMM capacity alone.

Networking and rack infrastructure usually expose weak phasing first. Reserve ports on both switches, management addresses, optical modules, cables, power supplies, outlets, and thermal capacity. If node five requires a switch replacement, "one server later" is actually a separate infrastructure project.

Software versions also set the window. A new server may support only a newer hypervisor release, while old hardware has not been certified for it. Adding capacity then becomes a whole-cluster upgrade under capacity pressure. Record the last compatible release, the support end for the current platform, and the point where phase two should create a new cluster instead of extending the old one.

A good phase-two specification describes a function rather than one part number:

• minimum usable CPU performance in an agreed test;
• RAM amount and layout, including the largest VM;
• network speeds, protocols, and required redundant paths;
• compatible drives, controllers, and firmware versions;
• the limit on licensed cores, sockets, or nodes.

That lets a supplier offer an available replacement while the architect retains control of the result. Without those constraints, a newer node can be nominally faster yet cost more in licenses or make balanced placement harder.

Licensing can make spare hardware the most expensive part

Every license has its own counting unit, and that unit often differs from useful capacity. Before choosing the node count, list the metric for the hypervisor, guest OS, database, backup, and management separately. One added host might increase only platform support, or it might require licensing dozens of physical cores again.

A Broadcom knowledge base article about VMware vSphere licensing says that every physical core needs a license, with a minimum of 16 cores per processor. It gives the formula MAX(cores_per_socket, 16) × hosts × sockets. The phased-purchase consequence is direct: a fourth single-socket, 32-core node adds 32 licensed cores when it enters service. If your contract does not allow you to buy the server now and activate the license later, the cost of waiting rises. Check the subscription and quote terms separately because the technical formula does not define billing dates or discounts.

Proxmox VE has different geometry. Its official subscription page counts occupied physical sockets rather than cores and requires a subscription for every cluster node; all nodes in a cluster must use the same subscription level. A single-socket server with many cores can therefore cost less for hypervisor subscriptions than two less-dense nodes. Higher density, however, increases the load lost with one failure and can raise the price of software licensed by core. Saving one line item must not silently change the failure model.

Microsoft Product Terms for Windows Server 2025 set minimums of 8 licenses per processor and 16 per server when licensing physical cores. A fully licensed Standard server grants rights for two virtual OSEs, and each extra full set of licenses across all server cores adds another two; Datacenter permits any number of OSEs on a fully licensed server. A host that receives Windows VMs during a failure is therefore not a free spare. Its rights must cover the actual peak of running instances. Microsoft's virtualization guidance also warns that physical-core licenses cannot be freely moved between servers for short periods.

Windows Server can be licensed by virtual machine with subscription licenses or active Software Assurance. Microsoft states a minimum of eight core licenses per VM and 16 per customer. That option can preserve the value of phased expansion when only a few Windows VMs exist, but compare it with physical licensing using your own counts. Datacenter often becomes easier to operate in a dense cluster, although the contract prices and number of concurrently running OSEs set the break point, not a popular rule copied from a blog.

SQL Server adds another layer. Current Microsoft guidance requires at least four core licenses for every virtual OSE under the Per Core VM model, plus active Software Assurance or a subscription. Enterprise licensed across all physical cores with the applicable rights can permit unlimited virtualization, but every potential database host then expands the license base sharply. Node five for a general cluster can cost less than the database licenses required to let the database run there. A dedicated database cluster, or rules that prohibit those VMs from part of the cluster, may save more than delaying the hardware.

Write the license layers as four separate lines:

• the hypervisor counts cores, sockets, or nodes, so a new host changes the metric through its physical configuration;
• the guest OS counts physical cores or vCPU and VMs, so emergency-run rights matter;
• the database counts a physical or virtual core, and permitted VM locations change the cost;
• support and management count a node, socket, or workload, often from the object's connection date.

This list does not replace licensing advice. It prevents the increase in a metric from disappearing inside the total project price. The contract and current Product Terms determine the final rights, not the architect's calculator.

A cold spare rarely replaces a running node

A boxed server preserves the original batch's compatibility and may avoid some operating licenses until commissioning if the contract allows it. In return, it brings recovery time and uncertain condition. The administrator discovers a failed module, old firmware, or missing component on the day of the incident, when there is the least time to investigate.

A running reserve participates in the cluster every day. It receives updates, stays under monitoring, accepts VMs during maintenance, and proves that networking, storage, and automation really work. A cold spare supplies only parts. To replace a failed node, the team must install it, align firmware, configure networking, register it with management, connect storage, and verify policies. That procedure must fit the RTO, or the spare does not solve the stated requirement.

There are cases where a stored node makes sense: a remote site with long delivery times, an isolated environment without continuous support, or a large fleet of identical servers where one spare covers several independent clusters. Power it up every quarter, run hardware diagnostics, install the approved firmware, and check the full cable and module set. Assign an owner and store the commissioning procedure with the inventory record.

Do not confuse a cold node with quorum or emergency capacity. While it is off and not joined, the running cluster cannot use its vote or restart VMs on it. If the business accepts four hours of manual recovery, that may satisfy the agreement. If it requires automatic restart within minutes, the server must be commissioned, tested, and available to the scheduler.

A compromise is to keep the extra node running but limit ordinary workload on it. That helps with tests and maintenance, although it does not remove license, power, and support costs. I prefer to distribute VMs evenly across all nodes and preserve a logical reserve through admission control. Every server then works, while the system still prevents the emergency capacity from being consumed. A separate "empty" host often ages unnoticed and fails on its first real migration.

Sign the expansion trigger with the specification

"We will buy later" becomes a complete decision only when the metric, threshold, response time, budget owner, and acceptable replacement model are known. Without them, monitoring reports the shortage but procurement starts after the reserve is gone. The trigger must look ahead by at least the full approval, delivery, and commissioning period.

Calculate the order point from forecast post-failure load:

order_threshold =
    failure_capacity
    - growth_rate_per_month × procurement_and_installation_months
    - forecast_error_margin

If post-failure capacity is 67.2 planning cores, the measured peak is 58, growth is one core per month, and purchase plus installation takes four months, you cannot wait for 67 cores. Even without forecast error, the order must start by 63 cores. With the narrow headroom in this example, the project probably needs to start immediately unless migration or optimization reduces growth.

The CPU threshold does not replace RAM, storage, network-port, and license thresholds. The first metric reached starts the order. I also record two events: the arrival of a VM that no node can hold with reserve, and the end of the current model's guaranteed compatibility window. Either can force a purchase before average utilization does.

The decision record must retain five answers:

1. Which simultaneous failure can phase one survive, and at what peak?
2. Which workload is temporarily limited after that failure?
3. Which licenses are paid now, and which start when the next node enters service?
4. What measurable threshold starts procurement, and who owns it?
5. Which configuration is acceptable if the original model is discontinued?

Then compare at least three options: a sufficient cluster now, a sufficient cluster plus a growth node, and a phase-one build with contractually prepared expansion. Do not ask a supplier for one "optimal" size without providing your metrics. The supplier will inevitably optimize against another risk boundary.

As a server manufacturer in Kazakhstan and a system integrator, GSE.kz can put the first and later phases into one specification, covering compute, data-center infrastructure, software, and support. The useful result of that discussion is not the largest configuration. It is two transparent bills of materials: what failure protection needs today and what gets purchased only after the agreed growth trigger.

Do not approve the budget until the model passes both a node-removal test and a license recalculation. Hardware can arrive later. Quorum, the right switch port, and the right to start an emergency VM do not appear retroactively.