Updating equipment without internet access

Network isolation does not make an update safe. It merely replaces automated delivery with a chain of people, media, and intermediary nodes, and every new handoff adds a place where a package can be confused, damaged, or replaced. Updating equipment without internet access must therefore be designed as a controlled supply process, not as copying files from a USB drive.

A working process gives verifiable answers to five questions: where the package came from, which equipment it is meant for, who authorized the transfer, what was actually installed, and how to restore the previous version. If the team answers even one of these questions from memory, the closed segment already depends on luck. This becomes especially expensive on servers, storage systems, operator workstations, and control devices, where a failed firmware update can stop a service before an engineer finds a compatible recovery image.

The update route must be one-way and clear

A safe design separates update acquisition, verification, and distribution. A computer with internet access must not connect first to a vendor site and then to the closed network. That kind of temporary bridge defeats segmentation and leaves little evidence for an investigation.

The minimum architecture includes a connected download node, an intermediary verification node, and an internal repository. The download node obtains packages only from approved sources and saves the accompanying files: the signature, manifest, release notes, and installation instructions. The intermediary node sits in a controlled zone, routes no traffic between networks, and accepts data through approved media or a one-way gateway. The team checks malware, signatures, hashes, archive contents, and correspondence with the change request there. The internal repository then distributes the approved snapshot to target systems.

Microsoft describes the same principle for disconnected WSUS deployments: administrators export metadata from a synchronized server, transfer content and license files separately, and then import the metadata into an autonomous server. Microsoft Learn notes an important detail: language settings and express installation file settings must match on the export and import sides. Otherwise the catalog appears complete while a client lacks the required content. This is a good example of why copying every file from a directory does not replace a defined process.

Send update content inward, and move reports outward through a separate, sanitized route. Do not return the same media from the closed segment to the connected node. It may collect logs, configurations, service names, or a malicious file. For status reports, use different media, a separate gateway that filters file formats, or manual transfer of short identifiers. Document the direction of data and the permitted file types in a diagram that an on-call engineer understands, not one understood only by the architect.

Inventory determines which packages are actually needed

Before downloading updates, the team must record the exact hardware and software baseline of the segment. A server family name is not enough: the same series may contain different board revisions, storage controllers, network adapters, BMCs, and bootloader versions. A package that fits the next rack can fail on a node that looks identical.

For each asset, record its identifier, model, serial or inventory reference, hardware revision, and current versions of BIOS or UEFI, BMC, drive and controller firmware, drivers, operating system, and critical applications. Add the node role, dependencies, permitted downtime, and emergency access method. The inventory must distinguish “version unknown” from “component absent”: the first calls for investigation, while the second closes an unnecessary branch of work.

The decision source is not the highest version number. It is the vendor compatibility matrix and the bulletin for the specific release. Check the supported starting version, required intermediate releases, order for related components, free-space requirements, expected reboot duration, and any rollback prohibition. If the vendor supplies a universal bundle, unpack its contents and map each image to a real device. Universal packaging does not make the firmware universal.

Store a state snapshot with the change request instead of linking to a live spreadsheet. That spreadsheet will change before the next maintenance window, and a month later nobody will be able to prove which source data supported the approval. Export the inventory snapshot with the request, assign it a version and a hash, and make the update bundle generator use that exact snapshot.

Find missing equipment by comparing three sources: the asset register, the list of clients in the internal repository, and an actual query inside the segment. Do not automatically resolve a discrepancy in favor of one source. It means the asset owner, network route, or management agent needs investigation. Resolving it before the maintenance window costs far less than discovering an unknown controller after updating its redundant partner.

A complete bundle matters more than individual files

A delivery must contain a self-contained set that the team can verify and install without another internet request. One primary package is rarely enough. The installer may need dependencies, repository metadata, a license, an intermediate firmware version, a flashing utility, or an updated key. If the missing item is discovered only inside the closed segment, the engineer will either lose the window or look for a shortcut. The second choice usually ends with a file of unknown origin.

Build the bundle from the fixed inventory snapshot and test it on a clean machine with no access to external repositories. This rehearsal exposes hidden dependencies. An installer that quietly downloads a library or a fresh index on a connected test system will behave differently in isolation. The test node's network log should show that installation made no mandatory external requests. If the product needs activation, a certificate revocation list, or a time server, arrange a supported offline mode in advance instead of imitating replies with an improvised service.

The bundle manifest records its identifier, build date, target models and revisions, starting and target versions, installation order, file names, sizes, and full hashes. It also includes the version of the vendor instructions, the signing key fingerprint, and known restrictions. People need a readable document, while verification tools need a strict format such as CSV or JSON. Generate both representations from one source so that a manual edit cannot make them disagree.

Do not mix full and incremental deliveries. A full internal repository snapshot is easier to restore and verify, but it takes more storage and transfer time. An incremental bundle contains changes after a named baseline and requires that exact baseline on the import side. If the team loses part of the incremental chain or applies bundles out of order, the catalog may point to files that do not exist. Publish a new full snapshot periodically, and name the parent identifier explicitly in every incremental delivery.

Check capacity in three places in advance: the transfer media, the receiving zone, and the internal repository file system. An archive may need space for both the original file and extracted data during unpacking. Red Hat documentation for disconnected Satellite deployments specifically warns about the extra space required to expand exported content. Running out of space during import does more than cause delay. It leaves an incomplete tree that someone may mistake for a ready repository.

File names should not carry meaning already held by the manifest. A long name containing a model and the word final is easy to change manually and hard to verify unambiguously. Give the delivery an immutable identifier and keep the vendor's original file names. The receiving directory begins with a “quarantine” status, becomes “approved” after every check, and moves to “rejected” with a reason if it fails. Renaming a folder is not an access control by itself. Permissions and a registry entry must enforce the status.

Finally, inspect the bundle for unwanted data. Transfer media must not contain browser history, stored credentials, internal configurations from the connected node, or unrelated downloads. Build the export in an empty directory from the manifest list, then compare the actual file list with the expected list. An unknown file stops the delivery even when all required packages are present.

A hash proves integrity, while a signature proves origin

A checksum shows that two byte sequences match. It does not prove who created those bytes, because an attacker can replace both the package and the neighboring SHA-256 text file. Authenticity comes from a digital signature verified with a key the organization trusted before receiving the particular update.

NIST SP 800-53 control SI-7 connects software and firmware integrity with cryptographic mechanisms, including digital signatures and signed hashes. The practical consequence is straightforward: do not accept a package merely because its hash matches a value on the same download page or transfer medium. Establish the trusted public key or certificate chain through a separate procedure. Verify the key fingerprint through an independent channel when enrolling it and during planned rotation.

On the intermediary node, verify the manifest signature first, then verify each file hash against the trusted manifest. For a set with a detached signature, the team can run:

gpgv SHA256SUMS.sig SHA256SUMS
sha256sum -c SHA256SUMS

Success from the first command means the signature is mathematically valid for a key in the trusted keyring. It does not yet mean policy permits that key for this vendor and equipment type. Success from the second command confirms that the files match the signed list. Save the full command output in the delivery log, including errors and warnings, not just the final word OK.

Verify embedded signatures with the format's native tool as well, such as rpm -K for RPM packages or verification of signed Release or InRelease files in APT. Ubuntu documentation recommends binding a third-party repository to a separate key through Signed-By instead of trusting one shared set of keys for every source. This is especially useful for a closed repository: compromise of one key must not authorize packages in the name of every vendor.

Signature checks have three awkward cases. First, the certificate has expired while the closed segment clock has drifted substantially. Correct trusted time and check again instead of disabling expiration checks. Second, the vendor has changed its key. Obtain the new key and rotation confirmation separately from the package, then process a change to the trust store. Third, old equipment accepts only an unsigned image. That package needs a documented exception, independent source checks by two people, testing on an identical bench, and a decision by the risk owner. An internal signature proves only that the organization has not changed the file since repackaging it. It does not prove the origin of the original file.

The intermediary node must not become another workstation

The intermediary node exists to receive and verify content, so nobody should read email, open office documents, or browse for drivers on it. The more routine tasks it performs, the harder it becomes to distinguish an authorized file from an accidental download and the more likely it is to infect transfer media.

CISA guidance for updating antivirus tools in industrial control systems describes a sequence that also fits firmware: verify the source, download to a dedicated host, scan the files, check a cryptographic hash, clean and rescan the removable media, and test the update away from a critical endpoint. I would strengthen that sequence with mandatory signature verification and an immutable delivery manifest. A hash alone, as explained above, does not establish authorship.

Use dedicated, numbered media for every transfer run. Before writing, sanitize it through the approved method, format it, and scan it. After writing, switch the media to read-only mode if the device supports it, seal it, or place it in a labeled container. The operator records the media number, delivery number, time, sender, and recipient. An ordinary USB drive taken from a drawer with no handling history must not cross the segment boundary.

Disable autorun on the intermediary node and prevent execution from the mount point. Extract archives into a disposable directory with limited permissions. Malware scanning helps, but a clean scan does not make a package trusted. A scanner may not recognize a new threat, while a legitimate installer can alter firmware and system files by design. The decision rests on the combined evidence of source, signature, expected contents, and testing.

After verification, create a closed delivery set containing the packages, signed or internally attested manifest, instructions, compatibility information, scan results, and change request identifier. Do not add tools “just in case.” Every extra executable expands the review scope and gives the operator another wrong path. Do not edit a bundle after approval. A correction creates a new delivery version, a new hash, and a new approval.

Update critical equipment in waves, not a whole rack

Dependencies and the ability to survive failure determine installation order, not the convenience of walking through a server room. First confirm backups of configuration and data, working emergency management access, sufficient power for the entire window, and a compatible recovery image. A backup that nobody has restored creates false confidence.

Make the test environment as close to production as practical: use the same board revision, controller, drive type, operating system version, and workload. A virtual machine can test an OS or application package, but it cannot expose a failed BMC, BIOS, network adapter, or drive firmware update. If no complete duplicate exists, use the least critical node with the same configuration and agree in advance that it may fail.

A rollout can be divided into four waves:

1. A laboratory system or spare node receives the entire approved bundle and passes reboot testing.
2. One noncritical production node runs under normal load for a defined observation period.
3. Secondary cluster nodes are updated one at a time, waiting for replication and quorum to recover after each one.
4. Active nodes and single-instance equipment are updated only after an explicit decision to continue.

Do not impose one universal order such as “BMC, BIOS, drivers” on every platform. A vendor may require an intermediate controller version, a matched BIOS and BMC pair, or a new driver before adapter firmware. Extract the dependency from the bulletin and place it in the plan. If the documentation is ambiguous, stop packaging and obtain confirmation from the vendor or integrator before the window.

For clusters, check more than port availability. After each node, wait for redundancy to return, data to synchronize, sensors to report normal state, and an application test transaction to succeed. For storage, check both controllers, multipath routes, and background jobs. For operator workstations, confirm that specialist applications start and connected devices work. A green power light says nothing about service readiness.

The maintenance window starts with stop conditions

A good plan says in advance when the engineer will halt deployment. Without that rule, a team tends to continue after the first strange symptom because the window is short and the change has already been approved. Stop conditions may include loss of BMC access, repeated boot failure, array degradation, loss of quorum, an unexpected component version, or an operation that exceeds its planned duration.

Before the first change, appoint a work lead, an operator, an observer, and an owner for the decision to continue. One person should not enter commands, interpret alerts, and authorize their own departure from the plan at the same time. A small site may combine roles, but the confirmation point must still be explicit and recorded.

The working sequence fits into five verifiable stages:

1. Match the asset identifier, starting versions, redundancy state, and approved bundle number.
2. Capture configuration, health logs, and baseline measurements before the change.
3. Install one logical update layer in the order specified by the vendor.
4. Reboot, run hardware diagnostics, and perform an application check against written criteria.
5. Record the result, and only then authorize the next system or wave.

Never replace a missing file during the window with a similar package from another delivery. Do not change configuration “while you are there” unless the request includes it. After a failure, you must be able to separate the effect of the update from the effect of a configuration change. Every deviation receives an identifier, author, time, and decision. If a change is urgent, issue an addendum to the plan instead of rewriting history afterward.

Do not interrupt a long firmware operation merely because its interface has stopped updating the percentage. Before the window, learn the normal duration, signs that writing continues, and a safe way to inspect status. A pre-set limit is still necessary because endless waiting may hide a hung controller. When that limit is reached, the appointed lead follows the vendor procedure. The most impatient person at the rack does not make the decision.

Test rollback before installation, not after failure

Not every update permits a return. Firmware may change data structures, configuration format, secure boot keys, or the minimum permitted version. The “rollback plan” field cannot say only “install the previous version.” The team must know whether the device accepts the old image, whether configuration survives, and how long restoration takes.

For an operating system, rollback may mean booting a previous snapshot or partition, but only if application data remains compatible. For a database, restoring binaries without a consistent data state often makes the incident worse. A backup bank and local console can help with BIOS, but their presence and behavior depend on the model. BMC recovery needs its own route because a controller failure removes the engineer's usual remote access.

Perform a test recovery before the window and record the observed result: elapsed time, messages, required actions, and versions after boot. Keep the previous approved package beside the new one in the internal repository, but separate them with unambiguous identifiers. A file named final_old_really.zip in a shared directory is not a recovery plan.

Analyze a failure at the boundary where it occurred. If the signature fails, the package never leaves the intermediary node. If the bench system fails to boot, the delivery never reaches production. If the pilot node degrades under load, stop the wave, preserve diagnostics, and decide whether to restore the pilot. If a problem appears on one secondary cluster node, do not update its partner for “symmetry.” Restore safe redundancy first.

After a rollback, do not record the status simply as “successful.” Record the failed target version, the version actually restored, data state, residual changes, and a block on repeat installation. Otherwise the internal repository may offer the same package in the next cycle and another shift may repeat the incident.

The version log connects a package to a specific asset

The log must answer incident-response, audit, and next-cycle questions. It needs to show not just the current version but the transition: which asset, which component, the starting and target versions, the package, the result, and the person who authorized it. A snapshot saying “all servers are current” goes stale quickly and cannot explain differences.

A practical record can use JSON Lines or a table with the same fields. One record for one component looks like this:

{"asset_id":"srv-042","component":"BMC","from_version":"3.10","to_version":"3.14","package_sha256":"7d2c...e91a","signature_result":"valid:vendor-key-2026","change_id":"CHG-1842","operator":"ops-07","installed_at":"2026-07-28T21:14:00Z","result":"success","rollback":"not-used"}

An operator identifier works better here than a free-form name. It maps unambiguously to an account and avoids search failures caused by spelling variants. Store time with its time zone and maintain trusted internal synchronization on the nodes. Record the complete hash in the real system; the example shortens it only for readability. The signature result must identify the key or certificate, because the word valid cannot be rechecked after key rotation.

Maintain a separate delivery registry. It links one manifest to every incoming file, source, acquisition date, verification result, transfer medium, and approval request. The asset log answers “what did we install,” while the delivery registry answers “what did we allow inside.” Combining the two entities creates empty rows for packages that were never installed and loses delivery history.

After every wave, compare the expected version with the version read independently from the device, automatically or manually. An installer exit code does not replace that query. Some programs return success after staging and apply the update only at the next reboot. Others update some components and leave a warning in another log. Store both the installation command result and the independent version check in the change record.

Restrict access to the log, but do not leave its only copy inside the segment being updated. A serious failure may make the segment or internal repository unavailable. Before the window, export a signed snapshot of the log to protected storage. After the work, add the final snapshot through the controlled outbound process. That export must not contain secrets, memory dumps, or full configurations.

The internal repository distributes only an approved snapshot

The internal repository must separate content intake from publication to clients. An imported catalog is initially unavailable to production nodes. After metadata, file completeness, and installation authorization checks, an administrator promotes a specific snapshot into a test channel. The same immutable snapshot then advances into pilot and production channels. Repackaging between channels destroys proof that the bench and production server received identical bytes.

Clients use a fixed internal repository name and trust only assigned keys. Do not point every server at a directory mounted from the latest USB drive. That bypasses centralized approval, complicates logging, and keeps the media accessible longer than necessary. The medium finishes its role in the import zone, after which packages move through the ordinary internal management channel.

Separate permission for four operations: import, verification, publication, and installation. The person who brought the bundle must not be able to publish it alone to every critical asset. The repository service account must not change the manifest or trusted keys. Clients need only read access to published content, while the management system or an approved local procedure grants the right to begin installation. This separation does not require a large team, but it does require distinct permissions and recorded approval.

Protect the repository from accidentally mixing releases. If the pilot receives release A but the production channel points to release B by the time of its window, the test result no longer applies to the bundle being installed. A channel reference must resolve to the immutable snapshot identifier written in the request. Before installation, the client or operator verifies that identifier and the manifest hash.

Removing old content is also part of the process. An endless archive makes selection harder and increases the impact of accidental publication, while premature deletion removes a tested rollback and evidence. The retention policy accounts for the support period, completion of every wave, rollback feasibility, audit requirements, and continued availability from the vendor. Do not merely hide a package withdrawn for a defect or revoked certificate. Put its identifier on a blocklist so an operator cannot import the same delivery again under another name.

Treat the repository as a separate critical service. Its backup must include content, metadata, channel states, trusted keys, and the promotion log. Test recovery on an isolated node: restore the catalog, verify the snapshot hash, and confirm that a test client sees exactly the approved versions. If an administrator must remember manually where each channel pointed after restoration, the backup is incomplete.

A regular cycle is safer than rare heroic windows

A closed segment does not justify waiting for months without a risk assessment. The team should collect bulletins regularly, match them against inventory, and assign a handling deadline according to the effect on the specific service. Urgency depends on more than a vulnerability score. Consider whether the vulnerable interface is reachable inside the segment, whether media provides an attack path, whether compensating controls exist, and what downtime will cost.

Useful measurements include the age of the last inventory snapshot, time from publication to an internal decision, share of assets with a confirmed version, number of exceptions, and number of failed waves. These measures expose process delays. Do not turn them into a race for 100 percent installation. A justified exception can be safer than incompatible firmware, while a hidden exception is more dangerous than an openly accepted risk.

Every few cycles, run a failure exercise on the bench: a damaged archive, an invalid signature, a missing dependency, or a hang after reboot. The operator should stop the process according to the instructions, preserve evidence, and move into recovery. This exercise quickly reveals that the key is inaccessible, the emergency image cannot be read, or the responsible person's phone number exists only in internal email.

An urgent release must not create a separate, unrecorded route. Prepare an emergency version of the same procedure in advance: identify who can decide outside the normal committee, where clean media is stored, which bench is available at night, and who may stop the service. Signature verification, content comparison, and recording the installed version remain mandatory. The team may shorten observation between waves or narrow the application test, but every reduction must be linked to a specific risk and a named decision owner.

Do not combine an urgent fix with a backlog of routine updates. The more independent components changed in one emergency window, the harder it becomes to find the cause of a failure and the longer rollback takes. Build the smallest bundle that addresses the stated threat and its mandatory dependencies. Leave other updates for the next normal cycle, even if the medium is ready and time remains in the window.

When a bulletin lacks complete information, the team can apply a temporary compensating measure: disable the vulnerable interface, restrict a route, prohibit a file type, or remove the service from external exchange. Give that measure an expiration date, an owner, and a test of effectiveness. The statement “the segment is isolated” is too broad. Show which attack path has been blocked and why removable media or a trusted internal node cannot bypass the restriction.

After the emergency window, complete the normal review while logs and details remain available. Match the downloaded package against the manifest, confirm versions on every affected asset, remove temporary permissions, and return the media to the defined sanitization cycle. If testing was reduced, keep the pilot under extended observation and schedule a final decision date. Urgency may change the timetable, but it must not erase package provenance or responsibility for installation.

For infrastructure built on locally manufactured GSE.kz computers and servers, compatibility, system integration, and continuing support can be connected in one update plan. The customer still needs its own decision log, because responsibility for admitting a package into a closed segment cannot be transferred through one line in a contract.

The first action before the next window is concrete: take one already approved package and try to prove its route from the source to the version on a particular device. If the signature, manifest, media record, or post-installation verification is missing, you have found a process break without suffering an outage. Close that break before the next firmware release requires an overnight recovery.