Inbound traffic reveals the source of a broadcast storm

You find a broadcast storm by tracing the direction from which an abnormal flow of frames enters switch ports, not by picking the loudest switch. A switch at the center of the network often looks worst even though it is only replicating traffic that originated one floor below, behind a wall outlet, a cheap mini switch, or two outlets connected by mistake.

When the network is already down, the order of operations matters more than analytical depth. Save a few readings first, break the loop at the smallest possible point, restore connectivity, and only then examine the configuration. If you immediately disable every uplink, the office may recover, but you will destroy the trail to the cause and the outage will return when someone reconnects the cable.

A storm appears at three levels at once

A broadcast storm shows up in user experience, port statistics, and the switch's own behavior at the same time. You can mistake any single symptom for another fault, but the combination gives you a working Layer 2 loop hypothesis.

Users report that the gateway answers intermittently, DHCP takes a long time to assign addresses, voice calls break up, and access to servers in the same office is unstable. Latency to the gateway jumps and packets disappear while physical port indicators remain lit. Restarting one computer changes nothing because the fault affects the entire broadcast domain, usually one VLAN or a group of VLANs carried by a common trunk.

Several switch links become busy, and the flow consists mostly of broadcast and often multicast frames. The same address may rapidly move between ports in the MAC address table. Logs report frequent STP topology changes, full queues, discarded frames, or storm control events. SSH management and the web interface respond slowly because the management CPU is processing control events even when hardware forwards the frames.

High utilization does not always mean a storm. A backup can fill a link with unicast traffic without harming its neighbors. A wave of ARP traffic after power returns creates a short burst that subsides on its own. A duplex mismatch produces CRC errors, late collisions, and poor useful throughput, but it does not make one valid frame circulate forever. Check traffic composition, the affected area, and counter growth together.

The boundary of the outage can identify the VLAN before you log in to the first switch. If wired users in one department lose connectivity while guest Wi-Fi and phones continue working, compare their VLANs and gateways. If several VLANs fail together, look for a shared trunk, an incorrectly assembled link aggregation group, or a loop on a device carrying all of those VLANs. A storm in one VLAN does not have to fill every physical port on a switch, but it can make switch management unavailable and create the appearance of a network-wide failure.

Watch the management CPU, but do not make it your main test. Some models forward the storm in hardware while CPU use stays moderate. Others send ARP, unknown protocols, or control frames to the CPU and quickly lose management access. A normal CPU does not excuse abnormal port counters, and high CPU does not prove a loop without traffic analysis.

RFC 919 describes the unpleasant economics of broadcasting: every host that hears a broadcast packet spends resources on it. In a healthy network, that cost stays inside one domain and is limited to a finite number of copies. An Ethernet frame has no Layer 2 equivalent of IP TTL, so switches in a loop keep copying it until STP, a protection mechanism, or an engineer breaks the loop.

A loop and a noisy host need different fixes

A Layer 2 loop replicates frames that already exist, while a noisy host generates too many new frames. Both raise the broadcast counter, but the investigation and permanent repair differ.

With a loop, several connected ports show high inbound and outbound rates. The same source MAC address jumps between ports because its frames return along another path. A packet capture often contains repeated ARP requests or other identical frames. Disabling one edge of the ring causes traffic throughout the domain to fall almost immediately.

With a noisy host, the main inbound rate is concentrated on one access port. Source MAC addresses remain stable, STP roles do not change, and captured packets differ or come from one sender with a changing sequence. Disabling that port also ends the incident, but the cause is a network adapter, driver, application, or misconfigured device rather than a cable loop.

There is a third case: multicast traffic floods like broadcast traffic because IGMP snooping is absent or broken. It can fill access ports even when there is no loop. Look at separate broadcast and multicast counters, the MAC table, and repeated frames. The observation that every light is flashing cannot distinguish these faults.

The practical distinction is simple. For a loop, find the closed path. For a generator, find one source. Storm control can contain either event on a particular port, but it cannot tell you why the traffic appeared or repair the cabling.

A snapshot before shutdown saves hours later

Before changing any port state, save a short snapshot of the facts if the switches still respond. It takes no more than a couple of minutes and preserves the direction of the investigation after service returns.

Record the time, affected VLANs, gateway reachability, and switches that still allow management access. Capture STP state, interface rates, broadcast and multicast counters, logs, LLDP or CDP neighbors, and MAC address moves. Do not clear counters until you have saved their initial values.

For Cisco IOS XE, a minimal set can look like this:

show clock
show spanning-tree vlan 120
show interfaces counters
show interfaces | include is up|input rate|output rate
show mac address-table notification mac-move
show logging
show lldp neighbors

Command names depend on the vendor and release. On other switches, look for equivalents of interface statistics, Ethernet switching table, spanning tree status, log buffer, and LLDP neighbors. Do not paste an unfamiliar clear or restart command into an emergency console until you have checked what it does.

In a switch stack, record the member number with the port. Gi1/0/24 and Gi3/0/24 refer to different physical cabinets even though their final numbers match. Check internal stack-link status because saturation or failure there changes the traffic path and can make external counters look surprising. In a virtual chassis, record the slot and interface owner for the same reason.

Take two snapshots 10 to 20 seconds apart. An absolute counter accumulated over a year tells you almost nothing. The difference between two values gives you a rate. If the broadcast counter rises from 18,240,100 to 18,690,100 in 15 seconds, that is 30,000 frames per second:

(18 690 100 - 18 240 100) / 15 = 30 000 pps

Compare ports that have the same role. A floor uplink naturally carries more traffic than a printer port. The useful signal is not merely a high counter, but a sudden increase relative to normal load and nearby links with the same role.

Cisco's Catalyst 9000 Layer 2 loop troubleshooting guide recommends looking first at inbound rate, the share of broadcast and multicast traffic, and average packet size. This is more useful than chasing a stream of STP topology change notices. Overloaded switches generate many of those notices themselves, so an event may identify a victim rather than the source.

Follow inbound counters against the flow

You find the source by moving from the core toward the neighbor that sends the largest abnormal inbound flow. At each step, check the inbound rate on suspect ports and the neighbor map until a trunk becomes an access port or two paths close back on each other.

Consider a network with a CORE-1 switch, FLOOR-2 and FLOOR-3 access switches, and VLAN 120. Two uplinks on CORE-1 transmit at nearly line rate, but inbound broadcast is much higher on the port toward FLOOR-2. Move to FLOOR-2. There, the heavy inbound flow arrives on Gi1/0/47, which LLDP maps to a small meeting-room switch. Two of its ports carry the flow at the same time, and MAC addresses jump between them. A physical check finds two patch cords connecting that switch to two outlets in the same VLAN.

This direction matters. High outbound traffic often leads away from the source toward another victim. High inbound traffic tells you where the switch receives the copies. Asymmetric topology, aggregated links, and platform-specific counters create exceptions, so compare readings at both ends of a link.

Use this five-action sequence:

1. Find a switch that shows the impact and identify the storm's VLAN.
2. Compare the growth of inbound broadcast and multicast traffic on active ports.
3. Identify the neighbor on the abnormal port through LLDP, CDP, its description, or the network diagram.
4. Move to that neighbor and repeat the measurement over the same interval.
5. At the network edge, map the MAC address to the patch panel, wall outlet, and physical device.

Do not trust a port description as your only evidence. After office moves, a label such as "accounting printer" often leads to an unmanaged switch under a desk. LLDP can also disappear under heavy load. Match at least two signals: counters and neighbor data, or counters and the physical cable path.

If you can capture traffic from a mirror port, check whether identical frames repeat. Matching source and destination addresses, EtherType, ARP content, and close arrival times strengthen the loop diagnosis. A capture must not delay containment. When the whole office is unavailable, counters and a careful shutdown of the suspected edge usually give an answer faster.

Check unknown unicast separately. A switch floods a frame with an unknown destination MAC across the VLAN in much the same way it floods a broadcast. When the MAC table becomes unstable, this traffic grows and adds to congestion even though the broadcast counter does not show the full picture. If the total inbound rate is high but broadcast is moderate, compare unknown unicast, MAC entry ages, and learning rates.

The search ends with a physical object in a room, not a port name. A forwarding-table entry must lead to a patch-panel position, a wall outlet, and a cable. Ask someone on site to photograph both ends before disconnecting anything. That simple record often reveals a loop between two wall outlets that cannot be seen on the logical diagram.

STP shows why protection failed to stop the loop

Spanning Tree Protocol should leave one path active and block the redundant path, but having STP in the configuration does not mean it can see a particular loop. Check port roles, the root bridge, BPDU reception, and the boundary of every VLAN.

First compare the actual root bridge with the intended one. An unexpected root often points to an attached switch with a more favorable priority. Then find ports in forwarding and discarding or blocking states. If two parallel paths carry the same VLAN and neither is blocked, determine whether both ends receive BPDUs and whether trunk VLAN configuration matches.

A common office cause hides behind edge or PortFast settings. The setting makes sense on an end-device port because it avoids the normal STP transition delay. It does not make the port safe. If someone connects a small switch to two edge ports, they create a path the designer never planned. BPDU Guard should disable such an access port when it receives a BPDU, but it works only where it is enabled and where the attached device actually forwards BPDUs.

Check four protection mechanisms without confusing their jobs:

• BPDU Guard closes an edge port when a BPDU appears and protects the access boundary.
• Root Guard prevents a device on the port from becoming the root.
• Loop Guard keeps a non-root port from moving to forwarding when expected BPDUs disappear.
• Storm control limits broadcast, multicast, or unknown unicast at a threshold and reduces the damage.

Storm control does not replace STP. A threshold that is too high lets a loop overload the network, while one that is too low cuts off legitimate bursts of ARP, DHCP, or device discovery. Choose thresholds from measured baseline traffic for each port type and decide the action in advance: discard, notify, or disable. After an event, the engineer needs to see the port, traffic type, threshold, and time. Otherwise the protection merely turns one large outage into a mysterious local failure.

Check aggregated links separately. If one end treats two links as one LAG while the other sees independent ports, STP and the forwarding table receive conflicting information. Compare port-channel membership, the negotiation protocol, VLANs, and each member's state at both ends.

A unidirectional link creates another dangerous case. One end receives BPDUs while the other stops seeing them because of a failed transmitter, optic, or filter. A port that used to block may decide that the path to the root is gone and start forwarding. Compare BPDU reception, the LLDP neighbor, and inbound rate at both ends. If each end claims to be designated and the neighbor appears on only one side, inspect the physical link and Loop Guard.

When the network is down, break the smallest edge

When connectivity is completely lost, restoration comes first, but you should disable one confirmed or most likely edge of the loop. Start at the perimeter if the counters point there, and avoid disabling every backbone link at once.

Before issuing shutdown, record the port, time, current counters, and reason for choosing it. Check whether the link is the only uplink for a critical segment, telephone system, access-control system, or medical equipment. If the risk is acceptable, disable the port and immediately watch inbound rate on the upstream switch.

The correct edge produces a sharp, sustained drop in broadcast traffic, restores management access, and normalizes latency to the gateway. Wait for at least two measurement intervals because queues need to drain and STP may reconverge. If the readings do not change, restore the port to its prior state, record the result, and move to the next candidate. Do not leave behind an arbitrary collection of disabled ports.

If management access is completely gone, physically disconnecting a cable is acceptable, but the on-site team must name both ends and report every action. Start with recently changed areas and peripheral dual connections when that choice agrees with the diagram. Do not pull core cables based on color or how quickly a light flashes.

Keep a short emergency procedure:

1. Declare the incident and stop uncoordinated cable or configuration changes.
2. Save available counters, STP state, logs, and the time.
3. Choose one edge from inbound traffic and neighbor data.
4. Disable it, then check the traffic drop and test reference hosts.
5. Record the temporary topology and keep the port down until the cause is verified.

After recovery, test more than ping. Clients must obtain DHCP leases, resolve DNS, reach the gateway, and access major internal services. The voice VLAN, Wi-Fi, and guest network may recover differently from user workstations. One successful gateway reply does not prove that the office is operating normally.

Assign one person to direct changes and another to keep the log. If three administrators disable different ports at the same time, nobody can connect a counter drop to one action. The log should contain the exact time, device, interface, state before and after, operator, and result of service checks. This discipline may sound slower, but it usually shortens the outage because the team does not repeat rejected hypotheses.

MAC flapping and STP changes give clues, not verdicts

Frequent moves of a MAC address between ports can outline a loop, but they do not prove one by themselves. A wireless client legitimately roams between access points, a cluster may use a shared virtual MAC, and a hypervisor can move a virtual machine between hosts.

A loop usually creates a broad pattern. Many MAC addresses move quickly across the same pair of interfaces or several links in one ring. Match the timing of those events to broadcast growth. One moving address without a storm calls for investigation of that device, not disconnection of an entire floor.

A Topology Change Notification does not point directly to the source either. In RSTP, a change occurs when a non-edge port moves to forwarding. During a storm, the management CPU may delay BPDUs, ports change state, and logs fill with secondary events. Cisco explicitly warns that following TCNs can lead to affected switches instead of the point where the loop began.

A timeline is more useful: which port first showed a sharp inbound flow, where MAC moves began, which interface changed its STP role, and which action stopped the growth. Switch clocks must be synchronized. Without synchronization, a claim that one event happened first is only a guess.

Compare logs with recent approved changes, but do not automatically blame the latest change. A new access point can coincide with a loop caused when cleaners disturb an old cable. A change becomes a credible suspect when its VLAN, port, and timing match the inbound traffic direction. This test avoids a convenient but false story.

CRC errors, drops, and no-buffer counters add context. CRC usually points to a physical link or a duplex mismatch, while drops and no-buffer events appear under congestion. They can accompany a storm, but replacing a bad patch cord will not break a logical loop if a second active path remains.

Prove the root cause after connectivity returns

The incident is not closed until the team has reconstructed the loop path on the diagram and explained which control should have stopped it. "We rebooted the switch and it helped" describes an action, not a cause.

Save the affected port configurations, logs, counters before and after shutdown, the MAC table, and photographs of the cabling. Trace both ends of every cable. If you find a small unmanaged switch, inspect every port. The loop may pass through two outlets, an IP phone with a pass-through port, or a docking station.

Answer four questions:

• Which physical or logical path closed the loop?
• Why did STP fail to block one path?
• Why did BPDU Guard, Loop Guard, or storm control fail to contain the event?
• Which change will prevent recurrence without creating a new single point of failure?

The repair must match the answer. Remove the extra cable and block the unused outlet. Mark only user-facing ports as edge ports and enable BPDU Guard according to the vendor's policy. Correct allowed VLANs and LAG negotiation on both ends. Set storm control from actual measurements. Add port descriptions and update the diagram so the next engineer does not follow obsolete labels.

The incident report should distinguish the triggering event, the missing protection, and the consequences. For example, the trigger was two cables between a meeting room and the floor switch. The missing protection was BPDU Guard on user ports. The consequences were full queues and loss of VLAN 120. If the report compresses all of that into "broadcast storm," the team will not know which control to change.

Do not enable the incident port "just to test" during working hours without watching counters and having a shutdown command ready. It is safer to reproduce the topology in an isolated segment or add one cable at a time, checking STP and broadcast growth after each connection.

For organizations whose switching network connects server-room and user infrastructure, GSE.kz can design and support the solution as a system integrator with 24/7 technical support and a service network across Kazakhstan. That does not remove the need for an internal diagram, baseline metrics, and console access for the team. An outside engineer also traces a loop from evidence.

Protection works only after a failure test

Prevention starts with a clear boundary. User ports must not become trunks, accept an outside root role, or form unplanned rings. A consistent configuration template on access switches reduces exceptions that nobody remembers.

Enable RSTP or MSTP consistently across the domain, deliberately assign root and backup bridges, and limit allowed VLANs on trunks. Use BPDU Guard on edge ports, consider Loop Guard on suitable redundant paths, and set tested storm control thresholds at boundaries. Do not copy one percentage to every interface. A phone, access point, uplink, and printer each have a different normal traffic profile.

Monitoring should retain inbound broadcast and multicast rates, port utilization, MAC moves, STP changes, and protection events. Establish a baseline by time of day. Otherwise an alert for "high traffic" will either remain silent during an outage or wake the on-call engineer whenever staff turn on their computers in the morning.

Run a controlled test in a lab or isolated VLAN. Connect a test switch to a protected edge port and verify that BPDU Guard creates the expected state and a clear log entry. Feed a limited broadcast stream from a traffic generator and test the storm control threshold without touching production. Document recovery commands because automatic shutdown without a clear procedure can extend the outage.

Finally, rehearse the search with two counter snapshots. An on-call engineer should be able to move from the core to an access port in a few hops without clearing the evidence or disabling half the network. During a real storm, elaborate monitoring dashboards may freeze with the network, while the console, LLDP map, and counter differences remain usable tools.