Thursday, June 23, 2011

Flow Control on Gigabit Ethernet Interfaces

Flow control is a feature defined in the IEEE 802.3x specification.
It enables a receiving device to signal congestion to a sending device, which allows the sending device to temporarily halt transmission in order to alleviate congestion at the receiving device.

IEEE 802.3x Flow Control

The figure above demonstrates how flow control works. The following events occur:
1. The transmitter is sending data (frames) to the receiver.
Note that the figure represents only one direction of the connection.
The transmitter and receiver roles are swapped for frames sent in the reverse direction.
2. The receive buffer on the interface connected to the transmitter is full and causing a congestion. This is common when traffic is sent to a switch on a high-speed interface and is then forwarded to a lower speed interface. If the receiver receives any more frames from the transmitter, they are discarded until the congestion is alleviated when the receive buffer is emptied. In order to avoid this situation, the receiver sends a pause frame with a wait time value to the transmitter to instruct the transmitter to stop sending frames for the specified wait time.
3. Assuming the transmitter supports flow control, it stops sending frames for the wait time period. After the wait time period is over, the transmitter starts transmitting frames again.
4. If the receiver clears its receive buffer before the wait time period is over, it sends a pause frame with a wait time value of 0 to informs the transmitter that it can starts transmitting again. This ensures that the connection is not idle if congestion is cleared before the transmitter is allowed to restart transmission.
5. The transmitter restarts transmission, either due to the wait time is expired or a pause frame with a wait time of 0 is being received.
If congestion occurs at the receiver interface again, the process above starts again.

All Cisco Catalyst switches that include Gigabit Ethernet capabilities include support for flow control. However, the support for flow control might be limited depends upon the type of Gigabit Ethernet port. All the Gigabit Ethernet interfaces on all Cisco Catalyst switches support the ability to receive and respond to pause frames – act like the transmitter in the figure above. However, some Gigabit Ethernet ports do not support the ability to send pause frames – act like the receiver in the figure above.
The ability to receive and respond to pause frames is referred to as input flow control;
and the ability to send pause frames is referred to as output flow control.

The table below describes the flow control capabilities of the various Cisco Catalyst switch Ethernet ports:
Oversubscribed ports are referred to oversubscription to the switch backplane.
Uplink ports are non-oversubscribed ports – they have at least a 1Gbps full-duplex connection to the switch backplane.

The show interface {intf-type intf-num} capabilities EXEC command can be used to determine whether a port supports the send and/or receive flow control features.

An example of modules that includes oversubscribed ports is the 18-Port 1000BASE-X module – WS-X4418-GB for the Catalyst 4000 / 4500 Series, which includes 2 uplink 1000BASE-X ports and 16 oversubscribed 1000BASE-X ports.

Internal Bandwidth Allocation for WS-X4418-GB

On the Catalyst 4000 / 4500 Series switch, each module is provided with 3 x 2Gbps full-duplex connections to the switching fabric.
The figure above shows how bandwidth is allocated internally for the WS-X4418-GB module.
Port 1 and Port 2 are uplink ports – they are each allocated 1Gbps full-duplex bandwidth (non-blocking) to the switching backplane. The receive buffers on these ports never experience congestion, as the ingress bandwidth is same as the egress bandwidth.
Even if frames received on an uplink port are eventually switched out a lower speed Fast Ethernet port, congestion occurs at the egress port, as the frames received have been emptied from the receive buffer of the uplink port and switched to the transmit buffer of the egress port.
Hence, the uplinks ports have no requirement for the ability to send flow control pause frames.
Ports 3 – 10 and ports 11 – 18 are oversubscribed ports – the total bandwidth of each group of ports (8Gbps) is shared among a 2Gbps connection to the backplane (blocking is possible).
The oversubscription rate is 4:1 – if all ports in a group are connected and receiving traffic at the maximum possible rate of 1Gbps, the bandwidth allocated to each port is only 250Mbps, causing congestion on the receive buffers of each port. Therefore, the oversubscribed ports must be able to send flow control pause frames in order to signal congestion to transmitting devices.

Note that flow control should not be configured in conjunction with QoS features on the Catalyst 2950 and Catalyst 3550 switches – all QoS features must be disabled globally before configuring flow control. This restriction does not apply to the Catalyst 4000 / 4500 and Catalyst 6000 / 6500 switches.

QoS features are disabled globally on Catalyst 2950 / 3550 switches by default.
However, it is a good practice to verify that QoS is disabled and disable QoS globally if required.
The no mls qos global configuration command is used to disable QoS globally.
Switch#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#no mls qos
Switch(config)#do sh mls qos
QoS is disabled

Switch(config)#mls qos
QoS: ensure flow-control on all interfaces are OFF for proper operation.
Switch(config)#do sh mls qos
QoS is enabled


The flow control capabilities are exchanged during the port negotiation process.
An interface can be configured to process pause frames received and/or send pause frames based on the flow control capabilities indicated by the remote device.
The flowcontrol {receive | send} {on | off | desired} interface subcommand configures an interface whether responds to pause frames received and generate pause frames when congestion occurs (if output flow control is supported).
The receive keyword is used to enable or disable a port responding to pause frames.
The send keyword is used to enable or disable the sending of pause frames when congestion occurs.
The on and off keywords enable or disable the send or receive feature respectively.
The desired keyword configures an interface to enable the send or receive feature only if the remote device indicates the support of the input or output flow control capability during port negotiation.

Note: On the Catalyst 2950 / 3550 switches, the receive feature is set to off and the send feature is set to desired for Gigabit Ethernet ports by default. The receive feature is set to off as the Catalyst 3550 is a non-blocking switch; if congestion does occur, it normally happens at the transmit queue of a lower-speed port, not at the receive queue (buffer) of a Gigabit Ethernet port.
For Fast Ethernet ports, the receive and send features are set to off by default.

Below demonstrates configuring a Gigabit Ethernet interface to enable the send and receive features only if the remote switch indicates the support of flow control capability during port negotiation.
Switch#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#int gi0/1
Switch(config-if)#flowcontrol send desired
Switch(config-if)#flowcontrol receive desired
Switch#sh int gi0/1
GigabitEthernet0/1 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 0011.9238.bb99 (bia 0011.9238.bb99)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX
  input flow-control is off, output flow-control is on
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:03, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
--- output omitted ---
Note that the shaded line indicates that the input (receive) flow control is off, which is due to the fact that it is connected to a Gigabit Ethernet interface that does not support output (send) flow control, eg: uplink port of a Catalyst 4000 / 4500 supervisor or module, or has disabled output flow control. The output (send) flow control is enabled, which is due to the fact that the remote switch port has indicated the support of input flow control during the port negotiation process.

No comments:

Post a Comment