Comparing Ethernet and RapidIO

A comprehensive flow control strategy

The transmitter and the receiver have agreed on how many packets of each priority/virtual channel can be transferred. The transmitter can use this information when scheduling packet transmission to optimize fabric latency performance.

The low-latency control mechanism allows RapidIO buffer size to be minimized while guaranteeing packet delivery and achieving line rate throughput.

IEEE 802.1p priority-based flow control uses an XON/XOFF mechanism, which must be activated before it is possible for the receiver to run out of buffer space. From Figure 2, it is clear why the ability to embed link-level flow control information within packets is such a key advantage for RapidIO. At point 1, the Receiver decides that it must turn off a particular priority. However, the yellow Max Frame (point 2) has started transmission, preventing transmission of the PAUSE XOFF frame.

This allows the corresponding yellow Max Frame to be received by the Receiver. Once the PAUSE XOFF frame has been received by the Transmitter, a similar problem can occur at point 3, where a Max Frame has already started transmission. The minimum amount of information received after XOFF is transmitted, or skid, that Ethernet links must account for is therefore two maximum-sized packets.

For a 10 Gbps link 100m in length, one computation of the skid length is approximately 10.1 KB (http://www.ieee802.org). This computation assumes a maximum packet length of 2000 bytes. If other storage-related protocols, such as FCoE with a 2.5 KB Maximum Transfer Unit (MTU) or iSCSI with a 9 KB MTU, are in use, the skid length will increase. Other Ethernet initiatives, such as Power Efficient Ethernet, may increase the skid length even more.

Skid length significance

For Network Interface Cards (NICs) and processors with gigabytes of DRAM, the skid length is not significant. However, for the small embedded memories found within switch chips, the skid length is worrying. Large skid length has important implications for an Ethernet switch designer and Data Center Ethernet users:

In order to avoid sending an XON followed immediately by an XOFF, more than the skid length of buffer space should be available before turning on a flow. This implies that packet transfers for given flows will be bursty, resulting in uneven network behavior.

• There is no guarantee that the “skid” packets sent match the priority that is being turned off. Therefore, some part of the buffer reserved for skid will not contain packets, increasing latency and decreasing throughput.

• To reduce the amount of memory required for an Ethernet port, one design optimization is to group multiple priorities together for flow control purposes. This can lead to head-of-line blocking, with commensurate reductions in throughput and increases in latency for all traffic in that priority group.

Scheduling algorithms for packet transmission can be rendered ineffective by flow control.

While short-term flow control has not traditionally been Ethernet’s strength, long-term flow control has been. The software-based flow control protocols generally depend on detecting packet loss when Internet Ethernet devices discard packets due to congestion. Since DCE no longer discards packets under congestion, long term flow control is supported using the 802.1Q Virtual LAN standard.

(ID:28270470)