Understand Etherchannel Load Balancing.

Let’s try to define what EtherChannel is and why it exists nowadays as a powerful feature.In the days of spanning-tree protocol we have always a need to block all redundant links except one. We have of course enhancements like Per Vlan Spanning Tree or Multiple Spanning Tree, but still one or more redundant link for one VLAN or VLAN instance was not available for us. Etherchannel is a way to fool STP in a way that you can have up to 8 non blocked links connected from one switch to another (or from switch to server, as we can do that too).

We have 2 flavors of Etherchannel protocol. One is Cisco proprietary PAgP (Port Aggregation Protocol). One is IEEE 802.3ad standard – LACP (Link Aggregation Control Protocol). You can also simply force switch to do port channel no matter what. You may say “your Etherchannel is ON and I do not care what is configured on the other side” (which should be forbidden in my opinion as it screams “I WANT A LOOP IN THE NETWORK”).

Now, the common mistake is the assumption that in the Etherchannel load would be equally distributed across all links bundled in Etherchannel. The assumption is wrong because of 2 reasons. First is if we have 3, 5, 6, or 7 ports bundled in one channel. Second is because the balancing is done based on flows, not based on packets. We will go back to this later on.

Etherchannel can have up to 8 ports, so number of links like 3, 5, 6 and 7 would never have equal distribution. Here is why and this is point where we are not able to influence:

Number of Ports in the EtherChannel Distribution across the links
2 50%:50%
3 37,5%:37,5%:25%
4 25%:25%:25%:25%
5 25%:25%:25%:12,5%:12,5%
6 25%:25%:12,5%:12,5%:12,5%:12,5%
7 25%:12,5%:12,5%:12,5%:12,5%:12,5%:12,5%
8 12,5%:12,5%:12,5%:12,5%:12,5%:12,5%:12,5%:12,5%

The switch is using per flow balancing because this is the way how hardware works. When switch receive the packet it would make a hash result from fields located in the header line source/destination MAC address, source/destination IP address or source/destination port number. From which fields our device would make a hash depends on certain ASIC. Not all switches were made equally. Software is using those hashes to make a decision on which port (Phisical port number) send the packet out to another device in the path. Hashes are also used to balance the traffic across etherchannel. By default Layer 2 packets are distributed on XOR computation of source and destination MAC address and Layer 3 packets based on XOR source and destination IP address:

Cat6500#show etherchannel load-balance
 EtherChannel Load-Balancing Configuration:
 mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
 Non-IP: Source XOR Destination MAC address
 IPv4: Source XOR Destination IP address

OK, we have our default load balancing algorithm in place. What other options do we have?

 Cat6500(config)#port-channel load-balance ?
dst-ip                   Dst IP Addr
dst-mac                Dst Mac Addr
dst-mixed-ip-port      Dst IP Addr and TCP/UDP Port
dst-port               Dst TCP/UDP Port
mpls                   Load Balancing for MPLS packets
src-dst-ip             Src XOR Dst IP Addr
src-dst-mac            Src XOR Dst Mac Addr
src-dst-mixed-ip-port  Src XOR Dst IP Addr and TCP/UDP Port
src-dst-port           Src XOR Dst TCP/UDP Port
src-ip                 Src IP Addr
src-mac                Src Mac Addr
src-mixed-ip-port      Src IP Addr and TCP/UDP Port
src-port               Src TCP/UDP Port

Now we can see that we have got the choice. We need however to remember that it depends on switch how many algorithms we could use. Why we need the choice? Does the default configuration is not good enough? In most cases you would not notice if your network is not under stress. The important point to understand is that this knowledge can influence your design. If you have 2 devices (i.e. PC and NAS) communicating with each other through one switch adding more links to eliminate the congestion would not work. When devices are talking using different port numbers than you can influence link usage changing the algorithms to one which include port number in the decision, but in this particular case is simply better to change the link to one with higher speed (or verify if you can use different port on non-oversubscribe card / ASIC).

Can we know which port is used by the algorithm to balance the traffic? Yes and now I would show you how. As an example I would use Cisco Catalyst 6500 switch (with sup 720) as it is the most complicated, because the results are given in RBH value which stands for Result Bundle Hash.

At first we need to go to switch processor with “remote login switch” command. Now, as the console changed from Cat6500# to Cat6500-sp# we can start our magic and test how the traffic is forwarded through our port-channel 5 containing 2 ports. The command is nice and simple:

test etherchannel load-balance interface port-channel 5 

We are testing which link would be used if switch receives packet from source to destination when packet would be leaving device. Now if your are lucky and have recent IOS version this would be the output that you would like to have:

 Computed RBH: 0x3
  Would select Te2/2 of Po5

In older IOS the result is:

Computed RBH: 0x3
Now we know everything right? We need to “exit” 
switch processor console and look at the command:

 show interfaces port-channel 5 etherchannel
 Port-channel5   (Primary aggregator)
Age of the Port-channel   = 0d:00h:08m:15s
Logical slot/port   = 14/1          Number of ports = 2
HotStandBy port = null
Port state          = Port-channel Ag-Inuse
Protocol            =   LACP
Fast-switchover     = disabled
Load share deferral = disabled

Ports in the Port-channel:
Index   Load      Port          EC state       No of bits
0      55          Te2/1             Active   4
1      AA          Te2/2             Active   4

We need to look at the bolded values (55 or AA). It is time for math. 55 is 0101 0101 in binary. It means that for interface te2/1 bits 0, 2, 4 and 6 are set. So RBH value of 0, 2, 4, and 6 belongs to te2/1. Our RBH value is 0x3, so the chosen port would be te2/2. Let’s check it out. AA is 1010 1010 in binary, so for te2/2 bits 1, 3, 5 and 7 are set, than RBH value of 1, 3, 5, and 7 belongs to port te2/2.

In the end let me just share with you command to verify transmitters of Catalyst 4500 and 2k/3k respectively:

show platform software etherchannel port-channel 5 
map ip
test etherchannel load-balance interface port-channel 1

Hope you were enjoy reading:)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s