MTU size recommendations for LHCONE

Problem description

The use of Jumbo frames (i.e. Ethernet frames bigger than the standard 1500 Bytes) usually improves the performance of data transfer, especially over long distances. However, in order to achieve these improvements, the larger frames need to be allowed to cross all the network segments without getting fragmented or, even worse, being dropped.
Jumbo frames are in fact plagued by these issues:

• Fragmentation by intermediate network devices (only for IPv4): if a router has to send jumbo frames into a connection with smaller MTU size, it will have to fragment the packets in smaller chunks. This operation may pose some stress on the router and reduce performances. In addition, fragments, after the first, don't have UDP port information in their header and may be dropped by border firewalls.

• Blackholing: if two routers connected at the end of a link don't have the same MTU settings, the router with the smaller MTU may drop bigger frames as Giant packets.
• Blackholing may also happens if the Path MTU discovery Protocol doesn't work. This protocol relies on ICMP Fragmentation Needed packets, sent back from the routers that would need to fragment. Unfortunately these packets are often filtered by firewalls, making impossible for end hosts to adjust the size of their packets

The blackholing issue is especially nasty to debug because ping and TCP connections may work because they use small packets; but as soon some data transfer starts, it hangs because all the "Giant" packets are dropped. In addition, there is no standard size for Jumbo frames, which is a word that describe any packet bigger than 1500 Bytes.

MTU size recommendations for LHCONE networks

In order to avoid issues with Jumbo frames, these recommendations on MTU size are given:

• The link layer MTU must be set to the maximum supported
• IPv4 and IPv6 MTU must be set to 9000 Bytes

Path MTU discovering

The Path MTU Discovery protocols need these ICMP packets to be allowed:

• IPv4: ICMP Fragmentation Needed - Type 3, Code 4
• IPv6: ICMPv6 Packet Too Big - Type 2 (Value 0)

It's important that these packets have a routable IP address as sources, because unroutable addresses may be dropped by antispoofing filters.

Vendor specific commands

Brocade/ExtremeNetworks NetIron

Configuration

default-max-frame-size 9216

interface ve 10
 ip mtu 9000
 ipv6 mtu 9000

Check

telnet@router# show interface ve 10 | inc MTU
  Internet address is 10.0.0.1/30, IP MTU 9000 bytes, encapsulation ethernet
  ipv6 address fe80::22:23ff:fe8a:e00 link-local, IPv6 MTU 9000 bytes

telnet@router# sh int eth 1/1 | inc MTU
  MTU 9216 bytes, encapsulation ethernet

Juniper Junos

Configuration

set interfaces et-0/0/10 mtu 9216
=or=
set interfaces ae10 mtu 9216

set interfaces irb unit 10 family inet mtu 9000
set interfaces irb unit 10 family inet6 mtu 9000

Check

user@device> show interfaces irb.10 | grep MTU 
    Protocol inet, MTU: 9000
    Protocol inet6, MTU: 9000

user@device> show interfaces et-0/0/10 | grep MTU 
  Link-level type: Ethernet, MTU: 9216, LAN-PHY mode, Speed: 100Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None,
    Protocol eth-switch, MTU: 9216

Cisco