Currently:

Disk servers and tape servers are individual machines connected via a Gigabit Ethernet line. Current disk servers can internally deliver up to 300MB/s and soon-to-come tapes are expected to be above 100MB/s.

Proposed ideas for changes:

• Multiple GE lines per machine - use either 802.3ad link aggregation (needs switches which support it) or use multiple IP addresses plus policy routing
• 10Gbase-SR (fibre) or 10Gbase-CX4 (copper) ten-gigabit ethernet lines (fibre is expensive; copper is limited in length)
• use fibre channel (up to 2Gbit/s; faster (than 2Gb/s) versions are incompatible with the slower ones)
• use Infiniband (no expertise, immature, same distance limitation as with 10Gbase-CX4)

Foreseen problems - multiple GEs

On the Linux side, one would use the 'bonding' driver (see bonding.txt in the Linux kernel tree). Normally, one would opt for 802.3ad and let the switches know about our intentions. The load balancing (chosing of sub-ingerface) will then be based on the local-remote MAC address pairs.

However, the CERN network is not a single Ethernet segment but filled with routers using Proxy ARP, faking their own MAC address for source addresses. Therefore, whole subnets' traffic would seem to originate from a single MAC address (those behind the router) and therefore always the same interface be chosen for the traffic. And this basically kills this solution, even if the 3com leaf switches spoke 802.3ad (which they don't, "decent" switches would be needed in any case).

Another option would be balance-alb (or mode 6) of the bonding driver. While we know that there had been successful experiments at CASPUR with this I consider the solution awkward and complicated (again see bonding.txt). In my opinion this would introduce too much complexity when it comes to debugging - and there's no guarantee that it wouldn't be killed by the proxy arp problem described above.

-- AndrasHorvath - 25 Jul 2005