LHCOPN - IP parameters, addresses and routing

Scope

This document gives some guidelines about the configuration of the LHCOPN network. It focuses on IP parameters, addresses and routing aspects; it uses and complements the instructions already provided by the LCG Network Architecture document. It also reports all the decisions taken by the LHCOPN routing working group.

Links' MTU

These are the recommended settings for the LHCOPN links:

• IP MTU = 9000 Bytes
• Ethernet MTU = 9026 Bytes (to support vlan tag and QinQ)
• ICMP unreachable = enabled to allow Path MTU Discovery protocol

These are recommendations only; the pairing site can decide any setting concerning MTU and IP MTU

IP addresses

• Use of public IP addresses: Every Tier must assign publicly routable IP addresses to the machines that need to be reached over the T0-T1 links; these IP networks are referred as " LHC prefixes ". A LHC prefix cannot be a RFC1918 network.
• Aggregation in few prefixes: every Tier must aggregate the address space dedicated to the LHCOPN traffic into one or few CIDR blocks.
• Addresses for the T0-T1 links: the network 192.16.166.0/24 has been allocated for the addressing of the point-to-point links between the T0 and the Tier1s. Please refer to LHCopnTables#AnchorASnumbersLHCprefixes for the allocation of the /30.
• Security: for security reason, only packets with source and destination IP address that belong to one of the LHCOPN prefixes can transit in the LHCOPN. Please refer to the Security policies document for more information about security in the LHCOPN.
• LHCOPN prefixes repository: the list of the LHCOPN prefixes is saved in the RIPE route-set object RS-LHCOPN. The object is mantained by CERN; all the request for modification must be sent to extip@cernNOSPAMPLEASE.ch.

Routing

• Topology: the LHCOPN is essentially a star with the Tier0 at the centre. Connections between pairs of T1s are also admitted.
  
• Routing: routing is ensured by BGP (see below); usage of static routes and/or default route is discouraged.
• Global connectivity: machines connected to the LHCOPN might need global connectivity. Sites that don't use dedicated routers for the LHCOPN can take advantages of Policy Based Routing to correctly steer only the LHC traffic. Every Tier is responsible to correctly route its traffic towards the correct upstream.
• T1 to T1 transit via T0: data transfers between T1 centres transiting to the T0 is technically feasible. However, at the time of writing there is not such request and it is not implemented.
  
• T0-T2 traffic: at the time of writing it is not allowed inside the LHCOPN.

Backup connectivity

There are severeal options for backup connectivity:

• Direct lightpath: a second direct lightpath between the T1 and the T0.
  
• Backup via T1s: a T1 can take advantage of its direct link to another T1 to reach the T0.
  
• Backup of last resort: backup via Layer 3 paths across NRENs and Research Backbones. This option is discouraged as it might heavily interfere with the traffic normally flowing through those backbones. However it is considered acceptable, especially in the warm-up phase of the LHCOPN.
  Traffic sent to Geant2-IP should be marked for the LBE service (DSCP=8, TOS=0x20, see here and rfc4594 for details).

BGP setup

BGP is the routing protocol that manage the routing in the LHCOPN. External BGP peerings are established among the T0 and the T1s.

• T0's BGP speakers: the T0's BGP speakers are two routers connected to the CERN's LCG backbone and terminating all the T0-T1 links.
• T1's BGP speaker: the T1's BGP speaker is the router that terminates the T0-T1 link in the T1 side.

Primary connectivity between the T0 and every T1

• AS number: An Autonumous System number is necessary to establish eBGP peers. Tiers need to use a valid public AS number; if they don't have one, they should contact their upstream NREN or their LIR to obtain one. The list of AS numbers used is in LHCopnTables#AnchorASnumbersLHCprefixes.
• T1's announces to the T0 : every T1 announces its own LHC prefixes.
  
• T0 announces to every T1: the T0 announces its own LHC prefixes.
  
• Prefixes accepted by the T1: a T1 must accepts the T0's prefixes.
• Prefixes accepted by the T0: the T0 accepts only the LHC prefixes related to the peering T1.

Backup connectivity

In case of a T1 with two direct lightpath to the T0:

• Metrics: the T0 and the T1 will set appropriate BGP MED values in order to prefer the main connection rather than the backup one.
• Annunces: except for the metrics, the announces are the same on the primary and the backup link.

In case of a T1 with a direct link to another T1:

• Peering: The two directly connected T1s must establish an eBGP peering over the direct link
• T1's announces to the peering T1: every T1 announces its own prefixes, the T0's prefixes and all the T1's prefixes received from other T1 peerings.
• T1's announces to the T0: every T1 announces its own prefixes together with all the T1s' prefixes received.
• Prefixes accepted by the T1 from the T1: each T1 must accept all the prefixes announced by a peering T1.
• Prefixes accepted by the T0: the T0 must accept all prefixes the T1s announce.

In case of backup via generic Internet [not recommended]:

• Announces to the generic Internet: T0 and T1s have to announce their LHC prefixes to their upstream networks (GÉANT2, Abilene, ESnet, for instance). Most probably the LHC prefixes are part of networks already announced; in this case there is no need to announce the more specific LHCOPN prefix.
  
• Special care must be taken by each Tier to not leak out BGP prefixes that belong to other Tiers.

T1-T1 transit via the T0

• T0 announces to every T1: The T0 announces all the T1s' prefixes received that are not tagged with the no-export community.
• T0's announces accepted by the T1: each T1 decides which prefixes accept or refuse according to its routing policy.

CERN BGP Communities

CERN LHCOPN routers honour these user communities:

• 513:64705  Set MED=5 (best)
• 513:64710  Set MED=10 (default for prefixes from primary peerings)
• 513:64750  Set MED=50
• 513:64790  Set MED=90 (default for prefixes from backup peerings)
• 513:ASN    Do not announce to ASN (for example: a prefix tagged with 513:43 513:137 won't be announced to AS43 nor to AS137)
  • 32 bits ASNs needs Large BGP Communities (RFC8092).
    • PL-NCBJ AS198743 uses community large:513:198743:0
    • NDGF-LHEP AS216467 uses community large:513:216467:0

References

LCG Network Architecture document v2.0

LHCOPN Twiki pages

Glossary

AS: Autonomous System
CIDR: Classless Inter-Domain Routing (cfr. RFC 1518)
LBE: Less than Best Effort
T0: Tier0
T1: Tier1
T2: Tier2