LHCOPN backup links test
Test of the 9th April 2008
Summary
The summary of the test has been moved HERETable of Content
Schedule
Wednesday 9th of April 2008- 13:00 TW-ASGC
- 13:15 IT-INFN-CNAF
- 13:30 FR-CCIN2P3
- 13:45 DE-KIT
- 14:15 NL-T1
- 14:35 US-FNAL-CMS
- 14:45 US-T1-BNL
- 14:30 CA-TRIUMF
- 14:45 shutdown CERN router 1 (BNL, TRIUMF, ASGC, NDGF, RAL, KIT, PIC)
- 15:15 shutdown CERN router 2 (FNAL, IN2P3, CNAF, NL-T1)
- 15:45 outages in SURFnet
- 16:15 to 17:00. Time available for other tests.
Reachable hosts
| Site | Reachable target |
|---|---|
| CA-TRIUMF | lcgvm.triumf.ca (206.12.1.13) |
| CH-CERN | cs-enmon-1.cern.ch (128.142.165.138) |
| DE-KIT | lhc-opn-mon-fzk.gridka.de (192.108.47.28) |
| ES-PIC | netflow01.pic.es (193.109.174.10) or netmon01.pic.es (193.109.174.21) |
| FR-IN2P3 | ccxfert01.in2p3.fr (193.48.99.201) |
| IT-INFN-CNAF | mastercr.cnaf.infn.it (131.154.128.2) |
| NDGF | |
| NL-T1 | 145.100.32.1 |
| TW-ASGC | 140.109.98.1 (to be confirmed) |
| UK-T1-RAL | 130.246.179.245 (to be confirmed) |
| US-CMS-FNAL | cmswn300.fnal.gov (131.225.206.185) |
| US-T1-BNL | dct00.usatlas.bnl.gov (192.12.15.8) |
Tools
- CERN looking glass
(query R01LCG and R02LCG)
Monitoring
- Interfaces statistics
- BGP routing status
- E2ECU link status
- Gridview
- Atlas dashboard
- Perfsonar for LHCOPN
Misc
Results
BGP monitoring and E2EMON
Primary links failure test
TW-ASGC
Normal RTT:312ms
Backup RTT: 302
Backup reaction time: 1'10''
Normal Traceroute T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.333ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 11.341ms 2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 6.274ms 3: cern-1.r0.ams.asgc.net (192.16.166.2) asymm 4 16.086ms 4: 117.103.111.206 (117.103.111.206) asymm 5 312.309ms 5: 140.109.98.1 (140.109.98.1) 313.436ms reachedT1-T0
not availableBackup Traceroute (primary link down)
T0-T1 (goes via the peering at Starligth)
1: cs-enmon-1.cern.ch (128.142.165.138) 0.328ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 0.710ms 2: cern-bk.r0.chi.asgc.net (192.16.166.98) asymm 3 113.505ms 3: 117.103.111.210 (117.103.111.210) asymm 4 301.864ms 4: 140.109.98.1 (140.109.98.1) 321.398ms reachedT1-T0
not availableBackup Traceroute (primary and secondary link down)
T0-T1 (goes via NL-T1)
1: cs-enmon-1.cern.ch (128.142.165.138) 0.333ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 4.216ms 2: te3-4.sara-r1.sara.nl (192.16.166.74) 25.106ms 3: 117.103.111.186 (117.103.111.186) asymm 9 118.578ms 4: 117.103.111.206 (117.103.111.206) asymm 9 311.922ms 5: 140.109.98.1 (140.109.98.1) asymm 9 312.954ms reachedT1-T0
not available
IT-INFN-CNAF
Normal RTT:10.8ms
Backup RTT: 22.3ms
Backup reaction time: 1''
Normal Traceroute T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.338ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 10.095ms 2: lhcopn-cnaf.cern.ch (192.16.166.18) asymm 3 7.791ms 3: no reply 4: no reply 5: mastercr.cnaf.infn.it (131.154.128.2) 11.061ms reachedT1-T0
BD-10808.2 # traceroute 128.142.165.138 from 131.154.128.1 traceroute to 128.142.165.138, 30 hops max 1 193.206.128.29 18 ms 2 193.206.134.193 10 ms 3 192.16.166.17 17 ms 4 128.142.165.138 11 msBackup Traceroute
T0-T1 (via KIT)
1: cs-enmon-1.cern.ch (128.142.165.138) 0.346ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 3.777ms 2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 4.218ms 3: r-inet-gis-i-b7-2-cern.gridka.de (192.16.166.34) 12.534ms 4: 192.108.46.158 (192.108.46.158) 12.548ms 5: lhcopn-cnaf-gridka.cern.ch (192.16.166.102) asymm 6 19.248ms 6: no reply 7: no reply 8: mastercr.cnaf.infn.it (131.154.128.2) 22.418ms reachedT1-T0
not available
FR-CCIN2P3
Normal RTT:2.8ms
Backup RTT: 24.9ms
Backup reaction time: 3'05'' (possible issue here!!)
Normal Traceroute T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.335ms pmtu 1500
1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 0.791ms
2: lhcopn-in2p3.cern.ch (192.16.166.42) 3.328ms
3: lhcopn-in2p3.cern.ch (192.16.166.42) asymm 2 3.328ms !H
Resume: pmtu 1500
T1-T0
[phynet@ccxfert01 phynet]$ traceroute 128.142.165.138 traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 38 byte packets 1 Lyon-OPN (193.48.99.100) 0.341 ms 0.176 ms 0.240 ms 2 l513-c-rftec-2-be7.cern.ch (192.16.166.41) 14.735 ms 2.976 ms 14.610 ms 3 cs-enmon-1.cern.ch (128.142.165.138) 2.867 ms 2.859 ms 2.868 ms*Backup Traceroute*
T0-T1
emartell@hump> tracepath 193.48.99.201 1: cs-enmon-1.cern.ch (128.142.165.138) 0.391ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 10.026ms 2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 11.547ms 3: r-inet-gis-i-b7-2-cern.gridka.de (192.16.166.34) 12.393ms 4: lhcopn-in2p3-gridka.cern.ch (192.16.166.110) 25.359ms 5: lhcopn-in2p3-gridka.cern.ch (192.16.166.110) asymm 4 25.472ms !HT1-T0 (via KIT)
traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 38 byte packets 1 Lyon-OPN (193.48.99.100) 0.338 ms 0.182 ms 1.114 ms 2 r-inet-gis-ii-b3-2-in2p3.gridka.de (192.16.166.109) 14.611 ms 14.467 ms 14.485 ms 3 l513-c-rftec-1-be9.cern.ch (192.16.166.33) 32.847 ms 24.966 ms 31.599 ms 4 l513-c-rftec-2-tl1.cern.ch (192.16.166.254) 25.101 ms 35.586 ms 35.972 ms 5 cs-enmon-1.cern.ch (128.142.165.138) 24.976 ms 24.968 ms 24.979 ms
DE-KIT
Normal RTT:11.3ms
Backup RTT: 24.9ms (it was asymmetric here, T0->T1 via NL-T1, T1->T0 via IN2P3
Backup reaction time: 1''
Normal Traceroute T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.313ms pmtu 1500
1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 4.544ms
2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 4.273ms
3: r-inet-gis-i-b7-2-cern.gridka.de (192.16.166.34) 12.484ms
4: tn-msfc-vrf-gtn3-1.gridka.de (192.108.46.137) 12.662ms
5: tn-vrf-gtn41-v1477.gridka.de (192.108.46.214) 13.149ms
6: lhc-opn-mon-fzk.gridka.de (192.108.47.28) 11.540ms reached
Resume: pmtu 1500 hops 6 back 6
T1-T0
[root@f01-120-102-e ~]# traceroute -I 128.142.165.138| tee traceroute.cern.woin2p3 traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 38 byte packets 1 def-gw-datamover-47-4-r-gridX-fzk (192.108.47.193) 0.844 ms 0.589 ms 0.598 ms 2 tn-msfc-vrf-r-inet-gis-I (192.108.46.141) 2.048 ms * 1.561 ms 3 * 192.108.46.158 (192.108.46.158) 1.209 ms * 4 lhcopn-cnaf-gridka.cern.ch (192.16.166.102) 8.083 ms * 8.074 ms 5 * l513-c-rftec-2-be2.cern.ch (192.16.166.17) 32.168 ms 37.128 ms 6 cs-enmon-1.cern.ch (128.142.165.138) 26.950 ms 26.965 ms 26.947 msBackup Traceroute (primary link down)
T0-T1 (via NL-T1)
1: cs-enmon-1.cern.ch (128.142.165.138) 0.363ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 13.028ms 2: te3-4.sara-r1.sara.nl (192.16.166.74) 24.934ms 3: r-inet-gis-i-b1-1-sara.gridka.de (192.16.166.105) 28.091ms 4: tn-msfc-vrf-gtn3-1.gridka.de (192.108.46.137) 28.588ms 5: tn-vrf-gtn41-v1477.gridka.de (192.108.46.214) 29.095ms 6: lhc-opn-mon-fzk.gridka.de (192.108.47.28) 27.971ms reachedT1-T0 (via IN2P3)
traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 38 byte packets 1 def-gw-datamover-47-1-r-grid4-fzk (192.108.47.1) 1.549 ms 0.945 ms 0.725 ms 2 tn-vrf-gtn31-v1477 (192.108.46.213) 0.857 ms 0.749 ms 0.616 ms 3 tn-msfc-vrf-r-inet-gis-I (192.108.46.141) 1.117 ms * 1.713 ms 4 * lhcopn-in2p3-gridka.cern.ch (192.16.166.110) 14.326 ms * 5 l513-c-rftec-2-be7.cern.ch (192.16.166.41) 32.929 ms *Icmp checksum is wrong 6 cs-enmon-1.cern.ch (128.142.165.138) 27.754 ms 27.685 ms 27.713 ms*Other Backup Traceroutes*
T0-T1 via CNAF
1: cs-enmon-1.cern.ch (128.142.165.138) 0.324ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 6.569ms 2: lhcopn-cnaf.cern.ch (192.16.166.18) asymm 3 7.818ms 3: r-inet-gis-ii-b2-2-cnaf.gridka.de (192.16.166.101) 16.673ms 4: tn-msfc-vrf-gtn3-1.gridka.de (192.108.46.137) asymm 5 16.419ms 5: tn-vrf-gtn41-v1477.gridka.de (192.108.46.214) asymm 6 16.741ms 6: lhc-opn-mon-fzk.gridka.de (192.108.47.28) asymm 7 15.534ms reached
NL-T1
Normal RTT:24.8ms
Backup RTT: 25.6
Backup reaction time: 1''
Normal Traceroute T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.290ms pmtu 1500
1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 1.712ms
2: te3-4.sara-r1.sara.nl (192.16.166.74) 25.128ms reached
Resume: pmtu 1500 hops 2 back 2
T1-T0
sara-r1#traceroute vrf lhcopn ip 128.142.165.138 1 l513-c-rftec-2-be3.cern.ch (192.16.166.73) [AS 1103] 32 msec 32 msec 36 msec 2 cs-enmon-1.cern.ch (128.142.165.138) [AS 1103] 24 msec 24 msec 24 msecBackup Traceroute (via KIT)
T0-T1 (to SARA)
1: cs-enmon-1.cern.ch (128.142.165.138) 0.357ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 3.460ms 2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 2.821ms 3: r-inet-gis-i-b7-2-cern.gridka.de (192.16.166.34) 12.639ms 4: 192.108.46.158 (192.108.46.158) 12.670ms 5: lhcopn-sara-gridka.cern.ch (192.16.166.106) 25.949ms reachedT0-T1 (to NIKHEF)
1: cs-enmon-1.cern.ch (128.142.165.138) 0.320ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 4.985ms 2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 11.991ms 3: r-inet-gis-i-b7-2-cern.gridka.de (192.16.166.34) 13.529ms 4: 192.108.46.158 (192.108.46.158) 12.796ms 5: lhcopn-sara-gridka.cern.ch (192.16.166.106) 25.743ms 6: no reply 7: tbn18.nikhef.nl (194.171.96.129) 25.781ms reachedT1-T0 (DE-KIT issue)
sara-r1#traceroute vrf lhcopn ip 128.142.165.138 1 r-inet-gis-i-b1-1-sara.gridka.de (192.16.166.105) [AS 1103] 16 msec 16 msec 16 msec 2 * * * 3 * * *
US-FNAL-CMS
Normal RTT:114ms
Backup RTT: 115ms
Backup reaction time: 1''
Normal Traceroute T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.320ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 9.457ms 2: lhcopn-fnal.cern.ch (192.16.166.26) 114.527ms 3: no reply 4: cmswn300.fnal.gov (131.225.206.185) 114.437ms reachedT1-T0
traceroute to cs-enmon-1.cern.ch (128.142.165.138), 30 hops max, 46 byte packets 1 vlan207-rgw1.r-cms-fcc2.fnal.gov (131.225.207.205) 0.713 ms 0.269 ms 1.024 ms 2 te4-2.r-s-starlight-fnal.fnal.gov (198.151.133.238) 7.137 ms 0.442 ms 1.002 ms 3 l513-c-rftec-2-ve1.cern.ch (192.16.166.25) 118.497 ms 124.882 ms 126.285 ms 4 cs-enmon-1.cern.ch (128.142.165.138) 114.185 ms 114.181 ms 114.203 msBackup Traceroute
T0-T1
martell@hump> tracepath 131.225.206.185 1: cs-enmon-1.cern.ch (128.142.165.138) 0.338ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 0.762ms 2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 0.868ms 3: lhcopn-fnal-bu.cern.ch (192.16.166.30) 115.222ms 4: no reply 5: cmswn300.fnal.gov (131.225.206.185) 115.148ms reachedT1-T0
root@cmswn300 ~]# traceroute cs-enmon-1.cern.ch traceroute to cs-enmon-1.cern.ch (128.142.165.138), 30 hops max, 46 byte packets 1 vlan207-rgw1.r-cms-fcc2.fnal.gov (131.225.207.205) 0.302 ms 0.522 ms 1.048 ms 2 te4-2.r-s-starlight-fnal.fnal.gov (198.151.133.238) 0.925 ms 0.605 ms 1.001 ms 3 192.16.166.29 (192.16.166.29) 127.890 ms 125.053 ms 125.853 ms 4 l513-c-rftec-2-tl1.cern.ch (192.16.166.254) 119.134 ms 125.105 ms 125.975 ms 5 cs-enmon-1.cern.ch (128.142.165.138) 114.845 ms 115.090 ms 114.808 msBackup Traceroute (both LHCOPN links down, path via ESnet)
T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.147ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 10.636ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 6.859ms 3: b513-b-rfte6-1-pl1.cern.ch (194.12.131.69) 8.443ms 4: g513-e-rci76-1-pg1.cern.ch (194.12.144.9) 1.244ms 5: e513-e-rci76-1-pe1.cern.ch (192.65.184.161) 0.752ms 6: e513-e-rci76-2-ne1.cern.ch (192.65.184.54) 54.463ms 7: chi-gev122-cern.es.net (198.125.140.89) 113.669ms 8: chislsdn1-chislmr1.es.net (134.55.219.25) asymm 9 113.392ms 9: fnalmr1-chislsdn1.es.net (134.55.217.77) 114.641ms 10: te4-2-esnet.r-s-bdr.fnal.gov (198.49.208.230) 114.898ms 11: vlan360.r-s-hub-fcc.fnal.gov (131.225.15.78) 114.941ms 12: s-cms-fcc2.fnal.gov (131.225.15.54) 114.758ms 13: cmswn300.fnal.gov (131.225.206.185) asymm 11 114.573ms reachedT1-T0
[root@cmswn300 ~]# traceroute cs-enmon-1.cern.ch traceroute to cs-enmon-1.cern.ch (128.142.165.138), 30 hops max, 46 byte packets 1 vlan207-rgw1.r-cms-fcc2.fnal.gov (131.225.207.205) 0.356 ms 0.513 ms 1.027 ms 2 te4-2.r-s-starlight-fnal.fnal.gov (198.151.133.238) 0.935 ms 0.551 ms 1.002 ms 3 vlan1802.r-s-starlight-fnal.fnal.gov (198.151.133.185) 1.470 ms 1.426 ms 1.417 ms 4 chislmr1-chislsdn1.es.net (134.55.219.26) 1.555 ms 1.522 ms 1.499 ms 5 ar3-chicago-esnet-vlan.cern.ch (198.125.140.90) 114.482 ms 114.262 ms 114.277 ms 6 e513-e-rci76-1-ne1.cern.ch (192.65.184.53) 114.303 ms 114.342 ms 114.289 ms 7 * * * 8 * cs-enmon-1.cern.ch (128.142.165.138) 114.889 ms 114.820 ms
US-T1-BNL
Normal RTT: 96ms Backup RTT: 135ms Backup reaction time:1''
Normal Traceroute T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.320ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 0.743ms 2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 0.864ms 3: lhcopn-bnl.cern.ch (192.16.166.10) 96.028ms 4: dct00.usatlas.bnl.gov (192.12.15.8) 96.066ms reachedT1-T0
not availableBackup Traceroute
T0-T1 (wrong because via Triumf, due to a BGP issue)
emartell@hump> tracepath 192.12.15.8 1: cs-enmon-1.cern.ch (128.142.165.138) 0.334ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 0.738ms 2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 0.824ms 3: lhcopn-triumf.cern.ch (192.16.166.82) 181.422ms 4: no reply 5: dct00.usatlas.bnl.gov (192.12.15.8) asymm 3 200.042ms reachedT0-T1 (OK, BGP "route-map in" re-applied on r01lcg.cern.ch)
emartell@hump> traceroute 192.12.15.8 traceroute to 192.12.15.8 (192.12.15.8), 30 hops max, 40 byte packets 1 l513-c-rftec-2-ip501 (128.142.165.129) 2.964 ms 2 lhcopn-bnl-bu (192.16.166.14) 135.228 ms 3 dct00.usatlas.bnl.gov (192.12.15.8) 135.143 msT1-T0 (always correct).
[bigrow@netmon ~]$ traceroute cs-enmon-1.cern.ch traceroute to cs-enmon-1.cern.ch (128.142.165.138), 30 hops max, 38 byte packets 1 hsrp (192.12.15.24) 0.414 ms 0.304 ms 0.362 ms 2 192.16.166.13 (192.16.166.13) 200.087 ms 206.566 ms 199.963 ms 3 cs-enmon-1.cern.ch (128.142.165.138) 199.865 ms 199.849 ms 199.868 ms
CA-TRIUMF
Normal RTT:181ms
Backup RTT: 172ms (1G link), 178ms (via BNL)
Backup reaction time: 1'
Normal Traceroute T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.370ms pmtu 1500
1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 11.301ms
2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 0.834ms
3: lhcopn-triumf.cern.ch (192.16.166.82) 181.410ms
4: lcgvm.triumf.ca (206.12.1.13) asymm 5 181.498ms reached
Resume: pmtu 1500 hops 4 back 5
T1-T0
[root@lcgvm ~]# traceroute 128.142.165.138 traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 40 byte packets 1 atlas-gw-new (206.12.1.252) 2.788 ms 2.801 ms 2.808 ms 2 atlas-gw (206.12.1.254) 0.303 ms 0.374 ms 0.445 ms 3 l513-c-rftec-1-be6.cern.ch (192.16.166.81) 193.677 ms 193.701 ms 193.680 ms 4 l513-c-rftec-2-tl1.cern.ch (192.16.166.254) 181.416 ms 181.432 ms 181.412 ms 5 cs-enmon-1.cern.ch (128.142.165.138) 181.256 ms 181.234 ms 181.256 msBackup Traceroute
T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.370ms pmtu 1500 1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 5.278ms 2: lhcopn-triumf-bu.cern.ch (192.16.166.86) 172.316ms 3: lcgvm.triumf.ca (206.12.1.13) asymm 4 172.369ms reachedT1-T0
[root@lcgvm ~]# traceroute 128.142.165.138 traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 40 byte packets 1 atlas-gw-new (206.12.1.252) 0.485 ms 0.505 ms 0.503 ms 2 atlas-gw (206.12.1.254) 0.289 ms 0.357 ms 0.420 ms 3 l513-c-rftec-2-re6.cern.ch (192.16.166.85) 183.615 ms 183.597 ms 183.613 ms 4 cs-enmon-1.cern.ch (128.142.165.138) 172.110 ms 172.161 ms 172.140 msBackup via BNL (primary and secondary links down)
T0-T1
emartell@lxfsra3003> tracepath 206.12.1.13 1: lxfsra3003.cern.ch (128.142.161.165) 0.171ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 2.894ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 5.736ms 3: l513-c-rftec-1-bl1.cern.ch (194.12.137.14) 8.378ms 4: lhcopn-bnl.cern.ch (192.16.166.10) 95.912ms 5: no reply 6: lcgvm.triumf.ca (206.12.1.13) asymm 7 179.025ms reachedT1-T0 (-- ChrisPayne - 10 Apr 2008)
traceroute to 128.142.165.138 (128.142.165.138), 15 hops max, 40 byte packets 1 206.12.1.252 12.483 ms 12.490 ms 12.499 ms 2 206.12.1.254 0.309 ms 0.370 ms 0.432 ms 3 206.12.9.38 83.330 ms 83.390 ms 83.429 ms 4 192.16.166.9 179.401 ms 179.409 ms 179.450 ms 5 128.142.165.138 178.815 ms 178.844 ms 178.769 ms
ES-PIC
Normal RTT:31.8ms
Backup RTT: unreachable
Backup reaction time: NA
Normal Traceroute T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.365ms pmtu 1500
1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 5.781ms
2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 1.806ms
3: lhcopn-pic.cern.ch (192.16.166.58) 32.847ms
4: netflow01.pic.es (193.109.174.10) 32.058ms reached
Resume: pmtu 1500 hops 4 back 4
T1-T0
[root@netflow01 ~]# traceroute 128.142.165.138 traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 38 byte packets 1 gw0-vlan400 (193.109.174.1) 1.300 ms 0.626 ms 0.592 ms 2 l513-c-rftec-1-be12.cern.ch (192.16.166.57) 38.392 ms 35.221 ms 35.942 ms 3 l513-c-rftec-2-tl1.cern.ch (192.16.166.254) 34.848 ms 32.830 ms 35.938 ms 4 cs-enmon-1.cern.ch (128.142.165.138) 31.803 ms 31.728 ms 31.747 msBackup Traceroute
T0-T1 (din't work, the traffic was filtered)
martell@lxfsra3003> tracepath 193.109.174.10 1: lxfsra3003.cern.ch (128.142.161.165) 0.142ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 1.497ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 0.703ms 3: b513-b-rfte6-1-pl1.cern.ch (194.12.131.69) 5.034ms 4: g513-e-rci76-1-pg1.cern.ch (194.12.144.9) 1.468ms 5: e513-e-rci76-1-pe1.cern.ch (192.65.184.161) 0.831ms 6: swice2-10ge-1-2.switch.ch (192.65.184.221) 0.731ms 7: switch.rt1.gen.ch.geant2.net (62.40.124.21) asymm 8 0.679ms 8: so-7-0-0.rt1.mad.es.geant2.net (62.40.112.26) asymm 9 120.509ms 9: rediris-gw.rt1.mad.es.geant2.net (62.40.124.54) asymm 10 22.696ms 10: XE0-1-0.EB-IRIS2.red.rediris.es (130.206.250.1) asymm 11 22.750ms 11: NAC.XE1-0-0.EB-Barcelona0.red.rediris.es (130.206.250.26) asymm 12 117.446ms 12: cesca-ge.red.rediris.es (130.206.202.2) 37.768ms 13: no reply 14: no reply 15: no reply 16: cesca-ge.red.rediris.es (130.206.202.2) asymm 12 37.749ms !HT1-T0
not available
UK-T1-RAL
Normal RTT:18.4ms
Backup RTT: Unreachable
Backup reaction time: NA
Normal Traceroute T0-T1
1: cs-enmon-1.cern.ch (128.142.165.138) 0.353ms pmtu 1500
1: l513-c-rftec-2-ip501.cern.ch (128.142.165.129) 0.738ms
2: l513-c-rftec-1-tl1.cern.ch (192.16.166.253) 10.824ms
3: lhcopn-ral.cern.ch (192.16.166.66) 18.954ms
4: perfsonar.gridpp.rl.ac.uk (130.246.179.245) 18.632ms !H
Resume: pmtu 1500
T1-T0
not availableBackup Traceroute
T0-T1 (unreachable)
1: lxfsra3003.cern.ch (128.142.161.165) 0.144ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 2.259ms 2: l513-b-rftec-2-rl3.cern.ch (194.12.138.1) 9.114ms 3: b513-b-rfte6-2-sl1.cern.ch (194.12.131.73) 5.359ms 4: g513-e-rci76-1-pg2.cern.ch (194.12.144.41) 1.601ms 5: e513-e-rci76-1-pe1.cern.ch (192.65.184.161) 0.720ms 6: swice2-10ge-1-2.switch.ch (192.65.184.221) 0.586ms 7: switch.rt1.gen.ch.geant2.net (62.40.124.21) asymm 8 0.709ms 8: so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) asymm 9 8.732ms 9: so-5-0-0.rt1.ams.nl.geant2.net (62.40.112.58) asymm 10 16.212ms 10: so-4-0-0.rt1.lon.uk.geant2.net (62.40.112.138) asymm 11 26.816ms 11: po2-0-0.gn2-gw1.ja.net (62.40.124.198) 24.172ms 12: po1-1.lond-sbr5.ja.net (146.97.35.97) 24.221ms 13: lond-sbr3.ja.net (146.97.33.1) asymm 14 24.323ms 14: so-0-1-0.read-sbr1.ja.net (146.97.33.141) asymm 15 25.680ms 15: ral-1.ja.net (146.97.41.166) 26.634ms 16: no reply 17: no replyT1-T0
not available
NDGF
NDGF didn't partecipate.Furthermore, the link to NDGF was down for an outage during the maintenance. Backup path to NDGF is via Geant2 IP at the moment
CERN router 1 power off
Primary link affected: NL, TRIUMF, ASGC, NDGF, RAL, KIT, PICTW-ASGC (r1)
RTT: 305ms Traceroute T0-T11: lxfsra3003.cern.ch (128.142.161.165) 0.145ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 0.855ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 0.881ms 3: l513-c-rftec-2-rl2.cern.ch (194.12.138.6) 4.888ms 4: cern-bk.r0.chi.asgc.net (192.16.166.98) asymm 5 114.831ms 5: 117.103.111.210 (117.103.111.210) asymm 6 301.944ms 6: 140.109.98.1 (140.109.98.1) 303.047ms reachedTraceroute T1-T0
IT-INFN-CNAF (r1)
RTT:11ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.132ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 0.474ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 0.719ms 3: l513-c-rftec-2-bl2.cern.ch (194.12.137.6) 3.694ms 4: lhcopn-cnaf.cern.ch (192.16.166.18) asymm 5 7.936ms 5: no reply 6: no reply 7: mastercr.cnaf.infn.it (131.154.128.2) 11.094ms reachedTraceroute T1-T0
FR-CCIN2P3 (r1)
RTT:3ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.137ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 4.871ms 2: l513-b-rftec-2-rl3.cern.ch (194.12.138.1) 6.938ms 3: l513-c-rftec-2-rl2.cern.ch (194.12.138.6) 13.755ms 4: lhcopn-in2p3.cern.ch (192.16.166.42) 3.350ms 5: lhcopn-in2p3.cern.ch (192.16.166.42) asymm 4 3.352ms !HTraceroute T1-T0
[phynet@ccxfert01 phynet]$ traceroute cs-enmon-1.cern.ch traceroute to cs-enmon-1.cern.ch (128.142.165.138), 30 hops max, 38 byte packets 1 Lyon-OPN (193.48.99.100) 0.260 ms 0.209 ms 0.243 ms 2 l513-c-rftec-2-be7.cern.ch (192.16.166.41) 13.735 ms 2.850 ms 14.608 ms 3 cs-enmon-1.cern.ch (128.142.165.138) 2.864 ms 2.858 ms 2.867 ms
DE-KIT (r1)
RTT: 27.9ms Traceroute T0-T11: lxfsra3003.cern.ch (128.142.161.165) 0.137ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 0.502ms 2: l513-b-rftec-2-rl3.cern.ch (194.12.138.1) 5.977ms 3: l513-c-rftec-2-rl2.cern.ch (194.12.138.6) 12.469ms 4: te3-4.sara-r1.sara.nl (192.16.166.74) 25.078ms 5: r-inet-gis-i-b1-1-sara.gridka.de (192.16.166.105) 40.063ms 6: tn-msfc-vrf-gtn3-1.gridka.de (192.108.46.137) 28.836ms 7: tn-vrf-gtn41-v1477.gridka.de (192.108.46.214) 29.254ms 8: lhc-opn-mon-fzk.gridka.de (192.108.47.28) 27.973ms reachedTraceroute T1-T0
[root@lhc-opn-mon-fzk ~]# traceroute -I 128.142.165.138| tee traceroute.cern.wosara traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 38 byte packets 1 def-gw-datamover-47-1-r-grid4-fzk (192.108.47.1) 3.592 ms 1.252 ms 0.909 ms 2 tn-vrf-gtn31-v1477 (192.108.46.213) 1.232 ms 0.727 ms 0.604 ms 3 tn-msfc-vrf-r-inet-gis-I (192.108.46.141) 1.495 ms * 28.289 ms 4 lhcopn-in2p3-gridka.cern.ch (192.16.166.110) 14.222 ms 14.252 ms * 5 l513-c-rftec-2-be7.cern.ch (192.16.166.41) 32.938 ms * 37.986 ms 6 cs-enmon-1.cern.ch (128.142.165.138) 27.679 ms 27.689 ms 27.702 ms
NL-T1 (r1)
RTT:25ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.140ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 0.443ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 0.718ms 3: l513-c-rftec-2-bl2.cern.ch (194.12.137.6) 3.572ms 4: te3-4.sara-r1.sara.nl (192.16.166.74) 25.129ms reachedTraceroute T1-T0
sara-r1#traceroute vrf lhcopn Protocol [ip]: Target IP address: 128.142.165.138 Source address: 145.100.32.1 Tracing the route to cs-enmon-1.cern.ch (128.142.165.138) 1 l513-c-rftec-2-be3.cern.ch (192.16.166.73) [AS 513] 36 msec 36 msec 36 msec 2 cs-enmon-1.cern.ch (128.142.165.138) [AS 513] 36 msec 24 msec 24 msec
US-FNAL-CMS (r1)
RTT:115ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.139ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 10.918ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 5.981ms 3: l513-c-rftec-2-bl2.cern.ch (194.12.137.6) 10.360ms 4: lhcopn-fnal.cern.ch (192.16.166.26) 114.639ms 5: vlan318.r-cms-fcc2.fnal.gov (198.151.133.237) asymm 12 114.658ms 6: cmswn300.fnal.gov (131.225.206.185) 114.495ms reachedTraceroute T1-T0
US-T1-BNL (r1)
RTT:unreachable
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.143ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 0.438ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 0.739ms 3: l513-c-rftec-2-bl2.cern.ch (194.12.137.6) 7.028ms 4: no reply 5: no reply 6: no replyTraceroute T1-T0
CA-TRIUMF (r1)
RTT:179ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.149ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 0.479ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 0.696ms 3: l513-c-rftec-2-bl2.cern.ch (194.12.137.6) 3.348ms 4: lhcopn-triumf-bu.cern.ch (192.16.166.86) 172.593ms 5: lcgvm.triumf.ca (206.12.1.13) asymm 6 172.486ms reachedTraceroute T1-T0
[root@lcgvm ~]# traceroute 128.142.165.138 traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 40 byte packets 1 atlas-gw-new (206.12.1.252) 4.870 ms 4.878 ms 4.885 ms 2 atlas-gw (206.12.1.254) 0.313 ms 0.379 ms 0.426 ms 3 l513-c-rftec-2-re6.cern.ch (192.16.166.85) 172.919 ms 172.925 ms 172.946 ms 4 cs-enmon-1.cern.ch (128.142.165.138) 182.013 ms 181.993 ms 182.010 ms
ES-PIC (r1)
RTT:unreachable
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.137ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 1.636ms 2: l513-b-rftec-2-rl3.cern.ch (194.12.138.1) 6.314ms 3: b513-b-rfte6-2-sl1.cern.ch (194.12.131.73) 5.971ms 4: g513-e-rci76-1-pg2.cern.ch (194.12.144.41) 1.366ms 5: e513-e-rci76-1-pe1.cern.ch (192.65.184.161) 0.706ms 6: swice2-10ge-1-2.switch.ch (192.65.184.221) 0.608ms 7: switch.rt1.gen.ch.geant2.net (62.40.124.21) asymm 8 0.665ms 8: so-7-0-0.rt1.mad.es.geant2.net (62.40.112.26) asymm 9 22.771ms 9: rediris-gw.rt1.mad.es.geant2.net (62.40.124.54) asymm 10 22.723ms 10: XE0-1-0.EB-IRIS2.red.rediris.es (130.206.250.1) asymm 11 22.775ms 11: NAC.XE1-0-0.EB-Barcelona0.red.rediris.es (130.206.250.26) asymm 12 37.551ms 12: cesca-ge.red.rediris.es (130.206.202.2) 37.698ms 13: cesca-ge.red.rediris.es (130.206.202.2) asymm 12 37.757ms !HTraceroute T1-T0
UK-T1-RAL (r1)
RTT:unreachble
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.144ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 2.259ms 2: l513-b-rftec-2-rl3.cern.ch (194.12.138.1) 9.114ms 3: b513-b-rfte6-2-sl1.cern.ch (194.12.131.73) 5.359ms 4: g513-e-rci76-1-pg2.cern.ch (194.12.144.41) 1.601ms 5: e513-e-rci76-1-pe1.cern.ch (192.65.184.161) 0.720ms 6: swice2-10ge-1-2.switch.ch (192.65.184.221) 0.586ms 7: switch.rt1.gen.ch.geant2.net (62.40.124.21) asymm 8 0.709ms 8: so-7-2-0.rt1.fra.de.geant2.net (62.40.112.22) asymm 9 8.732ms 9: so-5-0-0.rt1.ams.nl.geant2.net (62.40.112.58) asymm 10 16.212ms 10: so-4-0-0.rt1.lon.uk.geant2.net (62.40.112.138) asymm 11 26.816ms 11: po2-0-0.gn2-gw1.ja.net (62.40.124.198) 24.172ms 12: po1-1.lond-sbr5.ja.net (146.97.35.97) 24.221ms 13: lond-sbr3.ja.net (146.97.33.1) asymm 14 24.323ms 14: so-0-1-0.read-sbr1.ja.net (146.97.33.141) asymm 15 25.680ms 15: ral-1.ja.net (146.97.41.166) 26.634ms 16: no reply 17: no replyTraceroute T1-T0
NDGF (r1)
NA
CERN router 2 power off
Primary link affected: FNAL, IN2P3, CNAF, NL-T1TW-ASGC (r2)
RTT:312ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.144ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 2.811ms 2: l513-b-rftec-2-rl3.cern.ch (194.12.138.1) 0.720ms 3: l513-c-rftec-1-rl1.cern.ch (194.12.138.14) 0.789ms 4: cern-1.r0.ams.asgc.net (192.16.166.2) asymm 5 16.063ms 5: 117.103.111.206 (117.103.111.206) asymm 6 312.319ms 6: 140.109.98.1 (140.109.98.1) 313.036ms reachedTraceroute T1-T0
IT-INFN-CNAF (r2)
RTT: 22.2ms Traceroute T0-T11: lxfsra3003.cern.ch (128.142.161.165) 0.132ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 6.951ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 4.380ms 3: l513-c-rftec-1-bl1.cern.ch (194.12.137.14) 13.262ms 4: r-inet-gis-i-b7-2-cern.gridka.de (192.16.166.34) 12.260ms 5: 192.108.46.158 (192.108.46.158) 12.279ms 6: lhcopn-cnaf-gridka.cern.ch (192.16.166.102) asymm 7 19.225ms 7: no reply 8: no reply 9: mastercr.cnaf.infn.it (131.154.128.2) 22.390ms reachedTraceroute T1-T0
FR-CCIN2P3 (r2)
RTT: 25ms Traceroute T0-T11: lxfsra3003.cern.ch (128.142.161.165) 0.138ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 23.579ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 4.329ms 3: l513-c-rftec-1-bl1.cern.ch (194.12.137.14) 13.604ms 4: r-inet-gis-i-b7-2-cern.gridka.de (192.16.166.34) 12.493ms 5: lhcopn-in2p3-gridka.cern.ch (192.16.166.110) 25.325ms 6: lhcopn-in2p3-gridka.cern.ch (192.16.166.110) asymm 5 26.170ms !HTraceroute T1-T0
traceroute to cs-enmon-1.cern.ch (128.142.165.138), 30 hops max, 38 byte packets 1 Lyon-OPN (193.48.99.100) 0.258 ms 0.208 ms 0.243 ms 2 r-inet-gis-ii-b3-2-in2p3.gridka.de (192.16.166.109) 14.863 ms 14.467 ms 14.608 ms 3 l513-c-rftec-1-be9.cern.ch (192.16.166.33) 28.976 ms 35.462 ms 35.972 ms 4 * * * 5 * * * 6 cs-enmon-1.cern.ch (128.142.165.138) 25.004 ms 25.069 ms 25.103 ms
DE-KIT (r2)
RTT:11.4ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.143ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 4.910ms 2: l513-b-rftec-2-rl3.cern.ch (194.12.138.1) 1.684ms 3: l513-c-rftec-1-rl1.cern.ch (194.12.138.14) 0.820ms 4: r-inet-gis-i-b7-2-cern.gridka.de (192.16.166.34) 12.406ms 5: tn-msfc-vrf-gtn3-1.gridka.de (192.108.46.137) 12.292ms 6: tn-vrf-gtn41-v1477.gridka.de (192.108.46.214) 12.876ms 7: lhc-opn-mon-fzk.gridka.de (192.108.47.28) 11.424ms reachedTraceroute T1-T0
[root@f01-120-102-e ~]# traceroute -I 128.142.161.165 traceroute to 128.142.161.165 (128.142.161.165), 30 hops max, 38 byte packets 1 def-gw-datamover-47-4-r-gridX-fzk (192.108.47.193) 0.816 ms 2 tn-msfc-vrf-r-inet-gis-I (192.108.46.141) 1.528 ms 3 l513-c-rftec-1-be9.cern.ch (192.16.166.33) 17.952 ms 4 * * * 5 * * * 6 lxfsra3003.cern.ch (128.142.161.165) 11.280 ms
NL-T1 (r2)
RTT:25.8ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.151ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 9.407ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 5.070ms 3: l513-c-rftec-1-bl1.cern.ch (194.12.137.14) 11.762ms 4: r-inet-gis-i-b7-2-cern.gridka.de (192.16.166.34) 12.351ms 5: 192.108.46.158 (192.108.46.158) 12.202ms 6: lhcopn-sara-gridka.cern.ch (192.16.166.106) 25.762ms reachedTraceroute T1-T0
sara-r1#traceroute vrf lhcopn ip 128.142.165.138 1 r-inet-gis-i-b1-1-sara.gridka.de (192.16.166.105) [AS 1103] 16 msec 16 msec 16 msec 2 fn-r-inet-gis-I-v736.gridka.de (192.108.46.157) [AS 1103] 16 msec 16 msec 12 msec 3 l513-c-rftec-1-be9.cern.ch (192.16.166.33) [AS 1103] 28 msec 28 msec 32 msec 4 * * * 5 * * * 6 cs-enmon-1.cern.ch (128.142.165.138) [AS 1103] 24 msec 24 msec 24 msec
US-FNAL-CMS (r2)
RTT:116ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.133ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 12.381ms 2: l513-b-rftec-2-rl3.cern.ch (194.12.138.1) 0.672ms 3: l513-c-rftec-1-rl1.cern.ch (194.12.138.14) 0.788ms 4: lhcopn-fnal-bu.cern.ch (192.16.166.30) 115.204ms 5: vlan318.r-cms-fcc2.fnal.gov (198.151.133.237) asymm 12 114.986ms 6: cmswn300.fnal.gov (131.225.206.185) 115.108ms reachedTraceroute T1-T0
[root@cmswn300 ~]# traceroute 128.142.165.138 traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 46 byte packets 1 vlan207-rgw1.r-cms-fcc2.fnal.gov (131.225.207.205) 0.745 ms 0.309 ms 1.040 ms 2 te4-2.r-s-starlight-fnal.fnal.gov (198.151.133.238) 0.828 ms 0.451 ms 0.978 ms 3 192.16.166.29 (192.16.166.29) 114.980 ms 124.901 ms 125.955 ms 4 * * * 5 * * * 6 cs-enmon-1.cern.ch (128.142.165.138) 114.987 ms 115.010 ms 114.938 ms
US-T1-BNL (r2)
RTT:95ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.140ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 8.050ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 6.409ms 3: l513-c-rftec-1-bl1.cern.ch (194.12.137.14) 8.400ms 4: lhcopn-bnl.cern.ch (192.16.166.10) 95.943ms 5: dct00.usatlas.bnl.gov (192.12.15.8) 95.864ms reachedTraceroute T1-T0
[bigrow@netmon ~]$ traceroute 128.142.165.138 traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 38 byte packets 1 hsrp (192.12.15.24) 0.418 ms 0.264 ms 0.234 ms 2 l513-c-rftec-1-ve2.cern.ch (192.16.166.9) 102.022 ms 107.783 ms 107.922 ms 3 * * * 4 * * * 5 cs-enmon-1.cern.ch (128.142.165.138) 95.857 ms 95.789 ms 95.798
CA-TRIUMF (r2)
RTT:181ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.143ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 13.341ms 2: l513-b-rftec-2-rl3.cern.ch (194.12.138.1) 1.729ms 3: l513-c-rftec-1-rl1.cern.ch (194.12.138.14) 0.839ms 4: lhcopn-triumf.cern.ch (192.16.166.82) 181.352ms 5: lcgvm.triumf.ca (206.12.1.13) asymm 6 181.424ms reachedTraceroute T1-T0
[root@lcgvm LHCOPN_tests]# traceroute 128.142.165.138 traceroute to 128.142.165.138 (128.142.165.138), 30 hops max, 40 byte packets 1 atlas-gw-new (206.12.1.252) 0.331 ms 0.364 ms 0.347 ms 2 atlas-gw (206.12.1.254) 0.323 ms 0.391 ms 0.451 ms 3 (206.12.9.38) 83.404 ms 83.386 ms 83.412 ms 4 l513-c-rftec-1-ve2.cern.ch (192.16.166.9) 178.760 ms 178.740 ms 178.819 ms 5 cs-enmon-1.cern.ch (128.142.165.138) 178.829 ms 178.767 ms *
ES-PIC (r2)
RTT:31.8
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.132ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 0.457ms 2: l513-b-rftec-1-bl3.cern.ch (194.12.137.1) 3.782ms 3: l513-c-rftec-1-bl1.cern.ch (194.12.137.14) 8.057ms 4: lhcopn-pic.cern.ch (192.16.166.58) 32.917ms 5: no replyTraceroute T1-T0
UK-T1-RAL (r2)
RTT:18.4ms
Traceroute T0-T1
1: lxfsra3003.cern.ch (128.142.161.165) 0.148ms pmtu 1500 1: l513-c-rftec-3-ip202.cern.ch (128.142.161.129) 1.619ms 2: l513-b-rftec-2-rl3.cern.ch (194.12.138.1) 0.675ms 3: l513-c-rftec-1-rl1.cern.ch (194.12.138.14) 0.788ms 4: lhcopn-ral.cern.ch (192.16.166.66) 18.905ms 5: perfsonar.gridpp.rl.ac.uk (130.246.179.245) 18.533ms !HTraceroute T1-T0
NDGF (r2)
NAPinger-PS
The perfSONAR developers were able to deploy a perfSONAR PINGER-PS measurement service at CERN right before the LHCOPN backup test. Due to coordination issues we were only able to configure measurements to those sites who publicly responded to Edoardo's request for ping addresses in a timely manor. This is a demonstration pilot of the pinger-PS service. It's purpose is to collect data that is meaningful to the LHC community for demonstrating the functionality, and to refine deployment, configuration management and visualization tools. You can access the pilot info live at: http://packrat.internet2.edu:8008/
Then click on the CERN PingER MA to get started. Note that some sites
without adequate public location info (DNS LOC or whois data) might get
get dropped in the Bermuda Triangle. 
Here are the graphs of the data during the tests.
- ca-triumf.png:
- cern-ch.png:
- de-kit.png:
- es-pic.png:
* The ES-PIC test host is performing other tasks, and running close to capacity. The short duration of stable jitter was when these other tasks were disabled during the LHCOPN backup test.
- fr-in2p3.png:
- it-infn-cnaf.png:
- us-fnal-cms.png:
- us-t1-bnl.png:
BGP routing tables
CERN router 1 - all links up
L513-C-RFTEC-1# sh ip bgp s
BGP router identifier 194.12.136.21, local AS number 513
Neighbor AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/Pfx
192.16.166.2 24167 955758 665778 13903 0 0 6d4h 4
192.16.166.10 43 662709 662289 13903 0 0 6d7h 5
192.16.166.30 3152 331441 339251 13903 0 0 6d7h 5
192.16.166.34 34878 334608 340571 13903 0 0 2w4d 9
192.16.166.58 43115 331139 338308 13903 0 0 1w1d 2
192.16.166.66 43475 284783 288938 13903 0 0 10w4d 1
192.16.166.82 36391 319173 372587 13903 0 0 21:07:18 4
194.12.136.1 513 333280 340399 13903 0 0 32w1d 0
194.12.136.9 513 340597 340398 13903 0 0 32w1d 12
194.12.136.17 513 333273 340399 13903 0 0 32w1d 0
L513-C-RFTEC-1#sh ip bgp
BGP table version is 13903, local router ID is 194.12.136.21
Status codes: s suppressed, S stale, d dampened, h history, * valid, > best
Path source: I - internal, a - aggregate, c - confed-external, r - redistributed
n - network, D - denied, S - stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 117.103.96.0/20 192.16.166.2 10 0 24167 i
*>n 128.142.128.0/17 0.0.0.0 0 32768 i
* I 194.12.136.9 0 100 0 i
*> 130.199.48.0/23 192.16.166.10 10 0 43 i
* 192.16.166.82 10 0 36391 43 i
*> 130.199.185.0/24 192.16.166.10 10 0 43 i
* 192.16.166.82 10 0 36391 43 i
*> 130.246.179.0/24 192.16.166.66 1000 0 43475 ?
*>I 131.154.128.0/17 194.12.136.9 10 100 0 137 i
* 192.16.166.34 10 0 34878 137 i
* 131.225.2.0/24 192.16.166.30 100 0 3152 i
*>I 194.12.136.9 10 100 0 3152 i
*>I 131.225.160.0/24 194.12.136.9 10 100 0 3152 i
* 192.16.166.30 100 0 3152 i
*>I 131.225.188.0/22 194.12.136.9 10 100 0 3152 i
* 192.16.166.30 100 0 3152 i
*>I 131.225.204.0/22 194.12.136.9 10 100 0 3152 i
* 192.16.166.30 100 0 3152 i
*> 140.109.98.0/24 192.16.166.2 10 0 24167 i
*> 140.109.102.0/24 192.16.166.2 10 0 24167 i
*>I 145.100.32.0/22 194.12.136.9 10 100 0 1126 i
* 192.16.166.34 10 0 34878 1126 i
*> 192.12.15.0/24 192.16.166.10 10 0 43 i
* 192.16.166.82 10 0 36391 43 i
*>n 192.16.166.0/24 0.0.0.0 0 32768 i
* I 194.12.136.9 0 100 0 i
*> 192.16.166.108/30 192.16.166.34 10 0 34878 i
*>I 192.16.166.112/30 194.12.136.9 10 100 0 3152 i
* 192.16.166.30 100 0 3152 i
*> 192.16.166.124/30 192.16.166.10 10 0 43 i
*> 192.16.166.132/30 192.16.166.58 10 0 43115 i
*> 192.16.166.136/30 192.16.166.34 10 0 34878 i
*> 192.108.45.0/24 192.16.166.34 10 0 34878 i
*> 192.108.46.0/23 192.16.166.34 10 0 34878 i
*>I 192.135.23.0/24 194.12.136.9 10 100 0 137 i
* 192.16.166.34 10 0 34878 137 i
*>I 193.48.99.0/24 194.12.136.9 10 100 0 789 i
* 192.16.166.34 10 0 34878 789 i
*> 193.109.172.0/22 192.16.166.58 10 0 43115 i
*>I 194.171.96.128/25 194.12.136.9 10 100 0 1126 1104 i
* 192.16.166.34 10 0 34878 1126 1104 i
*> 202.169.168.0/22 192.16.166.2 10 0 24167 i
* 206.12.1.0/24 192.16.166.10 10 0 43 36391 i
*> 192.16.166.82 10 0 36391 i
---++++ CERN router 2 - all links up
L513-C-RFTEC-2#sh ip bgp s
Neighbor AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/Pfx
192.16.166.14 43 640311 638898 9958 0 0 5d2h 5
192.16.166.18 137 658967 646183 9958 0 0 2d21h 4
192.16.166.26 3152 318897 325384 9958 0 0 1w4d 5
192.16.166.42 789 322468 329702 9958 0 0 6w6d 3
192.16.166.74 1126 306193 311711 9958 0 0 5d14h 9
192.16.166.86 36391 327518 405397 9958 0 0 6d4h 1
192.16.166.98 24167 957849 642499 9958 0 0 1w4d 4
194.12.136.1 513 321898 329180 9958 0 0 32w1d 0
194.12.136.17 513 321893 329179 9958 0 0 32w1d 0
194.12.136.21 513 329042 329168 9958 0 0 32w1d 18
L513-C-RFTEC-2#sh ip bgp
BGP table version is 9958, local router ID is 194.12.136.9
Status codes: s suppressed, S stale, d dampened, h history, * valid, > best
Path source: I - internal, a - aggregate, c - confed-external, r - redistributed
n - network, D - denied, S - stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
* 117.103.96.0/20 192.16.166.74 10 0 1126 24167 i
* 192.16.166.98 100 0 24167 i
*>I 194.12.136.21 10 100 0 24167 i
*>n 128.142.128.0/17 0.0.0.0 0 32768 i
* I 194.12.136.21 0 100 0 i
*>I 130.199.48.0/23 194.12.136.21 10 100 0 43 i
* 192.16.166.14 100 0 43 i
*>I 130.199.185.0/24 194.12.136.21 10 100 0 43 i
* 192.16.166.14 100 0 43 i
*>I 130.246.179.0/24 194.12.136.21 1000 0 43475 ?
*> 131.154.128.0/17 192.16.166.18 10 0 137 i
*> 131.225.2.0/24 192.16.166.26 10 0 3152 i
*> 131.225.160.0/24 192.16.166.26 10 0 3152 i
*> 131.225.188.0/22 192.16.166.26 10 0 3152 i
*> 131.225.204.0/22 192.16.166.26 10 0 3152 i
* 140.109.98.0/24 192.16.166.74 10 0 1126 24167 i
* 192.16.166.98 100 0 24167 i
*>I 194.12.136.21 10 100 0 24167 i
* 140.109.102.0/24 192.16.166.74 10 0 1126 24167 i
* 192.16.166.98 100 0 24167 i
*>I 194.12.136.21 10 100 0 24167 i
*> 145.100.32.0/22 192.16.166.74 10 0 1126 i
*>I 192.12.15.0/24 194.12.136.21 10 100 0 43 i
* 192.16.166.14 100 0 43 i
*>n 192.16.166.0/24 0.0.0.0 0 32768 i
* I 194.12.136.21 0 100 0 i
*>I 192.16.166.108/30 194.12.136.21 10 100 0 34878 i
*> 192.16.166.112/30 192.16.166.26 10 0 3152 i
*>I 192.16.166.124/30 194.12.136.21 10 100 0 43 i
* 192.16.166.14 100 0 43 i
*>I 192.16.166.132/30 194.12.136.21 10 100 0 43115 i
*>I 192.16.166.136/30 194.12.136.21 10 100 0 34878 i
* 192.108.45.0/24 192.16.166.18 10 0 137 34878 i
* 192.16.166.42 10 0 789 34878 i
* 192.16.166.74 10 0 1126 34878 i
*>I 194.12.136.21 10 100 0 34878 i
* 192.108.46.0/23 192.16.166.18 10 0 137 34878 i
* 192.16.166.42 10 0 789 34878 i
* 192.16.166.74 10 0 1126 34878 i
*>I 194.12.136.21 10 100 0 34878 i
*> 192.135.23.0/24 192.16.166.18 10 0 137 i
*> 193.48.99.0/24 192.16.166.42 10 0 789 i
*>I 193.109.172.0/22 194.12.136.21 10 100 0 43115 i
*> 194.171.96.128/25 192.16.166.74 10 0 1126 1104 i
* 202.169.168.0/22 192.16.166.74 10 0 1126 24167 i
* 192.16.166.98 100 0 24167 i
*>I 194.12.136.21 10 100 0 24167 i
* 206.12.1.0/24 192.16.166.14 100 0 43 36391 i
* 192.16.166.74 10 0 1126 36391 i
* 192.16.166.86 100 0 36391 i
*>I 194.12.136.21 10 100 0 36391 i
Traffic statistics
total
ca-triumf
ca-triumf-buckup
de-kit
es-pic
fr-ccin2p3
it-infn-cnaf
ndgf
nl-t1
tw-asgc-primary
tw-asgc-backup
uk-t1-ral
us-cms-fnal
us-cms-fnal-buckup
us-ti-bnl
us-t1-bnl-backup
transit-cern
| I | Attachment | History | Action | Size | Date | Who | Comment |
|---|---|---|---|---|---|---|---|
 png |
asgc-bu.png | r1 | manage | 21.5 K | 2008-04-10 - 15:20 | EdoardoMARTELLI | |
| png |
asgc-pri.png | r1 | manage | 36.9 K | 2008-04-10 - 15:19 | EdoardoMARTELLI | |
| png |
bnl-bu.png | r1 | manage | 18.8 K | 2008-04-10 - 15:20 | EdoardoMARTELLI | |
| png |
bnl-pri.png | r1 | manage | 21.0 K | 2008-04-10 - 15:20 | EdoardoMARTELLI | |
| png |
ca-triumf.png | r1 | manage | 31.3 K | 2008-04-18 - 14:16 | EdoardoMARTELLI | |
| png |
cern-ch.png | r1 | manage | 37.2 K | 2008-04-18 - 14:16 | EdoardoMARTELLI | |
| png |
cnaf.png | r1 | manage | 38.8 K | 2008-04-10 - 15:21 | EdoardoMARTELLI | |
| png |
de-kit.png | r1 | manage | 38.9 K | 2008-04-18 - 14:17 | EdoardoMARTELLI | |
| png |
fnal-bu.png | r1 | manage | 23.2 K | 2008-04-10 - 15:21 | EdoardoMARTELLI | |
| png |
fnal-pri.png | r1 | manage | 37.2 K | 2008-04-10 - 15:21 | EdoardoMARTELLI | |
| png |
fr-in2p3.png | r1 | manage | 31.3 K | 2008-04-18 - 14:17 | EdoardoMARTELLI | |
| png |
in2p3.png | r1 | manage | 43.1 K | 2008-04-10 - 15:21 | EdoardoMARTELLI | |
| png |
it-infn-cnaf.png | r1 | manage | 35.1 K | 2008-04-18 - 14:18 | EdoardoMARTELLI | |
| png |
kit.png | r1 | manage | 42.8 K | 2008-04-10 - 15:22 | EdoardoMARTELLI | |
| png |
ndgf.png | r1 | manage | 23.3 K | 2008-04-10 - 15:22 | EdoardoMARTELLI | |
| png |
nlt1.png | r1 | manage | 30.6 K | 2008-04-10 - 15:23 | EdoardoMARTELLI | |
| png |
pic.png | r1 | manage | 39.7 K | 2008-04-10 - 15:23 | EdoardoMARTELLI | |
| png |
ral.png | r1 | manage | 34.5 K | 2008-04-10 - 15:24 | EdoardoMARTELLI | |
| png |
total.png | r1 | manage | 33.5 K | 2008-04-10 - 15:18 | EdoardoMARTELLI | |
| png |
transit.png | r1 | manage | 41.0 K | 2008-04-10 - 15:25 | EdoardoMARTELLI | |
| png |
triumf-bu.png | r1 | manage | 20.2 K | 2008-04-10 - 15:24 | EdoardoMARTELLI | |
| png |
triumf-pri.png | r1 | manage | 31.9 K | 2008-04-10 - 15:24 | EdoardoMARTELLI | |
| png |
us-fnal-cms.png | r1 | manage | 38.9 K | 2008-04-18 - 14:18 | EdoardoMARTELLI | |
| png |
us-t1-bnl.png | r1 | manage | 33.7 K | 2008-04-18 - 14:18 | EdoardoMARTELLI |
Topic revision: r28 - 2009-01-14 - EdoardoMARTELLI
- LHCOPN Web
- LHCOPN Web Home
- Changes
- Index
- Search
- ABATBEA
- ACPP
- ADCgroup
- AEGIS
- AfricaMap
- AgileInfrastructure
- ALICE
- AliceEbyE
- AliceSPD
- AliceSSD
- AliceTOF
- AliFemto
- ALPHA
- Altair
- ArdaGrid
- ASACUSA
- AthenaFCalTBAna
- Atlas
- AtlasLBNL
- AXIALPET
- CAE
- CALICE
- CDS
- CENF
- CERNSearch
- CLIC
- Cloud
- CloudServices
- CMS
- Controls
- CTA
- CvmFS
- DB
- DefaultWeb
- DESgroup
- DPHEP
- DM-LHC
- DSSGroup
- EGEE
- EgeePtf
- ELFms
- EMI
- ETICS
- FIOgroup
- FlukaTeam
- Frontier
- Gaudi
- GeneratorServices
- GuidesInfo
- HardwareLabs
- HCC
- HEPIX
- ILCBDSColl
- ILCTPC
- IMWG
- Inspire
- IPv6
- IT
- ItCommTeam
- ITCoord
- ITdeptTechForum
- ITDRP
- ITGT
- ITSDC
- LAr
- LCG
- LCGAAWorkbook
- Leade
- LHCAccess
- LHCAtHome
- LHCb
- LHCgas
- LHCONE
- LHCOPN
- LinuxSupport
- Main
- Medipix
- Messaging
- MPGD
- NA49
- NA61
- NA62
- NTOF
- Openlab
- PDBService
- Persistency
- PESgroup
- Plugins
- PSAccess
- PSBUpgrade
- R2Eproject
- RCTF
- RD42
- RFCond12
- RFLowLevel
- ROXIE
- Sandbox
- SocialActivities
- SPI
- SRMDev
- SSM
- Student
- SuperComputing
- Support
- SwfCatalogue
- TMVA
- TOTEM
- TWiki
- UNOSAT
- Virtualization
- VOBox
- WITCH
- XTCA
 |
|
Copyright &© 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors. or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback