The Computer Centre

Areas

For the computer centre tour (513 only) there are 6 key areas: the balcony with a view of the main room, the exhibit space, the computing room by the operators area, the physics computing area, the network exchange point, and the downstairs data storage room.

Sometimes for specialist groups you might care to show the UPS and battery rooms (513 s/s) or the view from the 'internal roof' of the computer room (access via large blue 'fire doors' on 2nd floor 513: only a few persons at a time!).

General

All physics computing is done using the Linux operating system and commodity PC hardware. We have a few Solaris server machines as well especially for databases (Oracle).

Most CERN staff have a Windows or Linux (or even dual system) desktop PC as well.

This is by far not enough for all the computing for LHC. That's why we need the Grid to make access to computing resources spread around the world easy.

Balcony

The balcony provides a view of the main computing room. From here you can point out the operational computing area on the right hand side, which includes our e-mail servers (99% of email received is spam), web servers and day-to-day operations systems.

The beige/brown boxes contain the internet exchange point where service providers connect their networks together. One of two in Switzerland (the other is in Zurich) it is actually on the French side of the border. It is used by various telecoms companies to exchange data between networks so if you SMS between networks, it is likely to go through those machines. This was the first IXP in Europe and the largest one is now in Amsterdam, but this is still significant. (The bright yellow tubes are cable guides).

On the far left hand side there is physics computing where data from the 4 soon to be active LHC experimental areas or 'pits' will be stored, sorted and analysed.

Operators Room

Staffed 24 hours a day, 365 days a year, normally just one person. Operating procedures for everything in HTML. However the CC is managed mostly by automated and semi-automated software (anything else would be impossible on this scale).

Main Room

Floor space: 1450m², cooling capacity: 2MW

Room contains: Computational resources and some disk arrays, database servers, etc.

Air conditioning a major problem for data centres everywhere at the moment. As processors get faster they also get hotter and at the moment we are getting a greater increase in heat than performance. Rack machines even worse as they are densely packed with processors.

(Those racks you see with only a few machines in them and fill panels -- well, there's not enough electrical capacity to fill them with machines. The room was designed with one supercomputer in a corner in mind, not several thousand PC servers)

Hot / Cold aisle configuration - the front of the racks face each other on the 'cold' aisle and expel heat out the back to the 'hot' aisle, and the doors and tops of the cold aisles increase efficiency by preventing warm air from mixing unhelpfully with the cold air.

Networking and cabling: a major clean out in 2005 removed ~100 km of cables, including a 1970's modem, which was still active!

Network area has multiple 10 Gigabit connectivity. Data to the remote TIER1 centres (such as Fermi Laboratory in the USA, DESY in Germany) will go on dark fibers or rented lambda and also on the GEANT network.

CERN provides a web hosting site for the UN Satellite Office (the only one to remain working during Asian Tsunami!).

To the right and back of the room is the 'critical area', backed by diesel capacity. Everything else has UPS, but only for a few minutes, which is enough to switch between the French and the Swiss power in case of problems. Some (but not all) the chillers are backed by UPS and diesel as well. (This is very different from, say, a bank or a large server hosting facility, where usually everything has UPS and diesel back-up.)

Power/cooling issues are addressed by buying always more efficient hardware, 'virtualisation' of machines (not all our systems are active all the time so are wasting power and producing waste heat to no purpose). We need to consider an almost inevitable move to a larger CC sometime in the future.

Basement

Floor space: 1200m², cooling capacity: 0.5MW. Additional watercooled racks for about 0.6MW are planned here.

Data Storage

Data is primarily stored on tape. This is not backup, this is the primary data store, all the (petabytes of) disks are just cache. (Backup also goes on the same robots but the volume is negligible compared to the physics data.) Tape is used for cost reasons and practicality. Much longer lasting and durable than disk and RAM memory, and involve much less power consumption (and thus cooling). Also, a factor 5-10 FIXME cheaper than disk per gigabyte.

Tape robots - we have three types in use at the moment - linear (IBM 3584 libraries, 3 here and 1 remote in 613), Sun/StorageTek Powderhorn silos (last 2 of 10 being decommissioned in 613), and Sun SL8500 (interconnected, 'u-bend track', 2 in 513, 1 soon to be 2 in 613).

Tape has life expectancy of ~30 years (but normally CERN uses them for only ~5 years before we 'upgrade to newer, higher capacity tapes).

Normally, CERN tapes are used ~500 times over their lifespan. They are filled up in ~20 write operations, and then some of the data is read back a few 100s of times. Most LHC tapes are expected to be fully re-read once a year or so.

CERN has some tapes used >15,000 times. This is not good for the tapes or the drives used. Bits of the plastic tape case can become weakened by such use and break off in the machines when the robots grab them, so staff have to go in there with dustpan & brush. The interiors of our libraires are surprisingly dusty when they are opened up for decommissioning.

Time to retrieve data, however, is typically ~1-3 minutes so really used for very large data files (GB's in size). Physicists have fads for data - want the latest for analysis - so fortunately we can keep a lot on disk where they can get at it almost immediately ( ~1 second perhaps to start to retrieve data).

Tape libraries

There are 3 linear IBM libraries (IBM 3584). The 2 large ones have 2 robotic accessors (higher throughput, redundancy) going up and down the aisle between sides filled with tapes. These are modular and can be extended, and tape drives can be installed at any of the frames of the machine. However, the bigger (longer) the system, the longer the average time for the robot to get the tape and return to the drive with it. The longest of our systems is currently IBM's largest model (~6600 tapes), and we plan to extend the others as LHC data begins to flow.

There is a Sun SL8500 library installation also in the basement. These libraries are made of interconnected modules (up to 10000 tapes each). They are built in 4 layers, each with a U-bend track and one 'handbot' accessing tapes. We will shortly upgrade to 2 handbots on each level (higher throughput, redundancy). Each module has 2 'vertical lifts' to pass tapes between layers, and can pass tapes to its neighbors via 'pass throughs' at every level. The 513 basement SL8500 has ~16000 tapes and the 613 SL8500 will soon be upgraded to 20000 tapes.

The Sun/Storagetek Powderhorn silos were the oldest, and the last 2 are now being retired (in 613). They have a central column and an outer and inner 'wall' of tapes. In these the robot spins round in a circle to retrieve tapes from either wall. This is very fast for retrieving tapes but limited in capacity - once all the slots are filled you have to upgrade to bigger capacity tapes.

Repack

When a new type of higher capacity tape media is introduced, or a new drive is purchased...

We can copy all the data from the 'obsolete' media to the new higher capacity media, and avoid buying extra robotic library space for a while...

We can copy all the data from the media that ca be re-written at a higher density by the new drive, rewrite it, and avoid buying more new media for a while...

However, this can take a year to move data onto new tapes and a lot of work.....

Interesting Facts

• Number of machines (as of June 2009)
  • 5700 systems, 34600 computing cores
• Storage Capacity
  • ~14 PB usable disk capacity (in PC disk servers, ~41,500 spindles)
  • ~34 PB tape capacity on ~45000 cartridges (IBM and Sun/StorageTek). Robots have 56000 slots and 160 tape drives.
• Planned capacity (tenders in progress, June 2009)
  • 3000 more systems, 20000 cores
  • ~19 PB usable disk capacity (mostly in PC disk servers, some iSCSI storage, ~21000 spindles)
  • There will be an additional 15 PB each year needed for the LHC data (3*10^6 DVDs!)
• Network Capacity
  • Connection at 10 gigabits/second to each Tier 1, plus backup, plus regular (firewalled etc) internet
• Length of wires during clean up
  • ~100 Km
• Number of staff
  • CERN, ~2700 and IT division (computing) ~250 and ~200 on shorter-term Grid projects

Links

• About tape storage: http://it-dep-fio-ds.web.cern.ch/it-dep-fio-ds/Documentation/tapedrive/Welcome.html

• Links to slideshows and Grid presentations: http://it-comm-team.web.cern.ch/IT-Comm-Team/repository/forguides.html

• Additional old figures and facts: https://espace.cern.ch/it-ff/IT%20figures/IT%20Relevant%20Numbers.aspx

-- AlasdairEarl - 03 Oct 2006

-- AndrasHorvath - 14 Feb 2008

-- CurranCharles

-- AlexIribarren - 28 Mar 2008