Visit to Reuters Computer
Centre, Geneva, 20th October 2000
Who: Tony & Dave. Main contacts there were Kristina Kamsaeter & Gilbert Banderet (Building Manager & designer). Keith Cronshaw from Bellwater (a Data Centre Consulting Company) was also present.
The building, which opened 3 years ago, was designed 5 years ago as a backup to the main data centre in London (Docklands). The height was restricted to 10m by Geneva building regulations, but this suited Reuters as they did not want a building split across many floors. During the design phase, Reuters (Gilbert Banderet) drew from experience with all the other data centres (180 worldwide) to plan a building that avoided all the problems they had seen elsewhere.The requirement was for 4,000m2 of raised floor area but this is broken up into 4 rooms to provide some isolation in the event of fire. The false floor depth is 80cm.
They have a very tight control over equipment location, even to the extent of having specially coloured floor tiles to indicate the rows where equipment racks can be installed! 432 19” racks (60cm wide by 90cm deep) fit into 800m2. [With 42 1U PCs per rack, this would fit 18,144 PCs into 800m2.]
Almost all equipment is installed in Knürr 19” racks, most of which are completely enclosed although the sidepanels have been removed from some. Ventilation is ensured with a fan unit (6 fans) on top. (There is a raised plate above this to keep the racks watertight.) No racks have major heating problems, but some run hot—although additional cooling could be provided by completely removing the floor panel underneath. No use is made of this racking for fire detection/extinction, though, the racks seem to be used more for aesthetics and control reasons.
The building is designed for an average power load of 400W/m2 and the occupied areas today operate at 300W/m2. Banderet & Cronshaw see this as being a reasonable level and that the 1,000W/m2 some ISP/co-location centres are using as being excessive. However, they agree that this is for room heights of 3m or so. “Old” machine room designs with high ceilings have a more even temperature distribution and can support greater loads more easily. The drawback for “cold air downflow” is that cooling can’t be routed where it is needed; this can be done easily with underfloor systems by removing tiles (see also http://www.availability.com/research/industry/uptime/reliable_cooling.cfm).
The building is essentially in two halves, each with two rooms (ground and 1st floor). Each side has its own power plant with a separate 18kV power line. Each side has two UPS systems. Either plant can power all of the building, but probably only some of the computer systems if only 1 UPS available.
Three power circuits are installed and colour coded—right down to the colour of the multi-socket boxes in racks. The circuits are “A”mber for live systems, “B”lue for backup systems and a C system for consoles. Console power is actually the most reliable as the C supply can be powered from whichever of A or B are available. Services are all fully duplicated in terms of machines with half on A power and half on B (and don’t forget this is a backup to the London centre …).
They have a “normabar” like system running through each room onto which taps are placed to power equipment. Equipment and racks are placed “back to back” (such tight control!) so that power and cable paths alternate along the clearance spaces. (i.e. the sequence is “power-equipment-cable-equipment-power” and so on.). Cable trays are also divided (and colour coded) between those for A systems and those for B systems.
Their EPO systems are at various granularities—whole site down to room level. As normal, there is a tradeoff between security and accident proofing. Confirms that there is no Swiss legal requirement for EPO systems to be tested annually. However, one of their systems is not working (possibly following modification, not entirely clear) and this will now need to be tested. On the other hand, there may be some Swiss requirement to “tighten screws” on electrical connections each year, and this requires power to be off. This rule is apparently rarely honoured and a better way of checking connection integrity is with an infrared camera—loose connections lead to local heating.
The UPS systems are rotary. They have no batteries as they found these to be problematic and consider battery based systems to be uneconomic over a 20+ year lifetime. Geneva installed rotary UPS systems first but London have now followed suit after experiencing problems with their batteries. Diesel synchronisation is maintained by 3 computers and they have never experienced any synchronisation problems. Ease of maintenance is apparently a factor in the design of the systems they have. Each UPS has only 5 bearings and only 4 lubrication interventions are needed per year—and of these only one requires the UPS to be taken out of circuit.
In normal circumstances, mains power runs a generator in reverse as a motor. This keeps a flywheel running (motor runs at 1500rpm, flywheel at 2600rpm). If mains power drops, the diesel engine starts in 1.8s and is joined to the generator by a clutch. the flywheel keeps the generator running for the 1.8s it takes for the diesel to start. The Reuters systems are made by Euro-Diesel. Apparently there are many suppliers, but Euro-Diesel and Holec have the best experience. [I can’t find a Web address for Holec, but the company is listed in a guide to UPS systems produced by Global Media Publishing.]
Each UPS is rated at 1.2MVA. Fuel consumption is consumes 200-250l/hour at full load and each diesel engine has a 40,000l main tank (2weeks autonomy) and a “day tank” of 100l. The day tank capacity is limited by law and the sensible limit for the main tanks is the limited lifetime (few years) of unused fuel. That problem is addressed at Reuters by a system which allows the diesel tanks to be used in turn by the heating system, thus allowing the fuel to be used up before it goes stale.
The test schedule for the UPS systems was decided in conjuntion with the supplier at installation time. (I got the impression that the schedule is programmed in and nothing needs to be done actively to perform the tests…) The engines are tested oncer per week and the whole system is tested once per month by really shutting off the mains supply.
There is a distinction between “critical power” and “essential power”. Critical power is maintained completely, but essential power sees a 10s break. This distinction is to avoid dropping the full load on the generators immediately. In the existing Reuters building, air-conditioning counts as an “essential load” and so sees a break. A new building is under construction, though, and here everything will be connected to the “critical” supply.
Each room is equipped with air conditioning equipment that blows colc air under the floor. These are glycol based with “dry coolers”—hot water goes from the units in the room to the roof rather than cold water from the roof to the units. The room width of 25m is near the maximum for efficient air flow to all parts of the room. A wider room would need a flase floor with a greater depth.
There are 5 air conditioning units in each room, split across the A and B power circuits. The split is asymmetric (3 and 2) in any room, but balanced across the two rooms in either half of the building. As noted above, the air conditioning is of for 10s before the UPS kicks in after a power failure.
As for the other centres we have visited, Reuters have a Cerberus Algorex detection system that raises a local alarm and then a “standard” system that calls the fire brigade if two detectors go off in coincidence. Gas based extinction systems are installed for the UPS rooms. All walls and doors are designed to withstand a fire for 1 hour.
The Reuters feature we have not seen elsewhere is a “dry sprinkler” system. With this arrangement, the pipes are normally empty, but fill with water following a Cerberus alarm—the heads then release water as normal under the influence of heat. If there is no real fire the pipes will be pumped out. There is a need to test the integrity of the system (using air pressure) to ensure that no heads have been knocked off leading to floods when the pipes are filled.
Cronshaw commented that, contrary to our experience, fire extinction systems are the norm—but could only cite places without extinction. In any case, HSSD systems are never used to trigger the extinction, always lower sensitivity systems and often requiring two different types of detector (both ionisation and optical) to fire in coincidence. Gas based systems must replace 40% of the volume of the air in the room to be effective. The Inergen based system at Reuters for the UPS generator rooms required 18 bottles for a room of around 10m´5m´4m. On this basis, we would need some 800 bottles for the machine room and some 325 for the vault. These bottles would require storage volumes of around 135m3 and 55m3 respectively (the bottles plus valves at Reuters were around 2m in height so the floor area needed is around 70m2 and 25m2). Apparently “Argonite” gas can be stored under higher pressure and so might reduce the number of bottles we would need to around 400 for the machine room (but remember that the machine room is not hermetic…). See http://www.hunc.org/hag1.html for a comparison of Inergen and Argonite.
Cronshaw also commented that HiFog systems have the disadvantage that the water mist won’t penetrate racks or equipment. It might therefore be safe to use in a machine room, but is unlikely to extinguish any fire caused by an overheating circuit board.
Finally, Reuters insist that tape robotics and storage devices are housed in a dedicated room—but this is probably because they are used to backup the local financial database. In Geneva they have a small room with a 3494 robot.