server-hardware-guide

set up remote management first

• Every server has a small computer of its own that runs as long as the power cords are in: iDRAC at Dell, iLO at HPE, XClarity Controller at Lenovo, IPMI or BMC on the rest. It shows the screen, mounts an installation image, switches the server on and off, and keeps the hardware log.
• Give it a fixed address, change the default password, and update its firmware. Do this on the day the server arrives, not on the day it stops answering.
• Put it in a management network of its own. Never make it reachable from the internet: these interfaces have a long history of serious holes.
• Write down the service tag or serial number. With it the vendor's site shows the warranty, the exact parts and the right firmware.

read the lights and the log before you touch anything

• A health light at the front, usually amber when something is wrong. Then one pair of lights per drive, per power supply, and often per memory slot on the board.
• The colours differ between vendors. Common: green for activity, amber for a fault or a rebuild, blue for "this is the one" when you switch on locate.
• The system event log in the management interface names the part and the slot: which memory module, which fan, which power supply. Read it first, then open the lid.

# from the running system, any vendor
ipmitool sel elist | tail -20
ipmitool sdr type Temperature
ipmitool chassis identify 60      # blink the locate light for 60 seconds

# Dell
racadm getsel

replacing a disk without making it worse

1. Find the failed disk by two things that agree: the slot number from the controller, and the serial number printed on the carrier.
2. Switch on the locate light for that slot and go and look. The disk you pull must be the one that blinks.
3. Check that the array is waiting for exactly this disk. Pulling a healthy disk out of an array that already lost one is how a failed disk becomes a lost array.
4. Release the carrier, pull it out two centimetres, and wait half a minute until the disk has stopped spinning. Then take it out.
5. The new disk must be at least as large and of the same kind. Push it in firmly and close the lever.
6. Watch the rebuild start. If the controller shows the disk as foreign or unconfigured instead, it has to be assigned as a replacement by hand.
7. Leave the server alone until the rebuild is finished, and check the backup before you start, not after.

How long a rebuild takes and how risky it is: RAID calculator. What the disk says about itself: SMART reader. Why a healthy array is still not a backup: a snapshot is not a backup.

memory: fill by channel

• The lid usually has the population order printed on the inside. Follow it: a wrong order costs speed or keeps the server from starting.
• With two processors, both get the same modules in the same slots.
• Use the kind the server was built for. Registered modules (RDIMM) and load-reduced ones (LRDIMM) cannot be mixed, and desktop memory does not work at all.
• Corrected memory errors in the log are an early warning. The server keeps running, the module is on its way out: replace it at the next maintenance window.
• Not sure whether the module or the slot is bad? Swap the suspect with its neighbour. If the error follows the module, it is the module.

power: two supplies, two feeds

• Plug the two power supplies into two different power strips, on two different circuits if the room has them, one of them behind the UPS at least.
• Each feed must be able to carry the whole rack alone. If both run above half load, the surviving one trips the moment the other fails.
• The management interface shows what the server draws right now. Add it up per power strip before you add a server.
• A UPS battery lasts three to five years. Run the self-test, look at the runtime it reports, and make sure the servers get told to shut down before it is empty.

rack units and mounting

• Count the holes before you clip in the cage nuts. One hole off and the server sits crooked, or the next device does not fit.
• Heavy things go at the bottom: UPS first, then storage, then servers.
• Both rails must click into the same unit, front and back. Pull on them before the server goes on.
• Lift with two people, or with a lift. A 2U server with disks weighs 25 to 30 kg and has to be held level while it meets the rails.
• Pull out only one server at a time. A rack that is not bolted down tips forward.
• Label every cable at both ends with where the other end is. Power on one side of the rack, data on the other.
• Leave enough slack that a server can slide out on its rails with the cables still in.

airflow

• Cold in at the front, hot out at the back, for every device in the rack. A switch mounted at the back that blows its hot air forwards heats the servers below it: many switches can be ordered or converted for the other direction.
• Close every empty unit with a blanking panel.
• The management interface shows the inlet temperature. 18 to 27 °C is the recommended range. A server room at 30 °C works, with fans at full speed and a shorter life for everything in it.
• A server in an office: not in a closed cupboard, not on carpet, not next to a radiator, and behind a locked door.

firmware: order and caution

• Use the vendor's bundle for the model instead of single files: it contains versions that were tested together.
• The usual order is the management controller first, then the BIOS, then storage controllers, network cards and disks. The release notes say when a step in between is required.
• Have a current backup and working remote access before you start, and plan for a restart that takes longer than normal.
• In a cluster, one node at a time, and let it rejoin and settle before the next.
• If nothing is wrong and no security fix is in it, a server that works does not need every update the day it appears.

RAID controller pitfalls

• The controller's write cache is protected by a small battery or capacitor. When that fails, the controller switches the cache off to protect the data, and the server becomes several times slower from one day to the next. A sudden slow file or database server: look at the cache battery status first.
• Disks moved to another server show up as a "foreign configuration". Import keeps the data. Clear deletes the array.
• Configure a hot spare, so that a rebuild starts at three in the morning without you.
• Let the controller's regular surface check (patrol read, consistency check) run. It finds bad sectors before a rebuild does.
• Make sure someone gets a message when a disk fails. An array that has been degraded for months unnoticed is the most common way to lose one.

reseating is a real fix

• After transport or after work inside, parts can sit a fraction of a millimetre out: memory, riser cards, power supplies, the cables to the drive backplane.
• Take the part out and put it back in firmly before you order a new one.
• A management controller that no longer answers, or a server that will not power on for no visible reason: pull both power cords, wait 30 seconds, plug them back in. That restarts the one computer in there that never restarts otherwise.

housekeeping

• Keep one spare disk of each kind in use, and for servers out of warranty a spare power supply.
• Photograph or export the BIOS and controller settings. After a mainboard replacement they are gone.
• Know when the warranty ends, and decide before that day whether to extend it or replace the machine.
• Before a server or a disk leaves the building, the data comes off: wipe the disks or keep them.