IntroductionThis describes how to build a server computer. It discusses hardware and software considerations. The target audience is individuals and small and medium sized businesses. If you are a large business, just ask IBM or other serious vendor to specify, configure and maintain your servers. What Is A ServerThere are many types of servers with different requirements. What they generally have in common is:
These requirements are different from typical computers and the server hardware and software is usually different from typical computers. |
It is desirable that the server not use lots of power. If the server uses lots of power, it will likely require lots of cooling. Cooling costs are higher than the servers electrical costs. If there are many servers in a room, minimizing power use is even more important. Reliability goes down when temperature goes up too high. There are a variety of ways of reducing power usage. There is no need for a fancy graphics card in a server. If the graphics card has a fan, it is time for a different graphics card. I recommend high efficiency power supplies, as they generate less heat. Look for 80+ label, which means the power supply is at least 80% efficient. I like Seasonic power supplies. I will discuss power usage later.
All things made by man will fail in time. The goal is to minimize downtime in a cost effective fashion. If you need 99.999% uptime that means 5 1/4 minutes of downtime a year. This takes serious system engineering, and it is time to call IBM. Look for lots of hot-swap (changeable without powering down) hardware including power supplies, hard drives, fans, PCI cards, and UPS batteries.
If you don't need quite as much reliability, your hardware costs will go down quite a bit. The most unreliable hardware are generally fans, hard drives, and power supplies. Be sure that the ones you use are as reliable as possible. I like Panasonic panaflow fans, as they are quite reliable (as well as quiet). For hard drives, there are some recent papers available which discuss hard drive reliability. Historically, SCSI has been more reliable than IDE drives, as well as much more expensive. SAS drives are likely as reliable as SCSI. I wouldn't buy a drive unless it is available with a 5 year warranty.
Memory is a special case. It is quite unlikely to have a permanent error,
however it is quite likely to have non-permanent errors. known as transient
errors. IBM has done lots of research and estimate that with 1 gigabyte of
memory there will be a transient error every week. These are caused by alpha
particles in the memory packaging, cosmic rays, and other stuff. The way to
deal with these transient errors is ECC (error correcting code) memory, which
has extra memory used to detect and correct memory errors. The common ECC
memory will detect all 2 bit errors in 64 bits of memory and correct all 1 bit
errors. There are higher levels of ECC available. See IBM's chipkill
technology for details. If you have a memory error in a part of memory that
isn't being used, it will cause no trouble. If you have a memory error in a
part of memory that will be written before it is read, it will cause no
trouble. Otherwise, the memory error will effect in some way the processing of
the computer, in a bad way. Serious server motherboards will log memory
errors, such as the Intel PR-440FX.
Cheaper motherboards such as the Asus CUR-DLS
and PC-DL will support ECC memory, but won't
log memory errors. There are some CPU 'chipsets' that do not support ECC
memory at all. Motherboards made with these chipsets will not support ECC
memory. I recommend using only motherboards that support ECC memory, and using
only ECC memory for all servers. See the following links for details:
ECC SDRAM Primer
IBM
Chipkill White Paper
EETimes on IBM Chipkill
There are several parts to security. If the server isn't physically secure, then it isn't secure at all. If the server is running software that isn't secure, then the server isn't secure. If the server's operating system isn't secure, then the server isn't secure. If the server isn't secure, then it is at best useless, and quite possibility a liability. Bad people will try to access the server and will try to break into the server, corrupt the server, steal data from the server, and use the server for their own ends. All of this is really bad for you. If the server is only used internally such as a media file server, then security is important, but not mission critical. If the server is used externally (that is accessible to the Internet), then security is mission critical. There are a variety of ways to help make the system secure. I recommend a hardware firewall between the external world and all internal computers. If the server has to be accessible to the outside world, allow just those ports that the server needs to get through the firewall, and to go just to the server that needs those ports. Hardware firewalls are so inexpensive there is no reason to not have one. I recommend a software firewall on all servers that allow access to only the ports that the server needs (this applies to internal as well as external servers).
For the operating system and server software, I recommend choosing reliable software. This does not include any product made by Microsoft. There are far too many vulnerabilities with Microsoft's operating systems as well as server programs such as Microsoft Internet Information Server (web server) and Microsoft Exchange Server (mail server). I am also unimpressed with Sun's recent unreliabilities with Solaris OS and applications. I think OpenBSD is likely the most secure general purpose server OS, although the other BSDs and Linux can be configured to be reasonably secure.
Easy maintenance is important for both the server's hardware and software. The hardware should be easily accessible, well known brands that are generally available, and easy to service. For example, on some Dell workstations that I have seen, the power supply folds out of the computer for easy access to the rest of the system. For an older IBM desktop it took quite a bit of disassembly to access the memory or the second hard drive. If you are unsure about a brand's reputation, you can get recommendations on the Internet. For example, I have had very good support from Adaptec with their SCSI controllers. When mine failed, they cross-shipped me a replacement. I have also had good support for Gateway servers.
For software, it should be easy to install, update, roll back, and check on the status.
First, select a processor and system. There are many options. The lowest power processors x86 processors are made by Via. I decided the Via processors are generally underpowered and somewhat expensive. The next lowest power processors are notebook processors. They are generally more expensive and motherboards for them are limited and expensive. If you had an old notebook lying around, that would make a fine bittorrent server (possibly needing an external disk).
The Pentium III 733 is a reasonably fast, reasonably low power processor. One limitation is there is almost no power saving when the cpu is idle. Virtually all processors made the last 3 years draw less power when idle. In the last two years processors drop their frequency and voltage when load goes down. This was first available with notebook processors, but migrated to the desktop, as desktop processors use more power than notebook processors. If I couldn't locate an old system, I would likely use a low power desktop processor, as they are cheap and readily available. Many motherboards allow the processor voltage to be lowered, or there are software programs such as RM Clock that will allow setting the processor voltage as a function of load. I have used RM Clock to drop my notebook voltages on Turion as well as Core 2 Duo systems. Dropping the voltage n percent will reduce power by approximately n^2 percent. I have dropped my notebook voltage by 20%, which means a 36% reduction in power.
Rather than buy a new computer, I decided to use an existing (free) IBM desktop Pentium III system. The cost is unbeatable, the form factor desirable, and the CPU is relatively low power. I read the IBM manuals and found this particular model doesn't accept ECC memory, though other IBM desktop models do. This isn't ideal, so I am now in the market for another low power system that takes ECC and is cheap. It came with a 20gb hard drive which was dedicated to the OS. A 200gb hard drive was added for storing the files. This machine was old enough that the BIOS didn't recognize the hard drive was over 137gb (new BIOS's know about big hard drives) but it wasn't a problem as the loaded OS didn't use the BIOS to access the big hard drive. The existing graphics card was adequate, as was the 10/100 Ethernet card. I eventually added a low power AGP video card with 32 mbytes of RAM, since I use a KVM switch, and I have a high resolution screen.
For software, a recent version of Linux was selected. In addition to being free, I think that Linux is much more secure than any version of Windows. It also can be configured to not require lots of resources, which is important with older hardware. ssh was enabled to transfer files to and from the server. A firewall was installed, and it allows ssh and the bittorrent protocol in. TCP wrappers was used limit ssh access. The machine has never crashed.
Power consumption:
I ordered an adapter so I can use a notebook hard drive rather than a 3.5 inch hard drive for the OS. (If you do this, get adapter rails to mount a 2.5 inch drive in a 3.5 inch bay. My solution was a kludge.) Power consumption with the notebook drive:
I may also buy a seasonix SFX form factor 80+ power which should also save a few watts. It won't be very cost effective however.
There are many ways to backup data. There are prebuilt boxes that do it. There are hardware raid controllers. There are computers dedicated to storing data. Lets narrow down the choices a bit.
Choosing a raid level: The simplest way to store data is to simply write it to a disk. Neat, simple, and quick. The downside is if the disk fails, your data is gone. Without going into lots of details RAID stands for Redundant Array of Inexpensive Disks. There are many raid levels, including some that aren't redundant. Raid-5 allows for the failure of a single disk to not lose any data. Raid-6 allows for the failure of two disks to not lose any data. Though disks are inexpensive, they aren't very inexpensive. In addition to the price of the disk, the disk takes up space inside the computer, and takes power. Based on my experience with disks, I decided that Raid-5 was the best solution for me.
A prebuilt NAS system: There are prebuilt systems where you just add hard drives. They are less flexible than a dedicated general purpose computer, but are likely simpler and use less power and volumn. Internally, they generally run linux. I will only consider units with gigabit ethernet, otherwise file transfer is limited to 12.5mbytes/second, which is too slow for me. I am only going to consider systems that support raid-5 for robustness. The minimum amount of drives you need for raid-5 is 3 drives (with 2 drives you might as well use mirroring). Lets see what is out there. There is the Promise SmartStor NS4300N that holds up to 4 disks. Write speed is around 15 mbytes/second. Cost is $400 without drives. Looking at Small Network Builder's NAS charts, only Thecus 1U4500 and Thecus N5200 have write speeds above 20mbytes/second. Thecus 1U4500 holds up to 4 drives, and has a write speed of 32mbytes/second and costs about $1000 without drives. Thecus NS5200 holds up to 5 drives and has a write speed of 25mbytes/second and costs about $700 without drives. It looks like these systems are not for me. They are somewhat lacking in flexibility, and pretty expensive, none hold more than 5 hard drives, and none have a write speed above 32mbytes/second.
The next big choice is software or hardware raid. There are basically two types of raid controllers, the type that rely on the operating system to do most of the work (known as software raid controllers) and the type that don't rely on the operating system to do most of the work (known as true hardware raid controllers). If a card costs under $100, or only works with microsoft operating systems, you can be sure it is a software raid controller. For a 8 disk true raid controller, there are several choices: Adaptec has some models around $575, Atto makes a nice model for $1095, ICP has a model for $650, Promise has some nice models starting around $400. In addition to raid controllers, you can do the raid entirely in software. This provides quite a bit of flexibility, although it can take quite a bit of cpu cycles. The good news is you can get a pretty decent motherboard, processor and memory for less than a true hardware raid controller card. Sun has a new filesystem called ZFS which can be configured to have raid-like properties (they call it raid-Z). It is done entirely in software. You can get solaris for personal use for free. ZFS is also supported under FreeBSD. In some respects ZFS is more robust than other filesystems. If you do use ZFS, there is no reason to buy a hardware raid controller. This is some of the possible flexibility you get when you do raid entirely in software. For me, a hardware controller was too expensive, so I decided to use software raid.
I wanted a fileserver to back up the rest of my computers to, at a low cost. I had an older Gateway 6400 server motherboard that was used. It has two Pentium III 933 processors, over a gigabyte of ECC memory, and some 64 bit PCI slots (in addition to the usual 32 bit PCI slots). Since hardware RAID is generally expensive, slow, and can be difficult to repair (if the controller fails you usually need to replace it with the exact same part), I choose software RAID. The main CPU(s) are used, and if some part fails, there is no need to find an exact replacement for it. Since the motherboard has a problem with its IDE controllers Asus CUR-DLS, I used two Promise Ultra TX 100 IDE controllers. These controllers each have two UDMA-133 connectors, however you cannot boot to a CDROM drive attached to them to install the OS. I could have used a SCSI drive attached to the motherboards SCSI controller, but I decided to use a generic IDE hard drive instead. So I attached a CDROM to the motherboard's IDE controller, and six 250gb hard drives for the RAID partition, and one IDE hard drive for the OS to the Promise controllers.
(Note it is generally accepted that it is bad to use the master and slave on a single IDE controller because if the master drive fails, it often takes the slave drive offline as well. This is why people use SCSI which doesn't have this problem, or SATA which doesn't have master and slave, but uses point to point. However, given the low failure rate of hard drives, I decided to risk it. Since this is a backup file server, its failure wouldn't jeopardize anything.)
As more hard drives became available, they would be added to the RAID array. Eventually, SATA drives would be added to the array, minimizing the risks of a master drive causing a double failure. After operating the system for a while, I realized that the 100mbit Ethernet was a bottleneck. I got a Broadcom gigabit Ethernet server card, which has a PCI-X interface. Since the motherboard has two 64 bit PCI slots, which are separate from the other PCI slots, the disk traffic wouldn't affect the Ethernet traffic. The array can sustain reading and writing at 50 mbytes/second according to the Bonnie+ benchmarks, included below. When backing up files from other computers, I can copy at abput 20-30mbytes/second across the ethernet.
Eventually a 300gb drive was added to the array, though the array only could use 250gb. A CoolerMaster Stacker case was chosen in order to provide good cooling for the seven hard drives.
For software, I chose Linux because it had mature RAID software. We configured Linux with ssh as well as samba, which emulates the windows SMB protocol used by windows file servers. That way, windows machines could see the files on the backup file server, and files could be copied back and forth using the windows GUI.
Here is how I configured the RAID-5 array:
# make a device-special file
mknod /dev/md0 b 9 0
# make the raid array with 6 hard drives
mdadm --create /dev/md0 --verbose --chunk=32 --level=5 --raid-devices=6 /dev/hd[abcdef]
# This will make a 232gb size set, and a chunk size of 32k
mdadm --detail --scan
# make the file system on the raid array
mkreiserfs /dev/md0
# make the mount point for the raid array
make directory /data
#the fstab entry for the raid array
/dev/md0 /data reiserfs notail 1 2
# to create mdadm.conf, as root run
#!/bin/sh
grep -qs '/^[[:space:]]*DEVICE' /etc/mdadm.conf || \
echo "DEVICE partitions" >> /etc/mdadm.conf
mdadm -Esc partitions | awk '
/^ARRAY[[:space:]]/ {
print $0, "auto=yes"
}
' >> /etc/mdadm.conf
# This will make /etc/mdadm.conf, which looks like:
# DEVICE partitions
# ARRAY /dev/md0 level=raid5 num-devices=5 UUID=ce5e9366:d83a232c:3da68804:f6814703 auto=yes
Power consumption:
Here are some performance numbers:
[fileserver ~]$ /sbin/bonnie++ -d /mnt/data/tmp
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
fileserver 2272M 13919 96 60733 89 14963 20 13685 86 35304 36 384.7 3
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 9601 100 +++++ +++ 3626 94 5255 97 +++++ +++ 7231 99
fileserver,2272M,13919,96,60733,89,14963,20,13685,86,35304,36,384.7,3,16,9601,100,+++++,+++,3626,94,5255,97,+++++,+++,7231,99
[fileserver ~]$ /sbin/bonnie++ -d /mnt/data/tmp -s 10000
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
fileserver 10000M 14254 96 57165 78 14495 21 15121 92 53172 50 173.3 2
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 7523 77 +++++ +++ 8500 99 9494 99 +++++ +++ 6762 92
fileserver,10000M,14254,96,57165,78,14495,21,15121,92,53172,50,173.3,2,16,7523,77,+++++,+++,8500,99,9494,99,+++++,+++,6762,92
I learned a lot building my first backup fileserver. It was time to build another. I bought a used Intel STL-2 motherboard, two Intel Pentium III 933 processors, and 4gb of ECC memory for $130. This is a really serious server motherboard. It has onboard ultra-160 scsi, and 2 pci-x slots (64 bit, 66mhz), which is faster than my gateway 6400 64 bit slots. I added a supermicro pci-x 8 port sata II controller, and an Intel pro 1000-mt pci-x gigabit ethernet card. I added 4 sata II 500gb hard drives. I used a promise Ultra TX 100 IDE controller for the OS disk.
Power consumption:
Here are some performance numbers:
/usr/local/sbin/bonnie++ -d /mnt/data/tmp
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
sata_fileserver 6G 11719 97 46545 73 33499 50 15310 98 102930 87 369.4 2
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 10307 99 +++++ +++ 8693 98 9832 99 +++++ +++ 7549 99
sata_fileserver,6G,11719,97,46545,73,33499,50,15310,98,102930,87,369.4,2,16,10307,99,+++++,+++,8693,98,9832,99,+++++,+++,7549,99
The writing speed is slower than backup file server V1, but I only have 4 drives in the array. Hopefully it will speed up when I add more hard drives. The reading speed is about twice as fast as the old backup file server. It is possible I was saturating the 33mhz pci bus, or perhaps I was having master/slave issues (which isn't a concern with sata).
I learned a lot building my first two backup fileservers. It is now clear to me that my first two were limited by relatively slow main memory speed. Using PC133 memory, the main memory speed is roughly 400 mbytes/second. Since there is no easy way to speed that up using pentium III processors, it was time for something newer. I bought a used Asus NCCH-DL motherboard and two LV Xeon 1.6ghz processors. Though the Xeons can draw more power than the Pentium III, they also draw less power when running under light loads. This motherboard, like the Intel STL-2, features 2 pci-x slots (64 bit, 66mhz), as well as built in Intel gigabit ethernet. It has a bunch of other nifty features as well, the best ones are the ability to set the clock speed and cpu multiplier as well as fan speed based on system and cpu temperatures. I added 1gb of pc3200 memory. I set the clock speed of the motherboard to 133mhz*14, which gives me a main memory speed of roughly 2000 mbytes/second. I used the same supermicro pci-x 8 port sata II controller, and hard drives as the previous fileserver.
Power consumption (with 4 disk array):
Here are some performance numbers:
/usr/local/sbin/bonnie++ -d /data/tmp
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
sata_fileserver 2G 20822 99 72873 70 39750 40 23786 97 119355 62 437.7 9
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 362 32 +++++ +++ 10892 98 267 34 +++++ +++ 7270 98
sata_fileserver,2G,20822,99,72873,70,39750,40,23786,97,119355,62,437.7,9,16,362,32,+++++,+++,10892,98,267,34,+++++,+++,7270,98
The writing speed is 72.9mb/sec (mega bytes per second) which is faster than V2 which was 46.5mb/second and faster than V1 which was 60.7mb/sec. The reading speed is 119mb/second which is faster than V2 which was 102mb/second and faster than V1 which was 35mb/sec or 53mb/sec (benchmark was run twice). I hope the faster writes translate into faster performance using samba. With the V2 system, I could write between 20 and 25 mb/second. With the V3 system, I can write between 25 and 35 mb/second. I decided to test by copying two disks over at a time. I had tried this before with the V1 and V2 system, but the overall speed didn't increase (as it was maxed out). With the V3 system, I can write to the fileserver with an aggreate speed of between 40 and 50 mb/sec. This is a huge increase over the V2 system. Perhaps I will increase the memory speed and see what that does for performance, but this version is almost twice as fast as the V2 system, which is a big win already.
I bought two more drives for my raid array. You can 'grow' the array without destroying the data. First you say:
mdadm --add /dev/md0 drive1 drive2
to add the drives to the array. Next you say:
mdadm --grow /dev/md0
To grow the array with the added drives. I did this. What I didn't know was the process takes over 100 hours when I am adding two 500gb drives. The system hung during this process, so I built the array from scratch.
The next step is to grow the filesystem via the command:
resize_reiserfs /dev/md0
The there will be more free disk space.
Power consumption (with 6 disk array):
Well, what did the extra two disks do for filesystem performance? Here are some performance numbers:
/usr/local/sbin/bonnie++ -d /data/tmp
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
localhost 2G 21890 99 92074 91 52511 36 27457 95 191331 79 511.9 15
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 15239 100 +++++ +++ 11601 94 14777 99 +++++ +++ 11679 100
localhost,2G,21890,99,92074,91,52511,36,27457,95,191331,79,511.9,15,16,15239,100,+++++,+++,11601,94,14777,99,+++++,+++,11679,100
So, sequential writes went from 73mbytes/second to 92mbytes/second. Sequential reads went from 119mbytes/second to 191mbytes/second. This is close enough to saturating gigabyte ethernet from my perspective. No need to crank up the cpu or memory speed.
I decided to build a media server. There are somewhat different requiements compared to most other servers. You need audio and video outputs. Being quiet is very important. Reliability is less important.
I started off with an E-Machines amd-2000 based system, because it was free. I added a dedicated video card since I wanted S-Video output for my tv set. It worked pretty well, but had some limitations. One significant limitation was the form factor. It was a mini-tower which didn't fit inside my video rack. I ended up buying an Antec NSK-2400 media computer case. It is a very well designed case, though a bit expensive. I used a wireless NIC card, as I had cleverly not put an ethernet drop near the tv.
The next limitation was the E-Machines motherboard. It had a fault where it would reboot when the system was shut down. I had a powerstrip that I would turn off when I turned the computer off. Not very convienient. Also it was a bit finicy with power supplies. It wouldn't work with any Antec high efficiency power supplies. I tested it with the two Antec SU-380 power supplies and two Antec EarthWatts 380 power supplies. The existing power supplies I had were a bit noisy, which isn't ideal for a media server.
I ended up buying a new motherboard and processor, the ECS P4M900T-M motherboard and an Intel E-4500 Core 2 Duo processor. I needed a motherboard that had a micro-atx form factor, and I decided that I wanted pci-e video. The board was inexpensive, but worked ok. I did have some blue screens of death that was caused by poorly seated memory, but once I fixed that, it worked fine. I used the same wireless NIC card, and an ASUS 7200 pci-e video card that has component video output. I hoped that the component video output would be significantly better than S-Video, but I was only a little better (sharper colors mainly). Another advantage of the new motherboard was the processor runs much cooler and quieter than the old AMD-2000 processor, and can be made to run even cooler. I was able to use an efficient, quiet Antec power supply. The new media computer is much quieter than the original one, and is faster (not that that is needed).
If you have comments or suggestions, Email me at turbo-www@weasel.com
Created with gnu emacs and template-toolkit, not some sissy HTML editor.
No Java or javascript needed to view my web pages. They
both have significant security issues.