Home server project
From Wikked
The idea is to build a compact home server, which can fulfill all needs ranging from data storage (NAS-like), media playing (XBMC/Boxee based), as well as general Linux internal and external services.
The original idea was to get a standard NAS. However, the requirements list grew, and the off-the-shelve NASs are either too slow and inflexible, or as expensive as a self built solution. Therefore, the current idea is to build a home server from scratch, and to use virtualization to separate some of its functions.
Contents |
Hardware
Base system
| Component | Part | Price |
|---|---|---|
| Mini-ITX Case | Chenbro ES34069-180W | 185 |
| Mini-ITX Motherboard | Intel DH67CF | 105 |
| CPU | Intel Core i5 2400S | 180 |
| Memory | Crucial CT2KIT51264BA1339 (4 GB x 2, PC3-10600) | 93 |
| Hard drives | 3x Seagate Barracuda LP ST32000542AS, 2TB | 270 |
| Total | 833 |
An Intel Core i3 2100T (35W TDP) would almost be ideal for this project, with its low power consumption, but it doesn't support VT-d, needed for clean PCI pass through in a virtual system. Only the Core i5 and up do. The Intel Core i5 2500T (45W TDP) would be the next ideal candidate, but it seems not available yet. So I'll settle for the 2400S (65W TDP). Fortunately, when idling, they all pretty use consume the same, about 30W for a reasonable full system.
Media player support
| Component | Part | Price |
|---|---|---|
| Bluetooth dongle (class 1) | 10 | |
| HDMI 1.3 extender over CAT5e | ATEN VE800 | 85 |
| Nintendo WiiMote | 40 | |
| WiiMote sensor bar | ||
| Total | 135 |
Benchmarks
Power usage
- First boot, just mainboard/cpu/memory connected:
- idle (during bootup pxe boot) 35W
- max 52W
- BIOS 51W
- First Linux boot off USB stick
- idle (X/GDM login screen) 22W
- max 52W
- First Linux boot off USB stick, cpuburn burnP6 (4x)
- 63W
- Linux boot off USB stick, one 2 TB hard drive enabled, cpuburn burnP6 (4x)
- 69W
- during power on:
- with one hard drive: 72W peak
- with four hard drives: 112W peak
I/O
First bonnie++ run on an XFS file system on a single 2 GB drive:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
malus 15824M 122594 8 54535 5 160290 9 281.0 9
Latency 348ms 615ms 54126us 675ms
Version 1.96 ------Sequential Create------ --------Random Create--------
malus -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 128 0 +++++ +++ 105 0 127 0 +++++ +++ 316 1
Latency 245ms 144us 216ms 233ms 34us 191ms
1.96,1.96,malus,1,1296605727,15824M,,,,122594,8,54535,5,,,160290,9,281.0,9,16,,,,,128,0,+++++,+++,105,0,127,0,+++++,+++,316,1,,348ms,615ms,,54126us,675ms,245ms,144us,216ms,233ms,34us,191ms
Naive software RAID-5 array over all 3 drives (no partitioning), XFS file system, 64KB chunk size:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
malus 15816M 120849 9 65215 7 204575 10 412.1 16
Latency 545ms 195ms 47607us 80199us
Version 1.96 ------Sequential Create------ --------Random Create--------
malus -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 93 0 +++++ +++ 27 0 36 0 +++++ +++ 60 0
Latency 691ms 139us 489ms 462ms 35us 408ms
1.96,1.96,malus,1,1296818087,15816M,,,,120849,9,65215,7,,,204575,10,412.1,16,16,,,,,93,0,+++++,+++,27,0,36,0,+++++,+++,60,0,,545ms,195ms,,47607us,80199us,691ms,139us,489ms,462ms,35us,408ms
Only 60 MB/s per drive... alignment problem? :)
iozone -s 512M:
Run began: Fri Feb 4 13:57:46 2011
File size set to 524288 KB
Command line used: iozone -s 512M
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
524288 4 1470026 3400030 7027703 6888009 5579368 3122282 6036151 5368056 5276818 1459802 2945226 6617289 6997313
After redoing the MD5 with aligned partitioning, and booted a server kernel with deadline I/O scheduler, MD5 resync rate was still only at ~ 65 MB/s. Increasing the block devices' read ahead size did not improve it.
Disabling NCQ (setting queue_depth to 1) helped a lot. MD5 resync rate went up from 65 MB/s to 90 MB/s.
Increasing the MD stripe_cache_size from 256 to 16386 helped even more: from 90 MB/s to 140 MB/s.
Reenabling NCQ (queue_depth 31) did not decrease the speed again.
bonnie++ now reports:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
malus 15816M 256775 18 68686 7 291297 13 430.3 12
Latency 544ms 1530ms 128ms 84882us
Version 1.96 ------Sequential Create------ --------Random Create--------
malus -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 82 0 +++++ +++ 30 0 42 0 +++++ +++ 48 0
Latency 165ms 138us 585ms 334ms 30us 390ms
1.96,1.96,malus,1,1296846405,15816M,,,,256775,18,68686,7,,,291297,13,430.3,12,16,,,,,82,0,+++++,+++,30,0,42,0,+++++,+++,48,0,,544ms,1530ms,,128ms,84882us,165ms,138us,585ms,334ms,30us,390ms
iozone -s 512M:
File size set to 524288 KB
Command line used: iozone -s 512M
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
524288 4 1716642 3540525 7327371 7593736 5724047 3198766 6442706 6436935 5920475 1501660 3213594 7041858 7539711
Btrfs on MD RAID5 and LVM
on a 1 TB LV:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
koekje 15816M 253296 14 85724 19 303787 29 330.9 211
Latency 457ms 1530ms 127ms 97011us
Version 1.96 ------Sequential Create------ --------Random Create--------
koekje -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 22728 34 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 360us 523us 411us 340us 31us 407us
1.96,1.96,koekje,1,1296923037,15816M,,,,253296,14,85724,19,,,303787,29,330.9,211,16,,,,,22728,34,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,457ms,1530ms,,127ms,97011us,360us,523us,411us,340us,31us,407us
with -d single -m single:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
koekje 15816M 64698 4 100832 22 305423 25 330.0 206
Latency 30272ms 1345ms 93025us 119ms
Version 1.96 ------Sequential Create------ --------Random Create--------
koekje -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 29120 39 +++++ +++ +++++ +++ 31636 40 +++++ +++ +++++ +++
Latency 442us 527us 636us 346us 46us 494us
1.96,1.96,koekje,1,1297465354,15816M,,,,64698,4,100832,22,,,305423,25,330.0,206,16,,,,,29120,39,+++++,+++,+++++,+++,31636,40,+++++,+++,+++++,+++,,30272ms,1345ms,,93025us,119ms,442us,527us,636us,346us,46us,494us
EXT4 on MD RAID5 and LVM
on a 1 TB LV:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
koekje 15816M 242849 17 96792 9 302060 14 531.9 11
Latency 512ms 1523ms 73743us 356ms
Version 1.96 ------Sequential Create------ --------Random Create--------
koekje -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 17704 15 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 331us 385us 797us 413us 47us 787us
1.96,1.96,koekje,1,1297457471,15816M,,,,242849,17,96792,9,,,302060,14,531.9,11,16,,,,,17704,15,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,,512ms,1523ms,,73743us,356ms,331us,385us,797us,413us,47us,787us
XFS on MD RAID5 and LVM
on a 1 TB LV:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
koekje 15816M 253642 15 97598 9 288273 13 432.5 13
Latency 347ms 1517ms 39224us 93855us
Version 1.96 ------Sequential Create------ --------Random Create--------
koekje -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 16 0 +++++ +++ 12 0 13 0 +++++ +++ 18 0
Latency 1239ms 139us 1312ms 1238ms 36us 1870ms
1.96,1.96,koekje,1,1297459424,15816M,,,,253642,15,97598,9,,,288273,13,432.5,13,16,,,,,16,0,+++++,+++,12,0,13,0,+++++,+++,18,0,,347ms,1517ms,,39224us,93855us,1239ms,139us,1312ms,1238ms,36us,1870ms
Software configuration
LM Sensors
CPU temperature can be read by manually modprobing coretemp and pkgtemp.
XBMC
First test on a bare, non-virtualized system.
The standard Ubuntu 10.10 Maverick X drivers and kernel do not support Sandy Bridge very well yet. The xorg-edgers ppa supplies newer X.org drivers backported to Maverick, and a 2.6.38 kernel as well. These support (limited?) video acceleration. According to the XBMC debug logs, VAAPI is not compiled in in the official Maverick packages. XBMC has been seen to use up to 110% CPU in some difficult 1080p full HD scenes (Planet Earth), but is generally around 40-60% CPU and smooth.
XBMC could be made to output to the HDMI port. For audio, it was necessary to have it output to custom device hw:0,7 (HDMI 1). HDMI 0 is probably the DisplayPort.
I'm curious whether it's easily possible to run two separate xbmc instances, one outputting to HDMI, one outputting to DisplayPort.
I found some libva 1.0.7 (as opposed to 1.0.4, which doesn't support Sandy Bridge yet) packages backported to Maverick, after which vainfo reported VA to work. Recompiling XBMC Dharma with VAAPI support succeeded, but playing video using it did not. Also, the machine crashed serveral times whenever VA got used, so this appears to be a bit too bleeding edge still.
Standalone install
XBMC comes with the xbmc-standalone debian package, which allows XBMC to run without a window manager. First, the XBMC user should be added to the video and audio groups, then a simple:
$ startx xbmc-standalone
suffices to start XBMC.
Virtualization
KVM seems to work nicely. Unfortunately, my entire plan with PCI pass through for the video/audio devices fell in the water, as the DH67CF main board does not support VT-d, apparently - the BIOS does not offer it as an option. And to think I bought a Core i5 instead of i3 specifically for its VT-d support...
TODO
- Check ACPI/sleep modes
-
Check BIOS settings (c't) -
Test Intel RAID -
Run long SMART self test on all HDDs - Check power modes of HDDs with Linux
- Setup regular SMART checking / configure smartmontools
- Get UEFI booting to work
-
Play with btrfs -
Get X working with Sandy Bridge - Swap mainboard to get Sandy Bridge SATA issues fixed
-
802.1Q - stable kernel for Sandy Bridge
Reminders for the final install
A fair bit of manual hand work is needed:
Partitioning
Ensure that the /boot becomes device /dev/md0.
- /home, 300 GB
- /media, 1 TB
- /var/lib/virt/images, 100 GB
- /backups
Btrfs root
- Set up btrfs "root" partition on one disk only, do the same on the other 2 devices but tell the Installer not to mount it.
- After the installer has formatted the file system, go into the shell, wipe it, and recreate with 3 devices in RAID1 mode.
- Create a new subvolume "root", and set it as the default subvolume to mount
- Go back to the partitioner, tell the installer to use /dev/sda2 as root, but not format it
- Go back into the shell to remount / with the compressoption
- Set the compress mount option in /etc/fstab afterwards
802.1Q
- BEFORE setting up the networking in the installer, load the vlan userland tools and kernel modules in the installer components screen
- Modprobe modules garp and 8021q
- vconfig add eth0 2
- In the installer, setup the network statically
- After the first boot, install the vlan package from cdrom
