| A/I Orange Book (1.0): An how-to for the realization of a resilient network of self-managed servers | ||
|---|---|---|
| Prev | Next | |
The filesystem organization is, for the most part, common to all servers. This allows for example to move easily services and other content from one server to the other without having to change paths all over the structure, or to put a path directly in the users database without the fear of having to change everything, when moving things around (e.g.: the users homedir or the website document root).
Devising a common partitioning scheme (which can be of course adapted to specific cases) makes it easier to manage the network and to install new boxes.
When possible, it is better to configure the servers with RAID and LVM, so as to manage the diskspace with the best level of flexibility (we generally use RAID1 or RAID5 and LVM on top of it). What follows is a trivial example of partitioning scheme we tried to stick to (but the scheme can vary very much depending on your hardware, disk space and economic needs).
/ - 500M
/tmp - 500M noexec,nosuid
/usr - 3G
/var - 6-10G
/home/admins - 4G, home of the administrators
/home/mail - >20G, mailboxes
/home/users - >20G, websites and ftp accounts
Aside from that, a number of filesystem optimizations had to be implemented, since we are expecting high levels of traffic. We will not go much into details here since it does not make very much sense to speak about performance in a general case: numbers will vary according to specific hardware and software characteristics, and benchmarking is really the only way to tell the useful from the useless solutions. There are, anyway, a few tips that might be valid: for example, you may want to issue the following command on each server:
chattr -R -S +j +A /var/spool/postfix/
This command allows the full journaling of data and metadata of the ext3
filesystem in the postfix spool directory (which helps preserving the
spool contents in case of a filesystem crash), as well as setting the
noatime flag, which improves performance by avoiding
updates of the access time on inodes.
(this example comes from:
"Postfix on an ext3 filesystem", R. Hildebrandt).
Another very useful thing is to setup Amavis to use a temporary directory in a tmpfs filesystem, when we need to save a MIME attachment file in its different parts. This further decreases the disk usage. You can do this with the following lines in /etc/init.d/amavis file:
mkdir /dev/shm/amavis
test -e /var/lib/amavis/tmp \
|| ln -s /var/lib/amavis/ /dev/shm/amavis
and modifying the /etc/amavis/amavisd.conf file as
follows:
$TEMPBASE = "$MYHOME/tmp";
The use of encrypted partitions has been excluded for what concerns the users data, not just for performance reasons, but since we thought quite unwise to be the only repository of a single key for thousands of different doors. Nevertheless, it is a valid strategy to protect the sensible data of the system as a whole: in our case the SSL certificates and the matching encrypting keys are a good place to start.
This mechanism implies the need to fill in a password when the server boots and is needed to avoid the unauthorized copy or modification of the data. We also considered the possibility of mounting the partition when starting each service, unmounting it just after it has started fully: this could protect the box also from intrusions when the box is up and running, but it seems to be only partially useful since at that level of compromise of your box it's very difficult to know what level of security your password will have while you are typing it.
Our implementation of encrypted partitions is based on dm-crypt. This more out of simplicity than for other reasons: if we had to encrypt larger partitions or partitions enduring higher levels of I/O, we would have probably chosen loop-AES (you can find interesting benchmarks and howtos here: http://deb.riseup.net/storage/encryption/benchmarks/).
To make it simple, this is how we create an encrypted image private.img and how we configure it:
$ dd if=/dev/urandom of=private.img bs=1024 count=512 $ export CALOOP=`losetup -f` $ losetup $CALOOP private.img
We now need to create the mapper device to create the encrypted partition and insert the passphrase for the simmetric encryption:
$ cryptsetup -v -y -c aes -s 256 -h ripemd160 create private $CALOOP
We can now create the ext2 filesystem and mount it on /mnt:
$ mkfs -t ext2 -v /dev/mapper/private $ mount -t ext2 /dev/mapper/private /mnt