Issue 23

Keeping Your Linux Servers Up To Date (I)

Sorin Pânca
Senior Systems Administrator
@Yardi România


In this article, which is split into multiple parts, I"m going to address the issue of keeping your Linux servers up to date. More often than not, when starting to administer the servers of a startup business, the systems administration team finds an already created datacenter setup around a bunch of servers installed with the needed software, which is often a chaotically configured, "barely functioning" setup done by the first team of developers, doing DevOps jobs.

This situation gets in the way when one wants to keep the production or even the development systems updated and it leads to long hours of work. Often, many sysadmins find systems with big uptimes that weren"t updated for years. From the business point of view, as long as the systems are up and running, providing their services, it does not matter if they are up to date or not. Only when something bad happens - like a password database leak or a security compromise that leads to stolen valuable data, or when the developers find their programs need new versions of software installed that are incompatible with the production operating systems" versions - does the management team rush the systems administrators to do upgrades a.s.a.p. This rush further causes poorly tested solutions to reach the production environment.

To simplify and streamline the update process without buying new hardware, we approached the problem in two steps: first we virtualized the application layer by using Linux Containers (LxC). This type of virtualization enabled us to create "virtual machines" without losing performance or overloading the servers with the hardware simulation layer, as is the case with VMWare or RedHat"s KVM or Oracle VirtualBox or XEN; the second step was to "virtualize" the hardware nodes by using a partition image file as the root filesystem.

The upside of the "root filesystem in a file" approach is that we are able to change the root filesystem with a new one only by rebooting the machine and we are able to reverse the change fast, by doing another reboot. Also, having a file as root filesystem, the operating system installation from inside can be upgraded and tested out-of-band and then published to all servers automatically. Then, when any of those servers needs to be upgraded, it only has to be rebooted. This type of approach is already common in the embedded world where vendors publish "image files" to be uploaded to routers, out-of-band management boards (DRAC, iLO, AMT), etc.

In my setup, I used a source based Linux distribution - Gentoo Linux. One may ask "Why would anyone want to compile everything from sources when you can easily install binary packages?" When installing a server, a sysadmin often finds himself compiling software from source for various reasons: outdated packages, custom build features, patches to be applied, etc. Couple that with constant system updates and he finds himself "really working to earn his money". Also, he often finds that not only one package needs to be compiled from sources but many of its dependencies, as well. On a binary distribution, one runs into multiple problems when compiling from source, like the well-known "dependency hell" or replacing system components (python, perl) with unsupported versions. So I better build everything from source using automation tools - the Gentoo"s package manager, portage (which resembles FreeBSD"s ports).


In this first part of the article, I"ll talk about the host system, not the virtualized containers and virtual machines.

When designing the host system root image, also known as a "hardware node""s root image file, we found that it needs to satisfy some requirements:

  • it only has to provide an environment to run LxC and KVM (and at some point in the future, OpenStack - not currently supported outside Ubuntu and RedHat) and Docker; we chose to use KVM to run some Windows Server instances;
  • the partitioning scheme should contain two partitions: an EFI partition to keep the boot loader - Grub v2, which is capable of accessing partition images directly after it creates a loop device; and a second one which will contain at least a partition image file from where the system can boot up;
  • the EFI partition will be mounted under the /boot/loader directory;
  • the host storage system will use GPT as a partitioning method instead of MBR, to allow the usage of a host storage system (RAID arrays) bigger than 2 TB;
  • on the host partition (the one holding the root image file) all other data will be stored inside a "data" directory;
  • the host partition will be mounted under the /hostpart directory
  • the partitions will need to be labeled and mounted by calling them using their labels: the EFI partition will be labeled "EFI-BOOT" and the host and data partition will be labeled "hostpart"; /data0 will be a symlink pointing to /hostpart/data
  • a clustered storage system needs to run on all hardware nodes to provide fault tolerant and distributed storage; we will not use OpenStack"s solutions (as OpenStack is not supported yet on Gentoo) and we will use XtreemFS; each node will act as all three components of the XtreemFS - directory server (DIR), metadata and replica catalog (MRC) and object storage device (OSD);
  • the cluster storage will be accessed under the /warehouse directory on all hosts and interested guest systems;
  • the root image file will be named "hn-root.img";
  • the updated root image file will be named "hn-root-new.img";
  • the old root image file will be named "hn-root-old.img";
  • if a file called "system-revert" is present, the system will rename the current hn-root.img to hn-root-broken.img and hn-root-old.img to hn-root.img and boot the old root image;
  • all root filesystem image processing fu? happens in the initrd boot phase;
  • a custom init script that runs early in the boot process initializes the new raw installation found in the image file (which can be viewed as a "hardware node class" in an object oriented way) to be instantiated with the specifics of each host - which is viewed as an "object" of the "hardware node class" - (hostname, IP address, network configuration, ssh keys, storage cluster node identities, puppet ID, etc.); the script also takes care of synchronizing this configuration with a common state repository hosted on the clustered filesystem.

Below is a diagram of the system"s host storage device partitions and the root image file:

The initrd phase of the boot process diagram

Genkernel is a script from the Gentoo project that helps Linux users compile their kernels; it should also work outside of the Gentoo distribution (if you want a binary distribution that is granted to allow genkernel to work, take a look at a Gentoo derivative: Sabayon Linux or Calculate Linux). To be able to boot from the image partition, I modified the genkernel"s default/linuxrc file and added the logic described in the above diagram. You can clone the modified genkernel from github at https://github.com/psihozefir/genkernel.git.

Also, the init script that cleans up the hn-root.img file of unneeded files and settings will be available soon. In the meantime, you can do it manually. This script is needed in order for you to be able to put the image file on multiple servers without causing confusion on your datacenter network and management applications (like Nagios or Icinga, or nodes` administration panels).

To install a system into a root partition image, the following steps should be followed (this procedure will wipe the storage drive clean, so backup everything before continuing):

1. Using parted, create a new GPT label if you didn"t partition your storage using GPT yet; this step will destroy all the data stored on the storage device;

2. Create a very small partition to host the Grub2 boot loader (128 MB), format it as FAT32 (or FAT16 if the mkfs tool complains about the size of the allocation table) and label it EFI-BOOT (notice the case);

3. Create another partition that fills up the rest of the storage space and format it with any Linux FS (BTRFS is recommended);

4. On a separate machine (can be a workstation), create a 15 GB partition (can be bigger or smaller, as you see fit; a bigger partition will take longer to copy to all servers and a smaller one may fill up more quickly) and install a Linux distribution of your choice; once the installation is done, create an image of that partition using the dd command; note that the /boot directory is inside this partition, so the kernel and the initramfs will be accessed by grub after setting up a "grub2 loop device" which is different than the kernel"s /dev/loop0 device;

5. Recompile the kernel using genkernel; if you only want to generate a compatible initramfs, you can do that using the following command genkernel initramfs, instead of genkernel --menuconfig all;

6. In the /boot directory, the file "kernel" should be a symlink to the actual kernel file and the file "initramfs" should be a symlink to the actual initramfs file;

7. Loop mount the partition image file and in /boot/loader/grub/grub.cfg, create a new menu entry (my partition image file is formated as reiserfs, so I added the reiserfs module; you"ll need to load the ext2 module if you have an ext2, ext3 or ext4 image file):

menuentry "GNU/Linux in a file" {
  insmod part_gpt
  insmod fat
  insmod reiserfs
  insmod ext2
  insmod gzio
  set root="hd0,gpt2"
  loopback loop ($root)/hn-root.img
  echo "Loading Linux…"
  linux (loop)/boot/kernel root=/dev/ram0 
    real_root=/hn-root.img raw_loop_root_host_  
    partition=LABEL=hostpart ro
  echo "Loading initial ramdisk…"
  initrd (loop)/initramfs
  1. Edit /etc/fstab, remove all entries and add the following entries:
    LABEL=EFI-BOOT /boot/loader vfat noauto,noatime 1 2
    LABEL=hostpart /hostpart auto noatime 0 1

    9. Dismantle the file and copy it to the destination server; install grub on its host storage drive: grub2-install --target=x86_64-efi --boot-directory=/boot/loader --efi-directory=/boot/loader; you"ll need to use chroot from a live USB Linux (System Rescue CD, for example) in order to be able to install grub, but that is beyond the scope of this article.

    Another advantage for using image files is that it makes switching Linux distributions really easy. Just put an hn-root-new.img containing another Linux distribution and you"re done!

    In my next part of this article I"ll talk about the XtreemFS cluster storage and in the 3rd part, I"ll talk about virtualization of systems (using LxC and KVM) and applications (using Docker). When OpenStack will become available in one of our images, I"ll also write the fourth part, where I detail how we will use OpenStack.

    This setup is currently work in progress and some parts are not even developed yet. So, stay tuned!




  • Accenture
  • BT Code Crafters
  • Accesa
  • Bosch
  • Betfair
  • MHP
  • BoatyardX
  • .msg systems
  • Yardi
  • P3 group
  • Ing Hubs
  • Colors in projects