Chapter 5 Advanced Storage

Introduction
Chapter 4 presented the basics of data storage setup in a Linux system. However, the general
guidelines of the technologies presented there were developed long time ago, even if they are
still in use today. Storage devices split in partitions are the building blocks of other
technologies that appeared during the last few years. We will discuss about Logical Volume
Manager and LUKS encrypted drives and see why and when you should consider using them.

First of all, we should discuss about some of the problems that appear when you use a simple
partitioning schema with primary and logical partitions.

One of the first problems is the lack of flexibility. Once you defined a partition resizing it is
quite complicated. Of course that are many tools that can help you, but most of them require
shutting down the machine and booting from another media, which means downtime for your
system. If we are discussing about a simple workstation, this is acceptable. However, if we are
discussing about servers, where you spend thousands of dollars for high availability, you
should look for some of the new technologies available.

Another problem is data protection. If someone steals your computer or just your hard drive, it
is very simple to install the HDD in a working PC and mount the partitions that you have
defined. After that, he can easily read your documents, emails and so on. This problem is
common to all storage devices: unless encrypted and secured, anyone can steal your device,
mount it in a machine and access the data that you have there. Moreover, if you have sensitive
data, like PIN numbers, digital certificates, private keys...imagine yourself what can happen.

The flexibility problem can be covered using the Logical Volume Manager technology. As
you will see in the section dedicated, LVM does not use directly partition for storing data. It
defines a storage pool composed by one or many real partitions, but it than creates logical
volumes that can be extended independently without losing data. Moreover, the storage pool
itself is intelligent. You can add or remove partitions from it without losing data (of course,
with some limitations).

On the other hand, data security can be achieved by using LUKS encrypted drives. LUKS
provides block level encryption to partitions, thus absolutely no data block can be read
without knowing the correct key. With LUKS, you simply cannot mount secured partitions
without knowing the secret passphrase.

Another technology that worths mentioning is the software RAID. RAID is used to provide
redundancy and protect data in case of a physical drive failure. RAID 0 is the only array type
used for drive striping and not for redundancy. However, in a previous chapter we mentioned
that you need a RAID controller to implement the technology in hardware, completely
transparent to the OS. However, sometimes you won't have such a controller and you will
need to use the Linux technology called multiple devices (md), controlled through the mdadm
tool. This is beyond the purpose of this chapter, but remember what you should search for on
google when you will need software RAID. Now that we have a general idea about the LVM
and LUKS, it is time to study them more carefully and see how can we use them in CentOS
and similar RHEL clones.
Logical Volume Manager (LVM)
LVM's point of view is that storage space allocation should be more flexible and should not
be limited by the size of the partitions available. LVM's architecture consists of three main
layers, as depicted bellow:

The building blocks are real storage blocks. Physical volumes can be partitions (eg. /dev/sda1)
or entire storage devices (eg. /dev/sda). These devices will be used for actual data storage, but
everything will be controlled by LVM driver.

Physical volumes (PV) will be grouped in one or more Volume Groups. Volume Groups
represent pools of storage space with an amount of available storage equal to almost the sum
of all PVs capacities.

From the available Volume Groups, users and administrators can request space for Logical
Volumes. The Logical Volumes (LV) will be used for actual data storage just like a normal
partition.

How does this work? Well, when you initialize a Physical Volume, the driver actually
performs a format-like operation. The drive is divided in small atomic storage elements called
Physical Elements (PE). This is the real cell of LVM. When you add a PV to a Volume Group
(and you can add it to only one Volume Group), all the physical elements are marked as being
part of the new Volume Group and can be used by any Logical Volume that requests space
from the VG.

Before moving to command line stuff, let's discuss about some of the features available when
using LVMs. The main point is that drives can be easily resized, thus providing the flexibility
required. Another important thing is that actual data destination is managed by the LVM
driver. Hence it is possible to add new physical volumes to the volume group, as well as to
remove them, without worrying if it is possible to move data from one physical volume to the
other. Also, it is possible to create snapshots of logical volumes and perform live backups.
Not to be neglected, logical volumes can be named using more relevant strings.

Let's see how can we initialize physical volumes. For this, we will use pvcreate command:

pvcreate <block_device>

Use pvs command to list the available physical volumes. In my case, /dev/vda2 is the partition
created by Disk Druid during the install process when Default Layout option was selected.
See that the new volumes are not associated with any VG? Let's create a new volume group
from the new PVs.

We should use the vgcreate command for this.

vg_create <volume_group_name> <pv_element> [<pv_element> ...]

vgs command will display information about existing volume groups.

Now let's create a logical volume and use it for data storage.

lvcreate -L <size> -n <name> <volume_group_name>

Now that we have a logical volume we can use it for storing data. We have to format and
mount it. But wait a minute... What entry from /dev should we used as mkfs argument? Well,
at this point we need to discuss where can you find the entries for logical volumes.

The first option is the /dev/<volume_group_name> folder. Here you will find a link with LV's
name. What is the target of the link? Well, it is a /dev/dm-* entry, the real device block from
/dev.

We should also mention the entries create by the Device Mapper driver. Look in /dev/mapper
folder for that.

See that in /dev/mapper you will find another link named by following the rule
<volume_group_name>-<logical_volulme_name>. What is the target of the link? Well, the
same entry from /dev. In out case we are talking about /dev/dm-2.

Let's move on and format the LV. You can use any of the following virtual entry for this:
/dev/vg_centos/lv_storage, /dev/mapper/vg_centos-lv_storage, /dev/dm-2.
And from now on you can easily use it as a normal partition. To unmount it use umount.

Now let's make something even more interesting. I will add a new 5GB drive to our machine
(/dev/sdc) and reboot the machine.
I also mount the previously defined logical volume. Than I will add the new drive to
vg_centos volume group.

You cannot just remove the /dev/sda, as you have to explicity move data from the /dev/sda.
While it is possible to omit the explicit destination, in out case we will indicate where
physical elements should be moved. Note that we are not unmounting the
/dev/vg_centos/lv_storage logical volume and everything is performed with the drive
mounted. However, it is highly recommended to unmount the drive if you worry about power
outages or similar threats, as such event can corrupt the data transfer.
Now wait for the transfer to complete and remove the old drive from the vg_centos group.
Voila, data was moved to /dev/sdc drive and now you can service your /dev/sda...and no
service downtime is recorded!
The LVM labs attached to this book show how to resize logical volumes and how to use LVM
snapshots. Let's walk through the most important commands of the labs to explain them.

First of all, let's discuss about LVM resizing. The LVM creation, formatting and mounting
should be quite straight forward. Note that LV can be added to /etc/fstab for persistent mounts
also.

lvcreate -L1G -n dsk1gb vg_centos

mkfs.ext4 /dev/vg_centos/dsk1gb

To extend volume with 1GB:

lvextend -L+1G /dev/vg_centos/dsk1gb

However, if you check now the output of the df -h you will still see that the capacity of
/dev/vg_centos/dsk1gb is 1GB. This happens because you must resize the file system stored
on the device also in order to benefit from the additional space. This operation cannot be
performed with the partition mounted for extended file systems and the procedure depends on
the file system type used.

umount /mnt
e2fsck -f /dev/vg_centos/dsk1gb
resize2fs -p /dev/vg_centos/dsk1gb
e2fsck -f /dev/vg_centos/dsk1gb

Ext4 requires a file system check before resizing and after resizing. With this procedure
completed, you will be able to use the entire allocated space for dsk1gb LV.

LVM snapshots are frozen images of a logical volume. You can create a new snapshot LV
related to an existing LV. The lab explains the procedure pretty well and the only thing that
worths mentioning is how to choose the size of a new snapshot LV.

First of all, we must know what is behind the scene. When you create a snapshot, the initial
volume is frozen and all changes are written to the new logical volume. How many changes
can the system write there? Well, that depends on the snapshot's size. The OS will fill the
snapshot LV with block that differ from the original LV up to its capacity. If you overflow the
snapshot, no more data will be saved and the snapshot LV will be marked as invalid.

This concludes the introduction to LVM. After this section you should be able to describe the
general architecture of LVM and to perform basic tasks like LVM creation, resizing and
snapshotting. The following paragraphs will make use of logical volumes, as it is really
simple to manage them.

Linux Unified Key Setup (LUKS) drives

LUKS drives are generally referred as encrypted drives. This technology performs block level
encryption before data is written to disk and block level decryption during data read process.
The dm_crypt module is responsible for this and AES is the default encryption algorithm
used. The architecture is presented bellow (credits to nnc3.com website) :
Just like LVM architecture, everything starts with a block storage device. In this example, we
see that /dev/hda2 drive is used for this purposes. /dev/hda2 drive will store the block
encrypted by the DM-Crypt layer. You should never try to mount /dev/hda2, because
everything on this drive is secured and tools like mount won't be able to read necessary data.
But what entry should we use to access the data stored on the encrypted drive? Well, when
you will open the encryption layer (with cryptsetup), you will give a name to the data partition
(which can differ from one opening to another). And guess what...the Device Mapper driver
will create an entry for you in /dev/mapper folder.

However, there is one important point that should be remarked here. Instead of using a normal
block device like /dev/hda2, you can also use a logical volume. It is more simple to do this
because LVs can be created and removed in few command lines.

Ok, so how can we do this in CentOS? First of all, you must have the cryptsetup package
installed on your system. And than you will see that cryptsetup command will do everything!

First, let's create a 1 GB LV for our lab and format it using the luksFormat command of
cryptsetup. During the format process you will be prompted for the passphrase used for
encryption.

Now we will open the DM Crypt layer and create an entry in /dev/mapper to access the data
on the encrypted device. Than we can use that entry just like a normal partition.
Note that once you don't need to access the encrypted data anymore you can close the drive
with luksClose command of cryptsetup.

You can list information about LUKS settings using:

cryptsetup luksDump <luks_drive>

Moreover, you can add multiple keys to protect your device (you can use anyone of them
during the opening process). Note that luksAddKey check first an existing passphrase and
only after a successful test it will add the newly supplied one.

Now you can use any passphrase added to your drive when opening it, so you won't have to
share passphrases.

LUKS drives can be configured for persistent mounts also. However, this will require you to
enter passphrases during the boot process, which is not a good idea for a server. It is also
possible to write the passphrases in files stored on un-encrypted partitions, but that would
compromise security in case of theft.

Conclusions
This chapter provided a brief description of two storage technologies available on Linux
systems. While this is only an introduction, it represents more than the basic knowledge
required to understand and use LVM in a development or production environment. I
recommend practicing LVM & LUKS tasks on virtual machines, as this approach will make
HDD add/remove operations more simple.