Setting up LVM to join EC2 instance store volumes as one disk

Sunday, January 24, 2016

Some types of Amazon EC2 instances come with multiple instance store volumes - for instance, the c3.xlarge instance type comes with two 40GB SSD volumes. This guide shows how to utilize them as one large 80GB volume via LVM, should this be more desirable for any particular application.

Note that instance store is ephemeral. When an instance is stopped or rebooted, the contents of both volumes will be lost, including the partition configuration set up in this document. On reboot, these steps will need to be re-executed.

This example uses a c3.xlarge instance with one 8GB EBS root volume, and two 40GB instance store volumes.

Unmount any existing instance store volumes

Some AMIs will automatically format and mount instance store volumes on boot. Run lsblk to determine if this is the case:

[root@app01 ~]$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
+-xvda1 202:1    0   8G  0 part /
xvdb    202:16   0  40G  0 disk /mnt
xvdc    202:32   0  40G  0 disk

Unmount any instance store volumes shown as mounted using umount:

[root@app01 ~]$ sudo umount /dev/xvdb
[root@app01 ~]$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
+-xvda1 202:1    0   8G  0 part /
xvdb    202:16   0  40G  0 disk
xvdc    202:32   0  40G  0 disk

Run fdisk to create partitions on both drives

The fdisk utility allows partitions to be created on instance store volumes. Partitions need to be created on all volumes. The following example will run fdisk against /dev/xvdb:

[root@app01 ~]$ sudo fdisk /dev/xvdb

The sequence of commands needing to be entered is: n p 1 t 8e w. Consult the man page for fdisk for more information on what each command does.

There is no need to reboot, as the root device has not changed.

Install LVM

LVM provides the functionality needed to glue multiple volumes together. For EL-compatible OSes:

[root@app01 ~]$ sudo yum install lvm2

For Debian, Ubuntu, and derivatives:

[root@app01 ~]$ sudo apt-get install lvm2

Setting up the first disk

First, let LVM build an index of all LVM volumes (of which there should be none):

[root@app01 ~]$ sudo vgscan

Create a volume group and logical volume - in this case, the volume group will be called “workspace” and the logical volume will be called “volume”, and the first volume is at /dev/xvdb1:

[root@app01 ~]$ sudo pvcreate /dev/xvdb1
[root@app01 ~]$ sudo vgcreate workspace /dev/xvdb1
[root@app01 ~]$ sudo lvcreate -l100%FREE -nvolume workspace

Format the newly created logical volume at /dev/workspace/volume as ext4 (or some other supported filesystem):

[root@app01 ~]$ sudo mke2fs -t ext4 /dev/workspace/volume

Setting up subsequent disks

For each extra disk, extend the volume group onto the new disk:

[root@app01 ~]$ sudo vgextend workspace /dev/xvdc

Determine the amount of free space on the volume group, and resize to fill it (take note of the “Free PE / Size” metric):

[root@app01 ~]$ sudo vgdisplay
  --- Volume group ---
  VG Name               workspace
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               79.97 GiB
  PE Size               4.00 MiB
  Total PE              20472
  Alloc PE / Size       10236 / 39.98 GiB
  Free  PE / Size       10236 / 39.98 GiB
  VG UUID               SfNNzy-cRZx-Nxe6-VA6d-L07c-wKp0-D2jLlC
[root@app01 ~]$ sudo lvextend -L+39.98G /dev/workspace/volume

Test the containing partition for errors, and resize it:

[root@app01 ~]$ sudo e2fsck -f /dev/workspace/volume
[root@app01 ~]$ sudo resize2fs /dev/workspace/volume

It’s now possible to mount the disk - here shown as /mnt/workspace:

[root@app01 ~]$ sudo mkdir /mnt/workspace
[root@app01 ~]$ sudo mount /dev/workspace/volume /mnt/workspace

The instance store volumes should now be shown by df as one large drive:

[root@app01 ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.8G 701M 6.7G 10% / tmpfs 3.6G 0 3.6G 0% /dev/shm /dev/mapper/workspace-volume 79G 56M 75G 1% /opt/workspace

Note that this volume should not be included in /etc/fstab, as it is located on ephemeral instance store. When the instance reboots, this process will need to be repeated. Alternatively, these commands can be run as part of init script, so that a reboot will automatically trigger the LVM build.