[Lxc-users] Rootfs as rw overlay on top of ro directory

[Ivan Vilata i Balaguer]

Hi everyone,

I'm doing some tests on containers having a union rootfs (using Aufs in
Debian) consisting of a writable directory overlaid on top of a
read-only mount coming from a Squashfs image file.  The configuration
described below seems to work pretty well with lxc 0.9.0.alpha3 and
Linux 3.8.13-1 (on Debian Sid), at least when the writable directory is
a plain one and not a mountpoint (see below).

--------

I have the following entries under ``/var/lib/lxc/sliver``:

- config: container configuration file
- rootfs: empty mountpoint directory for the root filesystem
- template.squashfs: read-only Squashfs root image file
- template: empty mountpoint directory for the template
- overlay: writable directory to store changes in

And I use a pre-mount hook like this:

    lxc.hook.pre-mount = /var/lib/lxc/sliver/mount-overlay

That script contains:

    #!/bin/sh
    LXC_DIR=$(dirname "$LXC_ROOTFS_PATH")
    mount -o ro "$LXC_DIR/template.squashfs" "$LXC_DIR/template"
    mount -t aufs -o "br=$LXC_DIR/overlay:$LXC_DIR/template" sliver "$LXC_ROOTFS_PATH"

This seems to work flawlessly: the container gets a writable root file
system and changes go to the ``/var/lib/lxc/sliver/overlay`` directory.
On container shutdown, everything seems to be automatically unmounted
(because the mounts happen inside the container's fs namespace, if I'm
not wrong) and the loop device used by the Squashfs gets detached.

--------

Now I want to put the overlay into an image file (as a simple way to
implement a per-container disk quota):

- overlay.ext4: writable Ext4 filesystem to store changes in
- overlay: empty mountpoint directory for the overlay

Now I use the following hook script (I added the middle mount):

    #!/bin/sh
    LXC_DIR=$(dirname "$LXC_ROOTFS_PATH")
    mount -o ro "$LXC_DIR/template.squashfs" "$LXC_DIR/template"
    mount -t ext4 "$LXC_DIR/overlay.ext4" "$LXC_DIR/overlay"
    mount -t aufs -o "br=$LXC_DIR/overlay:$LXC_DIR/template" sliver "$LXC_ROOTFS_PATH"

The container also works as expected and changes go to the
``overlay.ext4`` file.  However, when the container is shutdown this
filesystem isn't properly unmounted (``file -s
/var/lib/lxc/sliver/overlay.ext4`` reports "needs journal recovery") and
``losetup -a`` shows the file still attached to a loop device (and
worst, trying to detach it with ``losetup -d`` does nothing so the
device is undefinitely locked).  The Squashfs mount gives no problems,
though.

So I try and modify the hook script to mount and unmount the overlay at
the host before starting and after stopping the container:

    lxc.hook.pre-start = /var/lib/lxc/sliver/mount-overlay
    lxc.hook.pre-mount = /var/lib/lxc/sliver/mount-overlay
    lxc.hook.post-stop = /var/lib/lxc/sliver/mount-overlay

The script becomes:

    #!/bin/sh
    LXC_DIR=$(dirname "$LXC_ROOTFS_PATH")

    ACTION="$3"
    case "$ACTION" in
      (pre-start)
        mount -t ext4 "$LXC_DIR/overlay.ext4" "$LXC_DIR/overlay"
        ;;
      (pre-mount)
        mount -o ro "$LXC_DIR/template.squashfs" "$LXC_DIR/template"
        mount -t aufs -o "br=$LXC_DIR/overlay:$LXC_DIR/template" sliver "$LXC_ROOTFS_PATH"
        ;;
      (post-stop)
        umount "$LXC_DIR/overlay"
        ;;
      (*)
        echo "Invalid action: $ACTION" > /dev/null
        exit 1
        ;;
    esac

The result is more or less the same: the container runs and writes as
expected, but on shutdown the overlay gets badly umounted (yes it gets
unmounted in the host and yes I fscked it before starting the container)
and its loop device remains undetachable.

--------

Do you know why the overlay image doesn't get properly unmounted even if
I explicitly use pre-start and post-stop?  Maybe I should use different
hooks or some path different to ``$LXC_ROOTFS_PATH``?

BTW, I hope that this thread helps people set up containers with
overlaid/union rootfs, since I saw some interest on the topic lately in
the list.

Thank you very much!

-- 
Ivan Vilata i Balaguer -- https://elvil.net/