[lxc-users] unprivileged container with overlayfs

[Jan Kowalsky]

Hi all,

I'm experimenting with unprivileged containers based on overlayfs
snapshots. The goal is to have semi-unprivileged (running as root but
with mapped uids/gids) containers based on a template image.

What I found out: with a unpatched Kernel <= 4,9 there is no possibility
to use overlayfs for unprivileged containes out of the box. While Ubuntu
14.4 seems to have a patched kernel:
https://github.com/lxc/lxc/issues/1370 for making this possible it has
some security implications: https://lwn.net/Articles/671641/.

Instead of using the built in functionality in lxc for creating and
starting overlayfs containers it's possible to just create the overlay
mount before starting the container.

So my approach is:

  * creating a base-template-container with "dir" storage
  * cloning it as a overlayfs snapshot and create a pre-start hook,
    mounting the base-template rootfs and the delta0 to a newly created
    rootfs directory inside the new overlay snapshot
  * editing the config and adjust the lxc.rootfs and the
    lxc.rootfs.backend

The latter we can automate with a lxc.hook.clone.

My question to you: are there any problematic implications for this
approach? Why we should risk a kernel patch with possible security
implications while achieving the same with mounting the overlayfs on
pre-start?

My Configs:

# cat /var/lib/lxc# cat stretch-overlay-unpriv/config

lxc.include = /usr/share/lxc/config/debian.common.conf
# Container specific configuration
lxc.tty = 4
lxc.arch = amd64
# network
lxc.network.type = veth
lxc.network.link = lxc-bridge-nat
# unpriv
lxc.id_map = u 0 296608 65536
lxc.id_map = g 0 296608 65536
lxc.rootfs = /var/lib/lxc/stretch-overlay-unpriv/rootfs
lxc.rootfs.backend = dir
lxc.utsname = stretch-overlay-unpriv

# clone hook for unprivileged overlayfs container
lxc.hook.clone = /var/lib/lxc/overlay-clone.hook



# cat overlay-clone.hook

#!/bin/bash

touch $LXC_CONFIG_FILE.test

cat <<EOF>> $LXC_CONFIG_FILE
# hook for unprivileged overlayfs container
# the overlayfs will be mounted before starting container
lxc.hook.pre-start=/var/lib/lxc/mount_overlay.hook mount
lxc.hook.post-stop=/var/lib/lxc/mount_overlay.hook umount
EOF

# since the new overlay container starts as a normal container with
"dir" storage
# we have to change it:
sed -i -e 's|^lxc.rootfs.backend.*|lxc.rootfs.backend = dir|' -e
's|^lxc.rootfs = .*|lxc.rootfs = /var/lib/lxc/'$LXC_NAME'/rootfs|'
$LXC_CONFIG_FILE

# for unprivileged containers the directory have to be readable for non-root
chmod 755 /var/lib/lxc/$LXC_NAME

# cat mount_overlay.hook

#!/bin/bash
case $1 in
	mount)
		mount -t overlay overlay -o
lowerdir=/var/lib/lxc/stretch-overlay-unpriv/rootfs,upperdir=$LXC_ROOTFS_PATH/../delta0/,workdir=$LXC_ROOTFS_PATH/../olwork/
$LXC_ROOTFS_PATH
	;;
	umount)
		umount $LXC_ROOTFS_PATH

	;;
esac


And a new config after cloning a container with

  lxc-copy -n stretch-overlay-unpriv -N overlay-u3 -s

looks:

# cat overlay-u3/config
# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
# Container specific configuration
lxc.tty = 4
lxc.arch = amd64
# network
lxc.network.type = veth
lxc.network.link = lxc-bridge-nat
# unpriv
# 296608
lxc.id_map = u 0 296608 65536
lxc.id_map = g 0 296608 65536
# clone hook for unprivileged overlayfs container
lxc.hook.clone = /var/lib/lxc/overlay-clone.hook
lxc.rootfs = /var/lib/lxc/overlay-u3/rootfs
lxc.rootfs.backend = dir
lxc.utsname = overlay-u3
# hook for unprivileged overlayfs container
# the overlayfs will be mounted before starting container
lxc.hook.pre-start=/var/lib/lxc/mount_overlay.hook mount
lxc.hook.post-stop=/var/lib/lxc/mount_overlay.hook umount


Best Regards
Jan