[lxc-users] [debian/stretch64] lxc-copy: Snapshot with OverlayFS backingstorage fails with Linux-4.9.y

[Christian Brauner]

On Wed, Oct 25, 2017 at 11:45:52AM +0200, Sedat Dilek wrote:
> [ Please CC me - I am not subscribed to lxc-users and debian-kernel MLs ]
> 
> Hi,
> 
> I am playing with LXC and OverlayFS.
> 
> The Debian-kernel shipped with stretch64 fails when using lxc-copy for
> a snapshot.
> Later I tried the latest Debian-kernel from stretch-backports
> APT-repository [0] and the issue does *not* appear.
> 
> Debian/stretch64 ships LXC v2.0.7.
> 
> root# lxc-copy --version
> 2.0.7
> 
> [ Instructions to reproduce ]
> 
> [ Create LXC-Container ]
> 
> root# lxc-create -n stretch64-ovl -B overlayfs -t download -- -d
> debian -r stretch -a amd64
> 
> [ GOOD: linux-image-4.12.0-0.bpo.2-amd64 (4.12.13-1~bpo9+1) ]
> 
> root# lxc-copy -n stretch64-ovl -N snapshot1 -B overlay -s
> --logfile=lxc-copy-log_kernel-bpo9.txt --logpriority=DEBUG
> 
> [ BAD: linux-image-4.9.0-4-amd64 (4.9.51-1) ]
> 
> root at iniza:~# lxc-copy -n stretch64-ovl -N snapshot2 -B overlay -s
> --logfile=lxc-copy-log_kernel-deb9.txt --logpriority=DEBUG
> 
> I found lxc-issue #1395 [1] where a similiar problem was reported
> 
> Can this be a kernel issue?

Well, if the only different in your tests is the kernel then could be. But there
also have been quite some changes to the liblxc codebase that should improve
general user experience wrt to the overlay storage driver. So you might want to
test e.g. liblxc 2.1. If your environment allows you to do that.

> 
> Miklos and Christian any idea?
> 
> Do you need further informations and/or have instructions on debugging this?
> 
> Thanks for any hints.
> 
> Regards,
> - Sedat -
> 
> P.S.: I have attached the LXC configs of stretch64-ovl & snaptshot1
> and the lxc-copy logfiles.
> 
> [0] https://backports.debian.org/Instructions/
> [1] https://github.com/lxc/lxc/issues/1395

>        lxc-copy 20171025093206.756 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
>        lxc-copy 20171025093206.756 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
>        lxc-copy 20171025093206.756 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/stretch64-ovl/rootfs onto /var/lib/lxc/stretch64-ovl/rootfs with options upperdir=/var/lib/lxc/stretch64-ovl/delta0,lowerdir=/var/lib/lxc/stretch64-ovl/rootfs,workdir=/var/lib/lxc/stretch64-ovl/olwork.
>        lxc-copy 20171025093206.757 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/stretch64-ovl/rootfs onto /var/lib/lxc/snapshot2/rootfs with options upperdir=/var/lib/lxc/snapshot2/delta0,lowerdir=/var/lib/lxc/stretch64-ovl/rootfs,workdir=/var/lib/lxc/snapshot2/olwork.
>        lxc-copy 20171025093207.420 ERROR    lxcoverlay - bdev/lxcoverlay.c:ovl_rsync:717 - rsyncing /var/lib/lxc/stretch64-ovl/rootfs to /var/lib/lxc/snapshot2/rootfs
>        lxc-copy 20171025093207.421 ERROR    lxcoverlay - bdev/lxcoverlay.c:ovl_do_rsync:757 - copying overlayfs delta
>        lxc-copy 20171025093207.421 ERROR    bdev - bdev/bdev.c:bdev_copy:388 - failed getting pathnames for cloned storage: overlayfs:/var/lib/lxc/stretch64-ovl/rootfs:/var/lib/lxc/stretch64-ovl/delta0
>        lxc-copy 20171025093207.422 ERROR    lxc_container - lxccontainer.c:copy_storage:2867 - Error copying storage.
>        lxc-copy 20171025093207.430 INFO     lxc_container - lxccontainer.c:container_destroy:2414 - Destroyed directory for snapshot2



>        lxc-copy 20171025091007.914 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
>        lxc-copy 20171025091007.914 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
>        lxc-copy 20171025091007.914 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/stretch64-ovl/rootfs onto /var/lib/lxc/stretch64-ovl/rootfs with options upperdir=/var/lib/lxc/stretch64-ovl/delta0,lowerdir=/var/lib/lxc/stretch64-ovl/rootfs,workdir=/var/lib/lxc/stretch64-ovl/olwork.
>        lxc-copy 20171025091007.914 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/stretch64-ovl/rootfs onto /var/lib/lxc/snapshot1/rootfs with options upperdir=/var/lib/lxc/snapshot1/delta0,lowerdir=/var/lib/lxc/stretch64-ovl/rootfs,workdir=/var/lib/lxc/snapshot1/olwork.
>        lxc-copy 20171025091008.114 INFO     lxc_container - lxccontainer.c:copy_file:2641 - Error stat'ing /var/lib/lxc/stretch64-ovl/lxc_rdepends
>        lxc-copy 20171025091008.114 INFO     lxc_container - lxccontainer.c:copy_rdepends:2800 - Error copying reverse dependencies
>        lxc-copy 20171025091008.136 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/stretch64-ovl/rootfs onto /usr/lib/x86_64-linux-gnu/lxc/rootfs with options upperdir=/var/lib/lxc/snapshot1/delta0,lowerdir=/var/lib/lxc/stretch64-ovl/rootfs,workdir=/var/lib/lxc/snapshot1/olwork.
>        lxc-copy 20171025091008.136 INFO     lxc_conf - conf.c:run_script_argv:424 - Executing script "/usr/share/lxc/hooks/clonehostname" for container "snapshot1", config section "lxc".
>        lxc-copy 20171025091008.166 INFO     lxc_copy_ui - tools/lxc_copy.c:do_clone:376 - Created snapshot1 as snapshot of stretch64-ovl
>