[lxc-users] [debian/stretch64] lxc-copy: Snapshot with OverlayFS backingstorage fails with Linux-4.9.y

Fri Nov 3 10:40:38 UTC 2017

On Wed, Nov 1, 2017 at 2:18 PM, Christian Brauner
<christian.brauner at mailbox.org> wrote:
> On Wed, Oct 25, 2017 at 11:45:52AM +0200, Sedat Dilek wrote:
>> [ Please CC me - I am not subscribed to lxc-users and debian-kernel MLs ]
>>
>> Hi,
>>
>> I am playing with LXC and OverlayFS.
>>
>> The Debian-kernel shipped with stretch64 fails when using lxc-copy for
>> a snapshot.
>> Later I tried the latest Debian-kernel from stretch-backports
>> APT-repository [0] and the issue does *not* appear.
>>
>> Debian/stretch64 ships LXC v2.0.7.
>>
>> root# lxc-copy --version
>> 2.0.7
>>
>> [ Instructions to reproduce ]
>>
>> [ Create LXC-Container ]
>>
>> root# lxc-create -n stretch64-ovl -B overlayfs -t download -- -d
>> debian -r stretch -a amd64
>>
>> [ GOOD: linux-image-4.12.0-0.bpo.2-amd64 (4.12.13-1~bpo9+1) ]
>>
>> root# lxc-copy -n stretch64-ovl -N snapshot1 -B overlay -s
>> --logfile=lxc-copy-log_kernel-bpo9.txt --logpriority=DEBUG
>>
>> [ BAD: linux-image-4.9.0-4-amd64 (4.9.51-1) ]
>>
>> root at iniza:~# lxc-copy -n stretch64-ovl -N snapshot2 -B overlay -s
>> --logfile=lxc-copy-log_kernel-deb9.txt --logpriority=DEBUG
>>
>> I found lxc-issue #1395 [1] where a similiar problem was reported
>>
>> Can this be a kernel issue?
>
> Well, if the only different in your tests is the kernel then could be. But there
> also have been quite some changes to the liblxc codebase that should improve
> general user experience wrt to the overlay storage driver. So you might want to
> test e.g. liblxc 2.1. If your environment allows you to do that.
>

[ CC Evegeni (lxc debian maintainer team) ]

Hi Christian,

The background is a student project where we use Debian/stretch as a
base for the lxc-containers.
So upgrading liblxc is problematic.
I asked Evgeni for a stretch-backport package of v2.0.9 - I can test
with that and report when the package is available.

The issue in [1] is similiar and has status "incomplete"?
What was missing?
How can I help?

Thanks,
- Sedat -

[1] https://github.com/lxc/lxc/issues/1395

>>
>> Miklos and Christian any idea?
>>
>> Do you need further informations and/or have instructions on debugging this?
>>
>> Thanks for any hints.
>>
>> Regards,
>> - Sedat -
>>
>> P.S.: I have attached the LXC configs of stretch64-ovl & snaptshot1
>> and the lxc-copy logfiles.
>>
>> [0] https://backports.debian.org/Instructions/
>> [1] https://github.com/lxc/lxc/issues/1395
>
>>        lxc-copy 20171025093206.756 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
>>        lxc-copy 20171025093206.756 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
>>        lxc-copy 20171025093206.756 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/stretch64-ovl/rootfs onto /var/lib/lxc/stretch64-ovl/rootfs with options upperdir=/var/lib/lxc/stretch64-ovl/delta0,lowerdir=/var/lib/lxc/stretch64-ovl/rootfs,workdir=/var/lib/lxc/stretch64-ovl/olwork.
>>        lxc-copy 20171025093206.757 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/stretch64-ovl/rootfs onto /var/lib/lxc/snapshot2/rootfs with options upperdir=/var/lib/lxc/snapshot2/delta0,lowerdir=/var/lib/lxc/stretch64-ovl/rootfs,workdir=/var/lib/lxc/snapshot2/olwork.
>>        lxc-copy 20171025093207.420 ERROR    lxcoverlay - bdev/lxcoverlay.c:ovl_rsync:717 - rsyncing /var/lib/lxc/stretch64-ovl/rootfs to /var/lib/lxc/snapshot2/rootfs
>>        lxc-copy 20171025093207.421 ERROR    lxcoverlay - bdev/lxcoverlay.c:ovl_do_rsync:757 - copying overlayfs delta
>>        lxc-copy 20171025093207.421 ERROR    bdev - bdev/bdev.c:bdev_copy:388 - failed getting pathnames for cloned storage: overlayfs:/var/lib/lxc/stretch64-ovl/rootfs:/var/lib/lxc/stretch64-ovl/delta0
>>        lxc-copy 20171025093207.422 ERROR    lxc_container - lxccontainer.c:copy_storage:2867 - Error copying storage.
>>        lxc-copy 20171025093207.430 INFO     lxc_container - lxccontainer.c:container_destroy:2414 - Destroyed directory for snapshot2
>
>
>
>>        lxc-copy 20171025091007.914 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
>>        lxc-copy 20171025091007.914 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
>>        lxc-copy 20171025091007.914 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/stretch64-ovl/rootfs onto /var/lib/lxc/stretch64-ovl/rootfs with options upperdir=/var/lib/lxc/stretch64-ovl/delta0,lowerdir=/var/lib/lxc/stretch64-ovl/rootfs,workdir=/var/lib/lxc/stretch64-ovl/olwork.
>>        lxc-copy 20171025091007.914 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/stretch64-ovl/rootfs onto /var/lib/lxc/snapshot1/rootfs with options upperdir=/var/lib/lxc/snapshot1/delta0,lowerdir=/var/lib/lxc/stretch64-ovl/rootfs,workdir=/var/lib/lxc/snapshot1/olwork.
>>        lxc-copy 20171025091008.114 INFO     lxc_container - lxccontainer.c:copy_file:2641 - Error stat'ing /var/lib/lxc/stretch64-ovl/lxc_rdepends
>>        lxc-copy 20171025091008.114 INFO     lxc_container - lxccontainer.c:copy_rdepends:2800 - Error copying reverse dependencies
>>        lxc-copy 20171025091008.136 INFO     lxcoverlay - bdev/lxcoverlay.c:ovl_mount:436 - Overlayfs: Mounted /var/lib/lxc/stretch64-ovl/rootfs onto /usr/lib/x86_64-linux-gnu/lxc/rootfs with options upperdir=/var/lib/lxc/snapshot1/delta0,lowerdir=/var/lib/lxc/stretch64-ovl/rootfs,workdir=/var/lib/lxc/snapshot1/olwork.
>>        lxc-copy 20171025091008.136 INFO     lxc_conf - conf.c:run_script_argv:424 - Executing script "/usr/share/lxc/hooks/clonehostname" for container "snapshot1", config section "lxc".
>>        lxc-copy 20171025091008.166 INFO     lxc_copy_ui - tools/lxc_copy.c:do_clone:376 - Created snapshot1 as snapshot of stretch64-ovl
>>
>