[lxc-devel] Clone fails to start with overlayfslxc.mount.entry error

[Serge Hallyn]

Quoting David Richardson (pudnik019 at gmail.com):
> Yes, I recall reading that the C API should be thread-safe, but I may not
> be using the ref count primitives correctly?  At any rate, after cloning,
> starting, stopping, and destroying a few thousand containers from multiple
> concurrent threads, I eventually end up with a broken config file for a

We do test for that in lxc-test-concurrent, and Çaglar does some large
scale concurrency testing using the go api.  If we can see the code you're
using that would be helpful.

> container that looks like the following example:
> 
> ---------------
> 
> lxc.mount.entry = proc proc proc nodev,noexec,nosuid 0 0
> lxc.mount.entry = sysfs sys sysfs defaults 0 0
> lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none
> bind,optional 0 0
> lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
> lxc.mount.entry = /sys/kernel/security sys/kernel/security none
> bind,optional 0 0
> lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
> lxc.tty = 4
> lxc.pts = 1024
> lxc.devttydir = lxc
> lxc.arch = x86_64
> lxc.cgroup.devices.deny = a
> lxc.cgroup.devices.allow = c *:* m
> lxc.cgroup.devices.allow = b *:* m
> lxc.cgroup.devices.allow = c 1:3 rwm
> lxc.cgroup.devices.allow = c 1:5 rwm
> lxc.cgroup.devices.allow = c 5:0 rwm
> lxc.cgroup.devices.allow = c 5:1 rwm
> lxc.cgroup.devices.allow = c 1:8 rwm
> lxc.cgroup.devices.allow = c 1:9 rwm
> lxc.cgroup.devices.allow = c 5:2 rwm
> lxc.cgroup.devices.allow = c 136:* rwm
> lxc.cgroup.devices.allow = c 254:0 rm
> lxc.cgroup.devices.allow = c 10:229 rwm
> lxc.cgroup.devices.allow = c 10:200 rwm
> lxc.cgroup.devices.allow = c 1:7 rwm
> lxc.cgroup.devices.allow = c 10:228 rwm
> lxc.cgroup.devices.allow = c 10:232 rwm
> lxc.utsname = qpJ14Klr8KvP987h
> lxc.network.type = veth
> lxc.network.flags = up
> lxc.network.link = lxcbr0
> lxc.network.hwaddr = 00:16:3e:04:04:04
> lxc.cap.drop = sys_module
> lxc.cap.drop = mac_admin
> lxc.cap.drop = mac_override
> lxc.cap.drop = sys_time
> lxc.rootfs = overlayfslxc.mount.entry = proc proc proc nodev,noexec,nosuid
> 0 0
> lxc.mount.entry = sysfs sys sysfs defaults 0 0
> lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none
> bind,optional 0 0
> lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
> lxc.mount.entry = /sys/kernel/security sys/kernel/security none
> bind,optional 0 0
> lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
> lxc.tty = 4
> lxc.pts = 1024
> lxc.devttydir = lxc
> lxc.arch = x86_64
> lxc.cgroup.devices.deny = a
> lxc.cgroup.devices.allow = c *:* m
> lxc.cgroup.devices.allow = b *:* m
> lxc.cgroup.devices.allow = c 1:3 rwm
> lxc.cgroup.devices.allow = c 1:5 rwm
> lxc.cgroup.devices.allow = c 5:0 rwm
> lxc.cgroup.devices.allow = c 5:1 rwm
> lxc.cgroup.devices.allow = c 1:8 rwm
> lxc.cgroup.devices.allow = c 1:9 rwm
> lxc.cgroup.devices.allow = c 5:2 rwm
> lxc.cgroup.devices.allow = c 136:* rwm
> lxc.cgroup.devices.allow = c 254:0 rm
> lxc.cgroup.devices.allow = c 10:229 rwm
> lxc.cgroup.devices.allow = c 10:200 rwm
> lxc.cgroup.devices.allow = c 1:7 rwm
> lxc.cgroup.devices.allow = c 10:228 rwm
> lxc.cgroup.devices.allow = c 10:232 rwm
> lxc.utsname = ubuntu-14.04
> lxc.network.type = veth
> lxc.network.flags = up
> lxc.network.link = lxcbr0
> lxc.network.hwaddr = 00:16:3e:08:87:51
> lxc.cap.drop = sys_module
> lxc.cap.drop = mac_admin
> lxc.cap.drop = mac_override
> lxc.cap.drop = sys_time
> lxc.pivotdir = lxc_putold
> 
> ------------------
> 
> You can see pretty clearly here where things go wrong: the substitution for
> the rootfs.path into lxc.rootfs = PATH is getting garbled.  In fact, it
> appears that it is substituting an entire copy of the parent's config file
> (the clone's parent is named ubuntu-14.04) instead of the correct
> overlayfs:/path/to/rootfs:/path/to/delta0.  I am running LXC 1.0.4 on an
> Ubuntu-14.04 host.
> 
> ~Dave
> 
> 
> On Tue, Jul 22, 2014 at 9:10 PM, Serge Hallyn <serge.hallyn at ubuntu.com>
> wrote:
> 
> > However if you're using the C api it is meant to be thread-safe.  Which
> > version are you working with?  I'd love to see the program you're using,
> > if you don't mind.
> >
> > Quoting David Richardson (pudnik019 at gmail.com):
> > > Ah, yes, you are right.  Looks like the config file got garbled.  I
> > suspect
> > > it's because I am calling the C API's clone method from multiple threads
> > in
> > > a racey fashion.  I will look to ensure that any critical sections are
> > > protected.  Thanks.
> > >
> > > ~Dave
> > >
> > >
> > > On Tue, Jul 22, 2014 at 7:26 PM, Serge Hallyn <serge.hallyn at ubuntu.com>
> > > wrote:
> > >
> > > > Well, the configuration file is bad.
> > > >
> > > > overlayfslxc.mount.entry = proc proc proc nodev,noexec,nosuid 0 0
> > > >
> > > > is not valid.
> > > >
> > > > Please post the configuration file for the original and clone, as well
> > as
> > > > the exact way that you created the clone.
> > > >
> > > > -serge
> > > > _______________________________________________
> > > > lxc-devel mailing list
> > > > lxc-devel at lists.linuxcontainers.org
> > > > http://lists.linuxcontainers.org/listinfo/lxc-devel
> > > >
> >
> > > _______________________________________________
> > > lxc-devel mailing list
> > > lxc-devel at lists.linuxcontainers.org
> > > http://lists.linuxcontainers.org/listinfo/lxc-devel
> >
> > _______________________________________________
> > lxc-devel mailing list
> > lxc-devel at lists.linuxcontainers.org
> > http://lists.linuxcontainers.org/listinfo/lxc-devel
> >

> _______________________________________________
> lxc-devel mailing list
> lxc-devel at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-devel