[lxc-users] Converting from libvirt lxc

Wed Dec 2 15:30:54 UTC 2015

On Mon, Nov 30, 2015 at 02:43:39PM -0800, Peter Steele wrote:
> This message is a bit long and I apologize for that, although the bulk is
> cut-and-paste output. I'm migrating our container project from libvirt-lxc
> under CentOS 7.1 to LXC and I'm seeing some errors in /var/log/messages that
> I don't see in libvirt-lxc. The LXC containers I am creating are based on
> the same custom CentOS image that I've been using with libvirt-lxc. My
> assumption is that this image should be able to be used without any
> significant changes as long as I have the appropriate config file defined
> for this image when an LXC container is installed.
> 
> The lxc-create command I'm using looks generally like this:
> 
> # lxc-create -f /hf/cs/vm-03/config -t /bin/true -n vm-03
> --dir=/hf/cs/vm-03/rootfs
> 
> where the config file has the following options defined:
> 
>     lxc.tty = 4
>     lxc.pts = 1024
>     lxc.kmsg = 0
>     lxc.utsname = vm-03
>     lxc.network.type = veth
>     lxc.network.flags = up
>     lxc.network.link = br0
>     lxc.network.veth.pair = vm-03
>     lxc.network.hwaddr = fe:d6:e8:f2:aa:e6
>     lxc.rootfs = /hf/cs/vm-03/rootfs
> 
> When a container boots, I'm seeing the set of errors below:
> 
>     Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sdb, 10)
> failed: No such file or directory
>     Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sdb1, 10)
> failed: No such file or directory
>     Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sda, 10)
> failed: No such file or directory
>     Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sdb2, 10)
> failed: No such file or directory
>     Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sdb4, 10)
> failed: No such file or directory
>     Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sdb3, 10)
> failed: No such file or directory
>     Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sda4, 10)
> failed: No such file or directory
>     Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sda3, 10)
> failed: No such file or directory
>     Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sda2, 10)
> failed: No such file or directory
>     Nov 30 09:28:48 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sda1, 10)
> failed: No such file or directory
>     Nov 30 09:28:49 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sdc, 10)
> failed: No such file or directory
>     Nov 30 09:28:49 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sdc2, 10)
> failed: No such file or directory
>     Nov 30 09:28:49 vm-03 systemd-udevd: inotify_add_watch(7, /dev/sdc1, 10)
> failed: No such file or directory
>     ...
>     Nov 30 09:28:56 vm-03 systemd-udevd: Failed to apply ACL on
> /dev/snd/hwC0D0: No such file or directory
>     Nov 30 09:28:56 vm-03 systemd-udevd: Failed to apply ACL on
> /dev/snd/controlC0: No such file or directory
>     Nov 30 09:28:56 vm-03 systemd-udevd: Failed to apply ACL on
> /dev/snd/pcmC0D0c: No such file or directory
>     Nov 30 09:28:56 vm-03 systemd-udevd: Failed to apply ACL on
> /dev/snd/pcmC0D0p: No such file or directory
>     Nov 30 09:28:56 vm-03 systemd-udevd: Failed to apply ACL on
> /dev/dri/card0: No such file or directory
> 
> The host's drives have not been made available in the containers, and that's
> intentional. These errors are all being created by the udev service of
> course, and that's the ultimate cause. When I create a container under
> libvirt-lxc though, the udev service is not enabled and I therefore do not
> see these errors. Containers created with LXC using the same CentOS image
> have the udev suite of services enabled, and even if I explicitly disable
> them using
> 
> # systemctl disable systemd-udevd-kernel.socket
> # systemctl disable systemd-udevd-control.socket
> # systemctl disable systemd-udevd.service
> # systemctl disable systemd-udev-trigger.service
> 
> when I restart the container the services are enabled and I still see these
> errors. My guess is I'm missing something in the config file for my LXC
> containers but I'm not sure what's needed. This appears to be further
> indicated by the set of sys services that are running in my libvirt-lxc
> containers:

The systemd-udevd.service file has

   ConditionPathIsReadWrite=/sys

And libvirt LXC sets /sys as read-only, so if you have /sys as writable
that could explain the difference in behaviour.

The other notable thing libvirt does is drop  CAP_SYS_MKNOD. Previously
systemd would look at that capability when starting some things like udev,
but it looks like these days it triggers off /sys  read-only status.

> Is the udev service needed in LXC and if so, how do I keep it from
> complaining?

No, you really don't want udev enabled or running inside containers at all.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|