<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Serge,<br class=""><div><blockquote type="cite" class=""><div class="">On May 8, 2017, at 11:55 AM, Serge E. Hallyn <<a href="mailto:serge@hallyn.com" class="">serge@hallyn.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Quoting Ben Warren (</span><a href="mailto:ben@skyportsystems.com" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">ben@skyportsystems.com</a><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">):</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Hi Serge,<br class=""><br class=""><blockquote type="cite" class="">On May 4, 2017, at 9:00 AM, Serge E. Hallyn <serge at <a href="http://hallyn.com" class="">hallyn.com</a>> wrote:<br class=""><br class="">Quoting Ben Warren (ben at <a href="http://skyportsystems.com" class="">skyportsystems.com</a>):<br class=""><blockquote type="cite" class="">Hi,<br class=""><br class="">I’m stuck with Ubuntu 14.04 for now and would like to be able to run unprivileged containers that are systemd-based. I’ve found lots of examples of problems that are close, but nothing exactly matches. I got the lxc packages from trusty-backports.<br class=""><br class="">Versions:<br class=""><br class="">ben at ben-sc:~$ lxc-ls --version<br class="">2.0.7<br class="">ben at ben-sc:~$ cat /etc/lsb-release<span class="Apple-converted-space"> </span><br class="">DISTRIB_ID=Ubuntu<br class="">DISTRIB_RELEASE=14.04<br class="">DISTRIB_CODENAME=trusty<br class="">DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"<br class=""><br class="">To keep it simple, I created an unprivileged container of ‘trusty’ using the download method:<br class=""><br class="">ben at ben-sc:~$ lxc-create -n cd-build -t download<br class=""><br class=""><br class="">When I try to start the container, it won’t work:<br class=""><br class="">ben at ben-sc:~$ lxc-start -n cd-build -d --logfile cd-build.log<br class="">lxc-start: tools/lxc_start.c: main: 366 The container failed to start.<br class="">lxc-start: tools/lxc_start.c: main: 368 To get more details, run the container in foreground mode.<br class="">lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.<br class=""><br class="">Logfile contents:<br class=""><br class=""> lxc-start 20170503225525.382 ERROR lxc_cgfsng - cgroups/cgfsng.c:do_secondstage_mounts_if_needed:1557 - Operation not permitted - Error remounting /usr/lib/x86_64-linux-gnu/lxc/sys/fs/cgroup/cpu read-only<br class=""></blockquote><br class="">This is odd, not the error I would have expected.<br class=""><br class="">Can you tell me the exact version and from which ppa?<br class=""><br class=""></blockquote>$ dpkg -s lxc<br class="">Package: lxc<br class="">Status: install ok installed<br class="">Priority: extra<br class="">Section: oldlibs<br class="">Installed-Size: 77<br class="">Maintainer: Ubuntu Developers <<a href="mailto:ubuntu-devel-discuss@lists.ubuntu.com" class="">ubuntu-devel-discuss@lists.ubuntu.com</a>><br class="">Architecture: all<br class="">Version: 2.0.7-0ubuntu1~14.04.1<br class="">Depends: lxc1 (>= 2.0.7-0ubuntu1~14.04.1)<br class=""><br class="">I got it from here:<br class=""><br class=""><a href="http://us.archive.ubuntu.com/ubuntu/" class="">http://us.archive.ubuntu.com/ubuntu/</a> trusty-backports<br class=""><br class="">Here’s what gets installed:<br class=""><br class="">$ sudo apt-get install -t trusty-backports lxc<br class="">Reading package lists... Done<br class="">Building dependency tree <br class="">Reading state information... Done<br class="">The following extra packages will be installed:<br class=""> bridge-utils cgroup-lite cloud-image-utils debootstrap distro-info euca2ools<br class=""> libgnutls28 libhogweed2 liblxc1 libseccomp2 lxc-common lxc-templates lxc1<br class=""> python-distro-info python-requestbuilder python3-lxc uidmap<br class="">Suggested packages:<br class=""> shunit2 gnutls-bin btrfs-tools lvm2 lxctl<br class="">Recommended packages:<br class=""> lxcfs libpam-cgfs<br class="">The following NEW packages will be installed:<br class=""> bridge-utils cgroup-lite cloud-image-utils debootstrap distro-info euca2ools<br class=""> libgnutls28 libhogweed2 liblxc1 libseccomp2 lxc lxc-common lxc-templates<br class=""> lxc1 python-distro-info python-requestbuilder python3-lxc uidmap<br class=""><br class="">As for the overall environment, this is a VM that was originally set up almost 3 years ago, and as a lab machine has only been piecemeal updated over time as needed. The problem is that I have probably a hundred identical instances and am concerned that the package dependencies are maybe not quite right. I’m certainly willing to update whatever individual packages are necessary to get this going. I have the VM snapshotted before trying this, so it’s trivial to reproduce.<br class=""><br class=""><blockquote type="cite" class="">Is there anything in syslog about the failed mount?<br class=""><br class=""></blockquote>This is all I see. It’s at lxc install time, now when trying to start the container:<br class=""><br class="">May 7 21:01:01 ben-sc kernel: [ 103.486718] type=1400 audit(1494216061.420:68): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=5801 comm="apparmor_parser"<br class="">May 7 21:01:01 ben-sc kernel: [ 103.486925] type=1400 audit(1494216061.420:69): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=5801 comm="apparmor_parser"<br class="">May 7 21:01:01 ben-sc kernel: [ 103.487100] type=1400 audit(1494216061.420:70): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=5801 comm="apparmor_parser"<br class="">May 7 21:01:01 ben-sc kernel: [ 103.487292] type=1400 audit(1494216061.420:71): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=5801 comm="apparmor_parser"<br class="">May 7 21:01:01 ben-sc kernel: [ 103.519003] type=1400 audit(1494216061.452:72): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=5835 comm="apparmor_parser"<br class=""><br class=""><blockquote type="cite" class="">You might try some of the other cgroup auto-mount settings (see lxc.container.conf(5)0, maybe<br class=""><br class="">lxc.mount.auto = cgroup:rw<br class=""><br class=""></blockquote>I tried that, and get:<br class=""><br class=""> lxc-start 20170508041726.340 ERROR lxc_cgfsng - cgroups/cgfsng.c:do_secondstage_mounts_if_needed:1557 - Operation not permitted - Error remounting /usr/lib/x86_64-linux-gnu/lxc/sys/fs/cgroup/cpu read-only<br class=""> lxc-start 20170508041726.340 ERROR lxc_conf - conf.c:lxc_mount_auto_mounts:839 - Operation not permitted - error mounting /sys/fs/cgroup<br class=""> lxc-start 20170508041726.340 ERROR lxc_conf - conf.c:lxc_setup:3885 - failed to setup the automatic mounts for 'cd-build'<br class=""> lxc-start 20170508041726.340 ERROR lxc_start - start.c:do_start:811 - Failed to setup container "cd-build".<br class=""> lxc-start 20170508041726.340 ERROR lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 3)<br class=""> lxc-start 20170508041726.340 ERROR lxc_start - start.c:__lxc_start:1346 - Failed to spawn container "cd-build".<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""> lxc-start 20170503225525.382 ERROR lxc_conf - conf.c:lxc_mount_auto_mounts:839 - Operation not permitted - error mounting /sys/fs/cgroup<br class=""> lxc-start 20170503225525.382 ERROR lxc_conf - conf.c:lxc_setup:3885 - failed to setup the automatic mounts for 'cd-build'<br class=""> lxc-start 20170503225525.382 ERROR lxc_start - start.c:do_start:811 - Failed to setup container "cd-build".<br class=""> lxc-start 20170503225525.382 ERROR lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 3)<br class=""> lxc-start 20170503225525.382 ERROR lxc_start - start.c:__lxc_start:1346 - Failed to spawn container "cd-build".<br class=""> lxc-start 20170503225530.922 ERROR lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.<br class=""> lxc-start 20170503225530.923 ERROR lxc_start_ui - tools/lxc_start.c:main:368 - To get more details, run the container in foreground mode.<br class=""> lxc-start 20170503225530.923 ERROR lxc_start_ui - tools/lxc_start.c:main:370 - Additional information can be obtained by setting the --logfile and --logpriority options.<br class=""><br class="">Also:<br class=""><br class="">————————————<br class=""><br class="">ben at ben-sc:~$ cat /proc/self/cgroup<span class="Apple-converted-space"> </span><br class="">12:name=dsystemd:/<br class="">11:name=systemd:/user/1001.user/c2.session<br class="">10:hugetlb:/user/1001.user/c2.session<br class="">9:perf_event:/user/1001.user/c2.session<br class="">8:blkio:/user/1001.user/c2.session<br class="">7:freezer:/user/1001.user/c2.session<br class="">6:devices:/user/1001.user/c2.session<br class="">5:memory:/user/1001.user/c2.session<br class="">4:cpuacct:/user/1001.user/c2.session<br class="">3:cpu:/user/1001.user/c2.session<br class="">2:cpuset:/<br class=""></blockquote></blockquote></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">What the heck is cuasing this? When I log in on a trusty+backports system,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I get:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ubuntu@trusty:~$ cat /proc/self/cgroup</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">11:hugetlb:/user/1000.user/2.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">10:perf_event:/user/1000.user/2.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">9:blkio:/user/1000.user/2.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">8:freezer:/user/1000.user/2.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">7:devices:/user/1000.user/2.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">6:memory:/user/1000.user/2.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">5:cpuacct:/user/1000.user/2.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">4:cpu:/user/1000.user/2.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">3:cpuset:/user/1000.user/2.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">2:name=systemd:/user/1000.user/2.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">(on second login)</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ubuntu@trusty:~$ dpkg -l | egrep -e "(cgroup|lxc|cgfs)"</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii cgmanager 0.24-0ubuntu7.5 amd64 Central cgroup manager daemon</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii cgroup-lite 1.11~ubuntu14.04.2 all Light-weight package to set up cgroups at system boot</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii libcgmanager0:amd64 0.24-0ubuntu7.5 amd64 Central cgroup manager daemon (client library)</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii liblxc1 2.0.7-0ubuntu1~14.04.1 amd64 Linux Containers userspace tools (library)</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii libpam-cgfs 2.0.6-0ubuntu1~14.04.1 amd64 PAM module for managing cgroups for LXC</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii lxc 2.0.7-0ubuntu1~14.04.1 all Transitional package for lxc1</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii lxc-common 2.0.7-0ubuntu1~14.04.1 amd64 Linux Containers userspace tools (common tools)</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii lxc-templates 2.0.7-0ubuntu1~14.04.1 amd64 Linux Containers userspace tools (templates)</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii lxc1 2.0.7-0ubuntu1~14.04.1 amd64 Linux Containers userspace tools</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii lxcfs 2.0.6-0ubuntu1~14.04.1 amd64 FUSE based filesystem for LXC</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ii python3-lxc 2.0.7-0ubuntu1~14.04.1 amd64 Linux Containers userspace tools (Python 3.x bindings)</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Ah, now if I purge cgmanager, then upon login I get:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ubuntu@trusty:~$ cat /proc/self/cgroup</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">11:name=systemd:/user/1000.user/1.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">10:hugetlb:/user/1000.user/1.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">9:perf_event:/user/1000.user/1.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">8:blkio:/user/1000.user/1.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">7:freezer:/user/1000.user/1.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">6:devices:/user/1000.user/1.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">5:memory:/user/1000.user/1.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">4:cpuacct:/user/1000.user/1.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">3:cpu:/user/1000.user/1.session</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">2:cpuset:/</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">which looks more like yours, but my container still starts...</span></div></blockquote></div><br class=""><div class="">I’ve made some progress, but still don’t fully know what’s going on. When I build lxc from source (top-of-tree <a href="http://github.com" class="">github.com</a>:lxc/lxc) and compile with full cgmanager and libcap support, the generated binaries work, and I can start not only my ‘trusty’ container, but also ones that are farther from the host, such as ‘delian-stretch’, which is systemd-based.</div><div class=""><br class=""></div><div class="">The difference I see in the log is which cgroup driver is used.</div><div class="">When I build using the binaries from ’trusty-backports’, I see this:</div><div class=""> lxc-start 20170509054154.989 INFO lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for cd-build</div><div class=""><br class=""></div><div class="">When using the binaries I built from source, I see this:</div><div class=""> lxc-start 20170509053256.861 INFO lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgmanager initing for cd-build</div><div class=""><br class=""></div><div class="">Assuming cgmanager support is compiled in to the ‘trusty-backports’ version, the following code determines if the cgmanager driver is used (non-NULL return code means cgmanager is to be used):</div><div class=""><br class=""></div><div class=""><div class="">struct cgroup_ops *cgm_ops_init(void)</div><div class="">{</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>check_supports_multiple_controllers(-1);</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>if (!collect_subsystems())</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>return NULL;</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>if (api_version < CGM_SUPPORTS_MULT_CONTROLLERS)</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cgm_all_controllers_same = false;</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>// if root, try to escape to root cgroup</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>if (geteuid() == 0 && !cgm_escape(NULL)) {</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>free_subsystems();</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>return NULL;</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>}</div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>return &cgmanager_ops;</div><div class="">}</div></div><div class=""><br class=""></div><div class="">I have no context for how any of this is dependent on the environment, although I’m sure you do :)</div><div class=""><br class=""></div><div class="">regards,</div><div class="">Ben</div><div class=""><br class=""></div><div class=""> </div></body></html>