[lxc-users] Can't start unprivileged container in Ubuntu 14.04 with LXC 2

Serge E. Hallyn serge at hallyn.com
Tue May 9 15:10:04 UTC 2017


Quoting Ben Warren (ben at skyportsystems.com):
> Hi Serge,
> > On May 8, 2017, at 11:55 AM, Serge E. Hallyn <serge at hallyn.com> wrote:
> > 
> > Quoting Ben Warren (ben at skyportsystems.com <mailto:ben at skyportsystems.com>):
> >> Hi Serge,
> >> 
> >>> On May 4, 2017, at 9:00 AM, Serge E. Hallyn <serge at hallyn.com> wrote:
> >>> 
> >>> Quoting Ben Warren (ben at skyportsystems.com):
> >>>> Hi,
> >>>> 
> >>>> I’m stuck with Ubuntu 14.04 for now and would like to be able to run unprivileged containers that are systemd-based.  I’ve found lots of examples of problems that are close, but nothing exactly matches.  I got the lxc packages from trusty-backports.
> >>>> 
> >>>> Versions:
> >>>> 
> >>>> ben at ben-sc:~$ lxc-ls --version
> >>>> 2.0.7
> >>>> ben at ben-sc:~$ cat /etc/lsb-release 
> >>>> DISTRIB_ID=Ubuntu
> >>>> DISTRIB_RELEASE=14.04
> >>>> DISTRIB_CODENAME=trusty
> >>>> DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"
> >>>> 
> >>>> To keep it simple, I created an unprivileged container of ‘trusty’ using the download method:
> >>>> 
> >>>> ben at ben-sc:~$ lxc-create -n cd-build -t download
> >>>> 
> >>>> 
> >>>> When I try to start the container, it won’t work:
> >>>> 
> >>>> ben at ben-sc:~$ lxc-start -n cd-build -d --logfile cd-build.log
> >>>> lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
> >>>> lxc-start: tools/lxc_start.c: main: 368 To get more details, run the container in foreground mode.
> >>>> lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.
> >>>> 
> >>>> Logfile contents:
> >>>> 
> >>>>     lxc-start 20170503225525.382 ERROR    lxc_cgfsng - cgroups/cgfsng.c:do_secondstage_mounts_if_needed:1557 - Operation not permitted - Error remounting /usr/lib/x86_64-linux-gnu/lxc/sys/fs/cgroup/cpu read-only
> >>> 
> >>> This is odd, not the error I would have expected.
> >>> 
> >>> Can you tell me the exact version and from which ppa?
> >>> 
> >> $ dpkg -s lxc
> >> Package: lxc
> >> Status: install ok installed
> >> Priority: extra
> >> Section: oldlibs
> >> Installed-Size: 77
> >> Maintainer: Ubuntu Developers <ubuntu-devel-discuss at lists.ubuntu.com>
> >> Architecture: all
> >> Version: 2.0.7-0ubuntu1~14.04.1
> >> Depends: lxc1 (>= 2.0.7-0ubuntu1~14.04.1)
> >> 
> >> I got it from here:
> >> 
> >> http://us.archive.ubuntu.com/ubuntu/ trusty-backports
> >> 
> >> Here’s what gets installed:
> >> 
> >> $ sudo apt-get install -t trusty-backports lxc
> >> Reading package lists... Done
> >> Building dependency tree       
> >> Reading state information... Done
> >> The following extra packages will be installed:
> >>  bridge-utils cgroup-lite cloud-image-utils debootstrap distro-info euca2ools
> >>  libgnutls28 libhogweed2 liblxc1 libseccomp2 lxc-common lxc-templates lxc1
> >>  python-distro-info python-requestbuilder python3-lxc uidmap
> >> Suggested packages:
> >>  shunit2 gnutls-bin btrfs-tools lvm2 lxctl
> >> Recommended packages:
> >>  lxcfs libpam-cgfs
> >> The following NEW packages will be installed:
> >>  bridge-utils cgroup-lite cloud-image-utils debootstrap distro-info euca2ools
> >>  libgnutls28 libhogweed2 liblxc1 libseccomp2 lxc lxc-common lxc-templates
> >>  lxc1 python-distro-info python-requestbuilder python3-lxc uidmap
> >> 
> >> As for the overall environment, this is a VM that was originally set up almost 3 years ago, and as a lab machine has only been piecemeal updated over time as needed. The problem is that I have probably a hundred identical instances and am concerned that the package dependencies are maybe not quite right.  I’m certainly willing to update whatever individual packages are necessary to get this going.  I have the VM snapshotted before trying this, so it’s trivial to reproduce.
> >> 
> >>> Is there anything in syslog about the failed mount?
> >>> 
> >> This is all I see.  It’s at lxc install time, now when trying to start the container:
> >> 
> >> May  7 21:01:01 ben-sc kernel: [  103.486718] type=1400 audit(1494216061.420:68): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=5801 comm="apparmor_parser"
> >> May  7 21:01:01 ben-sc kernel: [  103.486925] type=1400 audit(1494216061.420:69): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=5801 comm="apparmor_parser"
> >> May  7 21:01:01 ben-sc kernel: [  103.487100] type=1400 audit(1494216061.420:70): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=5801 comm="apparmor_parser"
> >> May  7 21:01:01 ben-sc kernel: [  103.487292] type=1400 audit(1494216061.420:71): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=5801 comm="apparmor_parser"
> >> May  7 21:01:01 ben-sc kernel: [  103.519003] type=1400 audit(1494216061.452:72): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=5835 comm="apparmor_parser"
> >> 
> >>> You might try some of the other cgroup auto-mount settings (see lxc.container.conf(5)0, maybe
> >>> 
> >>> lxc.mount.auto = cgroup:rw
> >>> 
> >> I tried that, and get:
> >> 
> >>      lxc-start 20170508041726.340 ERROR    lxc_cgfsng - cgroups/cgfsng.c:do_secondstage_mounts_if_needed:1557 - Operation not permitted - Error remounting /usr/lib/x86_64-linux-gnu/lxc/sys/fs/cgroup/cpu read-only
> >>      lxc-start 20170508041726.340 ERROR    lxc_conf - conf.c:lxc_mount_auto_mounts:839 - Operation not permitted - error mounting /sys/fs/cgroup
> >>      lxc-start 20170508041726.340 ERROR    lxc_conf - conf.c:lxc_setup:3885 - failed to setup the automatic mounts for 'cd-build'
> >>      lxc-start 20170508041726.340 ERROR    lxc_start - start.c:do_start:811 - Failed to setup container "cd-build".
> >>      lxc-start 20170508041726.340 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 3)
> >>      lxc-start 20170508041726.340 ERROR    lxc_start - start.c:__lxc_start:1346 - Failed to spawn container "cd-build".
> >> 
> >>>>     lxc-start 20170503225525.382 ERROR    lxc_conf - conf.c:lxc_mount_auto_mounts:839 - Operation not permitted - error mounting /sys/fs/cgroup
> >>>>     lxc-start 20170503225525.382 ERROR    lxc_conf - conf.c:lxc_setup:3885 - failed to setup the automatic mounts for 'cd-build'
> >>>>     lxc-start 20170503225525.382 ERROR    lxc_start - start.c:do_start:811 - Failed to setup container "cd-build".
> >>>>     lxc-start 20170503225525.382 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 3)
> >>>>     lxc-start 20170503225525.382 ERROR    lxc_start - start.c:__lxc_start:1346 - Failed to spawn container "cd-build".
> >>>>     lxc-start 20170503225530.922 ERROR    lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.
> >>>>     lxc-start 20170503225530.923 ERROR    lxc_start_ui - tools/lxc_start.c:main:368 - To get more details, run the container in foreground mode.
> >>>>     lxc-start 20170503225530.923 ERROR    lxc_start_ui - tools/lxc_start.c:main:370 - Additional information can be obtained by setting the --logfile and --logpriority options.
> >>>> 
> >>>> Also:
> >>>> 
> >>>> ————————————
> >>>> 
> >>>> ben at ben-sc:~$ cat /proc/self/cgroup 
> >>>> 12:name=dsystemd:/
> >>>> 11:name=systemd:/user/1001.user/c2.session
> >>>> 10:hugetlb:/user/1001.user/c2.session
> >>>> 9:perf_event:/user/1001.user/c2.session
> >>>> 8:blkio:/user/1001.user/c2.session
> >>>> 7:freezer:/user/1001.user/c2.session
> >>>> 6:devices:/user/1001.user/c2.session
> >>>> 5:memory:/user/1001.user/c2.session
> >>>> 4:cpuacct:/user/1001.user/c2.session
> >>>> 3:cpu:/user/1001.user/c2.session
> >>>> 2:cpuset:/
> > 
> > What the heck is cuasing this?  When I log in on a trusty+backports system,
> > I get:
> > 
> > ubuntu at trusty:~$ cat /proc/self/cgroup
> > 11:hugetlb:/user/1000.user/2.session
> > 10:perf_event:/user/1000.user/2.session
> > 9:blkio:/user/1000.user/2.session
> > 8:freezer:/user/1000.user/2.session
> > 7:devices:/user/1000.user/2.session
> > 6:memory:/user/1000.user/2.session
> > 5:cpuacct:/user/1000.user/2.session
> > 4:cpu:/user/1000.user/2.session
> > 3:cpuset:/user/1000.user/2.session
> > 2:name=systemd:/user/1000.user/2.session
> > 
> > (on second login)
> > 
> > ubuntu at trusty:~$ dpkg -l | egrep -e "(cgroup|lxc|cgfs)"
> > ii  cgmanager                        0.24-0ubuntu7.5                            amd64        Central cgroup manager daemon
> > ii  cgroup-lite                      1.11~ubuntu14.04.2                         all          Light-weight package to set up cgroups at system boot
> > ii  libcgmanager0:amd64              0.24-0ubuntu7.5                            amd64        Central cgroup manager daemon (client library)
> > ii  liblxc1                          2.0.7-0ubuntu1~14.04.1                     amd64        Linux Containers userspace tools (library)
> > ii  libpam-cgfs                      2.0.6-0ubuntu1~14.04.1                     amd64        PAM module for managing cgroups for LXC
> > ii  lxc                              2.0.7-0ubuntu1~14.04.1                     all          Transitional package for lxc1
> > ii  lxc-common                       2.0.7-0ubuntu1~14.04.1                     amd64        Linux Containers userspace tools (common tools)
> > ii  lxc-templates                    2.0.7-0ubuntu1~14.04.1                     amd64        Linux Containers userspace tools (templates)
> > ii  lxc1                             2.0.7-0ubuntu1~14.04.1                     amd64        Linux Containers userspace tools
> > ii  lxcfs                            2.0.6-0ubuntu1~14.04.1                     amd64        FUSE based filesystem for LXC
> > ii  python3-lxc                      2.0.7-0ubuntu1~14.04.1                     amd64        Linux Containers userspace tools (Python 3.x bindings)
> > 
> > Ah, now if I purge cgmanager, then upon login I get:
> > 
> > ubuntu at trusty:~$ cat /proc/self/cgroup
> > 11:name=systemd:/user/1000.user/1.session
> > 10:hugetlb:/user/1000.user/1.session
> > 9:perf_event:/user/1000.user/1.session
> > 8:blkio:/user/1000.user/1.session
> > 7:freezer:/user/1000.user/1.session
> > 6:devices:/user/1000.user/1.session
> > 5:memory:/user/1000.user/1.session
> > 4:cpuacct:/user/1000.user/1.session
> > 3:cpu:/user/1000.user/1.session
> > 2:cpuset:/
> > 
> > which looks more like yours, but my container still starts...
> 
> I’ve made some progress, but still don’t fully know what’s going on.  When I build lxc from source (top-of-tree github.com:lxc/lxc) and compile with full cgmanager and libcap support, the generated binaries work, and I can start not only my ‘trusty’ container, but also ones that are farther from the host, such as ‘delian-stretch’, which is systemd-based.
> 
> The difference I see in the log is which cgroup driver is used.
> When I build using the binaries from ’trusty-backports’, I see this:
>       lxc-start 20170509054154.989 INFO     lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for cd-build
> 
> When using the binaries I built from source, I see this:
>       lxc-start 20170509053256.861 INFO     lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgmanager initing for cd-build
> 
> Assuming cgmanager support is compiled in to the ‘trusty-backports’ version, the following code determines if the cgmanager driver is used (non-NULL return code means cgmanager is to be  used):
> 
> struct cgroup_ops *cgm_ops_init(void)
> {
> 	check_supports_multiple_controllers(-1);
> 	if (!collect_subsystems())
> 		return NULL;
> 
> 	if (api_version < CGM_SUPPORTS_MULT_CONTROLLERS)
> 		cgm_all_controllers_same = false;
> 
> 	// if root, try to escape to root cgroup
> 	if (geteuid() == 0 && !cgm_escape(NULL)) {
> 		free_subsystems();
> 		return NULL;
> 	}
> 
> 	return &cgmanager_ops;
> }
> 
> I have no context for how any of this is dependent on the environment, although I’m sure you do :)

Mine were starting with cgfsng which yours is using also, so you don't *need*
the cgmanager driver.  But I'm pretty sure that if you build your own with
it enabled it will work.

Is it possible that you have lxc.cgroup.use set in /etc/lxc/lxc.conf or in
~/.config/lxc/lxc.conf, and that it includes 'cpu'?  If so, assuming you
don't need it, removing cpu should work around this failure.

Does adding ',cpu" to the end of the pam_cgfs.so line in /etc/pam.d/common-session
help?

The other thing is back to your core problem - why is /sys/fs/cgroup/cpu not
remountable read-only?  It may be related to why you have a dsystemd cgroup
hierarchy.  Do you recall setting that up and/or why it's there?  Can you
show the contents of /proc/1/mounts and /proc/self/mounts on the host and a
fresh host boot log?


More information about the lxc-users mailing list