[lxc-users] Can't start unprivileged container in Ubuntu 14.04 with LXC 2

[Serge Hallyn]

after adding Cpu to common-session, did you log back in? Actually I suspect that you did, since the remount error this time is about cpuset.

You could try two more things,

1. Set lxc.cgroup.use in your ~/.config/lxc/lxc.conf to 'freezer,name=systemd

2. You could try installing cgroup-lite or cgroupfs-mount package, to make sure that /sys/fs/cgroup/controller is mounted for every controller you need. From your /proc/self/cgroup it doesn't look like they are, which could cause your problem. 


  Original Message  
From: Ben Warren
Sent: Tuesday, May 9, 2017 11:40 AM
To: Serge E. Hallyn
Cc: lxc-users at lists.linuxcontainers.org
Subject: Re: [lxc-users] Can't start unprivileged container in Ubuntu 14.04 with LXC 2


> On May 9, 2017, at 8:10 AM, Serge E. Hallyn <serge at hallyn.com> wrote:
> 
<snip>
> 
>> 
>> I’ve made some progress, but still don’t fully know what’s going on. When I build lxc from source (top-of-tree github.com:lxc/lxc) and compile with full cgmanager and libcap support, the generated binaries work, and I can start not only my ‘trusty’ container, but also ones that are farther from the host, such as ‘delian-stretch’, which is systemd-based.
>> 
>> The difference I see in the log is which cgroup driver is used.
>> When I build using the binaries from ’trusty-backports’, I see this:
>> lxc-start 20170509054154.989 INFO lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for cd-build
>> 
>> When using the binaries I built from source, I see this:
>> lxc-start 20170509053256.861 INFO lxc_cgroup - cgroups/cgroup.c:cgroup_init:68 - cgroup driver cgmanager initing for cd-build
>> 
>> Assuming cgmanager support is compiled in to the ‘trusty-backports’ version, the following code determines if the cgmanager driver is used (non-NULL return code means cgmanager is to be used):
>> 
>> struct cgroup_ops *cgm_ops_init(void)
>> {
>> check_supports_multiple_controllers(-1);
>> if (!collect_subsystems())
>> return NULL;
>> 
>> if (api_version < CGM_SUPPORTS_MULT_CONTROLLERS)
>> cgm_all_controllers_same = false;
>> 
>> // if root, try to escape to root cgroup
>> if (geteuid() == 0 && !cgm_escape(NULL)) {
>> free_subsystems();
>> return NULL;
>> }
>> 
>> return &cgmanager_ops;
>> }
>> 
>> I have no context for how any of this is dependent on the environment, although I’m sure you do :)
> 
> Mine were starting with cgfsng which yours is using also, so you don't *need*
> the cgmanager driver. But I'm pretty sure that if you build your own with
> it enabled it will work.
> 
> Is it possible that you have lxc.cgroup.use set in /etc/lxc/lxc.conf or in
> ~/.config/lxc/lxc.conf, and that it includes 'cpu'? If so, assuming you
> don't need it, removing cpu should work around this failure.
> 
Neither of these files is present. This is it for config:

ben at ben-sc:~/tmp/lxc/src$ cat /etc/lxc/default.conf 
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx
ben at ben-sc:~/tmp/lxc/src$ cat ~/.config/lxc/default.conf 
lxc.id_map = u 0 165536 65536
lxc.id_map = g 0 165536 65536

> Does adding ',cpu" to the end of the pam_cgfs.so line in /etc/pam.d/common-session
> help?
> 
I added like this:

session optional pam_cgfs.so -c freezer,memory,cpu,name=systemd

but it doesn’t seem to make a difference
> The other thing is back to your core problem - why is /sys/fs/cgroup/cpu not
> remountable read-only? It may be related to why you have a dsystemd cgroup
> hierarchy. Do you recall setting that up and/or why it's there? Can you
> show the contents of /proc/1/mounts and /proc/self/mounts on the host and a
> fresh host boot log?

I think the dsystemd thing was left over from me trying something else. It’s not there now, after reverting to before any LXC installation and just installing the backports version of lxc.

Here’s the current state. If I run ‘lxc-start’ runtime-linked against the ‘back ports’ shared libraries I get this message:
lxc-start 20170509161114.691 INFO lxc_conf - conf.c:mount_file_entries:1985 - mount points have been setup
lxc-start 20170509161114.691 ERROR lxc_cgfsng - cgroups/cgfsng.c:do_secondstage_mounts_if_needed:1557 - Operation not permitted - Error remounting /usr/lib/x86_64-linux-gnu/lxc/sys/fs/cgroup/cpuset read-only
lxc-start 20170509161114.691 ERROR lxc_conf - conf.c:lxc_mount_auto_mounts:839 - Operation not permitted - error mounting /sys/fs/cgroup

If I change LD_LIBRARY_PATH to use the .so that I built, the container start as previously mentioned, using cgmanager.

ben at ben-sc:~$ cat /proc/self/cgroup
11:name=systemd:/user/1001.user/c2.session
10:perf_event:/user/1001.user/c2.session
9:memory:/user/1001.user/c2.session
8:hugetlb:/user/1001.user/c2.session
7:freezer:/user/1001.user/c2.session
6:devices:/user/1001.user/c2.session
5:cpuacct:/user/1001.user/c2.session
4:blkio:/user/1001.user/c2.session
3:cpu:/user/1001.user/c2.session
2:cpuset:/user/1001.user/c2.session

ben at ben-sc:~$ cat /proc/1/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=4073948k,nr_inodes=1018487,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=816968k,mode=755 0 0
/dev/disk/by-uuid/0fdaee58-1394-4338-9eed-95ab207f0de6 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset,clone_children 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
cgmfs /run/cgmanager/fs tmpfs rw,relatime,size=100k,mode=755 0 0
cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuacct 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory,release_agent=/run/cgmanager/agents/cgm-release-agent.memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices,release_agent=/run/cgmanager/agents/cgm-release-agent.devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer,release_agent=/run/cgmanager/agents/cgm-release-agent.freezer 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio,release_agent=/run/cgmanager/agents/cgm-release-agent.blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb 0 0
name=systemd /sys/fs/cgroup/systemd cgroup rw,relatime,release_agent=/run/cgmanager/agents/cgm-release-agent.systemd,name=systemd 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
gvfsd-fuse /run/user/1001/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1001,group_id=1001 0 0

ben at ben-sc:~$ cat /proc/self/mounts
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=4073948k,nr_inodes=1018487,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=816968k,mode=755 0 0
/dev/disk/by-uuid/0fdaee58-1394-4338-9eed-95ab207f0de6 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
none /sys/fs/cgroup tmpfs rw,relatime,size=4k,mode=755 0 0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /run/user tmpfs rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755 0 0
none /sys/fs/pstore pstore rw,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset,clone_children 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
cgmfs /run/cgmanager/fs tmpfs rw,relatime,size=100k,mode=755 0 0
cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuacct 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory,release_agent=/run/cgmanager/agents/cgm-release-agent.memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices,release_agent=/run/cgmanager/agents/cgm-release-agent.devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer,release_agent=/run/cgmanager/agents/cgm-release-agent.freezer 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio,release_agent=/run/cgmanager/agents/cgm-release-agent.blkio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb 0 0
name=systemd /sys/fs/cgroup/systemd cgroup rw,relatime,release_agent=/run/cgmanager/agents/cgm-release-agent.systemd,name=systemd 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
gvfsd-fuse /run/user/1001/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1001,group_id=1001 0 0