[lxc-devel] cgfs vs unified hierachy

[Serge Hallyn]

Hi,

Yesterday I was testing a small patch to test containers with the cgroup
namespace patchset from Aditya (cc:d).  Currently that features only
works with the unified hierarchy, so my first step was actually to make
containers work at all with a unified hierarchy.  I hit several issues
which I worked around, but am hoping someone else will send a proper fix
for.

Firstly, in order to test this yourself, use a recent enough kernel (I
used three 3.18 kernels - the stock Ubuntu Vivid kernel, a hand-build
3.18-rc7, and a hand-build kernel from
https://github.com/adityakali/linux/tree/cgroupns_v3).  Boot with the
option cgroup__DEVEL__legacy_files_on_dfl, and do a mount like

mkdir -p /sys/fs/cgroup/unified
mount -t cgroup -o __DEVEL__sane_behavior cgroup /sys/fs/cgroup/unified

early on.

So on with the issues:

1. Hierarchy_num in /proc/cgroups and /proc/self/cgroup start at 0.
lxc appears to depend on them starting with 1.  I worked around it by
bumping the hierarchy_num that we read by one :)

2. In unified hierarchy we cannot create tasks in a cgroup which will
have sub-cgroups.  So when creating a cg for /lxc/c1, we will want to
make /lxc/c1 owned by the container, but put the container into
/lxc/c1/leaf.  I worked around this by just making /lxc/c1 a leaf
node, which is not acceptable for a real solution.

3. Every cgroup we create which is not a leaf node will need its
cgroup.subtree_cgroups file filled with "+cpu +cpuset +cpuacct..."
(I worked around this for /lxc with a function that reads
cgroup.controllers, prepends every word with '+', and writes that
to cgroup.subtree_cgroups - iow i hardcoded it for /lxc, but we
need to do it more generally for all non-leaf nodes)

4. Tasks file no longer exists, so we will need to use cgroup.procs.

-serge

PS - the good news is it did work, I ended up with a container in
cgroup /lxc/t1 which saw itself as being under /.

PS2 - I do hope that cgroupns can be made to work with non-unified
hierarchies.  While lxc should definately be fixed to support the
unified hierarchy, it seems clear that older userspace in containers on
newer hosts will have problems - for instance (let's say) a Ubuntu Trusty
container running on a Ubuntu vivid system with unified hierarchy will
have serious issues, if only because of the cgroup.subtree_cgroups file
and lack of tasks file.

PS3 - I didn't test unprivileged containers, but did test unprivileged
clone(CLONE_NEWCG) by hand, so at least that bit works.  I expect other
problems in unified hierarchy due to capable() vs ns_capable() calls)