[lxc-devel] [RFC] cgmanager and lxc: handle unified hierarchy

[Serge Hallyn]

Hi,

following are two patches which enable cgmanager and lxc to run on top of
the unified cgroup hierarchy.  

So far cgmanager and lxc (as most userspace software) has only supported the
legacy hierarchy.  Here each controller can be mounted in separate hierarchies
or co-mounted as desired (so for instance memory controller can be administered
per-uid, while cpuset could be administered per-application indepedent of
uids).  Tasks may be added to any cgroup.  When a new cgroup is created, it
automatically is setup to use the co-mounted controllers.

With the unified hierarchy, there is only one hierarchy.  That's not really
a problem for cgmanager and lxc, though it may be for some users.  More
importantly, (a) newly created cgroups have no enabled controllers until
they are manually enabled using the cgroup.subtree_control file, and (b) tasks
may only exist in leaf nodes with no controllers enabled.

This patchset updates cgmanager to offer the existing (legacy-based) API
over both legacy and unified hierarchies.  (This leaves us free to (soon)
offer a v2 API (alongside the v1 API) which is less filesystem-like and
a more general resource limit API.)  When a v1 API call is made with the
unified hierarchy mounted, cgmanager will emulate the legacy API.  Any
Create call will create a cgroup with all controllers enabled, while a
MovePid will create a private leaf node called ".cgm_leaf" with no controllers
enabled, and move tasks there.  SetValue will set the requested value to
both the cgroup and, if it exists, the leaf directory.

The current patches allow both privileged and unprivileged containers to
be started.  It should also allow lxc in containers running older releases
(with minimal, SRU-able changes) to be run on future hosts with the
unified hierarchy, i.e. Ubuntu 16.04.

========================  Testing ==========================

To use the unified hierarchy on an ubuntu vivid or wily system, I had to

1. install upstart, as systemd does not yet support it:

	sudo apt-get install upstart-bin upstart-sysv

2. add a boot flag to allow legacy controllers to be mounted under the unified
hierarchy:

	sed -i '/^GRUB_CMDLINE_LINUX=/s/"$/ cgroup__DEVEL__legacy_files_on_dfl"/' /etc/default/grub
	update-grub

3. disable cgmanager from starting at boot:

	cat > /etc/init/cgmanager.override << EOF
manual
EOF

4. I created an upstart job to mount the unified hierarchy:

description "mount unified cgroup"
author "Serge Hallyn <serge.hallyn at ubuntu.com>"
task
manual
script
        if ! mountpoint /sys/fs/cgroup; then
                mount -t tmpfs tmpfs /sys/fs/cgroup
        fi
        mkdir /sys/fs/cgroup/unified
        mount -t cgroup -o __DEVEL__sane_behavior unified /sys/fs/cgroup/unified
end script

then did 'sudo start unified' when I wanted the unified hierarchy (and
simply started cgmanager if I did not)

That should be it.

-serge

Note, lxcfs is not yet enabled and will need a few changes of its own.
More work is also needed in cgmanager - tests need to be updated, and
we need to refuse any API calls to administer the ".cgm_leaf" directory.