[lxc-users] Is this expected isolcpus behaviour with lxc

[Emmanuel Livingstone]

Hi,

We use lxc along with qemu/kvm in our boxes to provide containers and VMs.
We rely on isolcpus to ensure that the kernel scheduler by default does not
schedule OS processes on the cores that we've reserved for customer's
virtual instances(containers/VMs).
As per these 2 PRs(https://github.com/lxc/lxc/pull/1282,
https://github.com/lxc/lxc/pull/1309), cpuset.cpus inside cpuset:/lxc is
configured to have only those cpus that are not present in isolcpus. This
effectively means setting lxc.cgroup.cpuset.cpus for a lxc container to any
cpu that is part of isolcpus setting causes the container not to start.
This is because cpuset cgroup allows cpuset.cpus to be set to a value that
is a subset of the parent's value. Quoting from
https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt

For any cpuset, one can define child cpusets containing a subset
   of the parents CPU and Memory Node resources.

I've verified this behaviour

root at fk-cloud-none-1080125:/home/iaas# cgcreate -g cpuset:/test
root at fk-cloud-none-1080125:/home/iaas# echo 0-1 >
/sys/fs/cgroup/cpuset/test/cpuset.cpus
root at fk-cloud-none-1080125:/home/iaas# cgcreate -g cpuset:/test/container
root at fk-cloud-none-1080125:/home/iaas# echo 0 >
/sys/fs/cgroup/cpuset/test/container/cpuset.cpus
root at fk-cloud-none-1080125:/home/iaas# echo 1 >
/sys/fs/cgroup/cpuset/test/container/cpuset.cpus
root at fk-cloud-none-1080125:/home/iaas# echo 2 >
/sys/fs/cgroup/cpuset/test/container/cpuset.cpus
bash: echo: write error: Permission denied

In our case, we have 48 core boxes on which we reserve 4 cores for OS
processes and remaining 44 cores for customers virtual instances.
It was working fine for us until we upgraded from debian 8 to debian 9,
i.e., kernel 3.16 to 4.9. In kernel 3.16 even if isolcpus was enabled, the
file /sys/devices/system/cpu/isolated was missing and is available in 4.9
through this commit
https://github.com/torvalds/linux/commit/59f30abe94bff50636c8cad45207a01fdcb2ee49

I had come across a recommended way of overcoming this issue in this comment
<https://discuss.linuxcontainers.org/t/lxc-lxd-2-0-11-isolcpu-limits-cpu/1845/6>
by stgraber where he had recommended that all system processes should be
put inside a restricted cpuset cgroup and the remaining cores to be used
for scheduling by lxc/lxd using a script. Is there alternate way in systemd
to do this without writing an ad-hoc startup script.

Also since we use both libvirt(qemu/kvm) and lxc(without libvirt), I was
wondering why the behaviour of isolcpus is different in both these cases.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20180606/9faa82d9/attachment-0001.html>