[lxc-users] Is this expected isolcpus behaviour with lxc

[Emmanuel Livingstone]

For now to overcome this issue, we've added a patch to an internal mirror
of lxc debian repo https://anonscm.debian.org/git/pkg-lxc/lxc.git
If you guys think adding a config parameter to not mask out the isolcpus in
cpuset:/lxc would be welcome, please let me know. I can work on a patch and
submit the same.

On Wed, Jun 6, 2018 at 9:34 AM Emmanuel Livingstone <
livingstone.s.e at gmail.com> wrote:

> Hi,
>
> We use lxc along with qemu/kvm in our boxes to provide containers and VMs.
> We rely on isolcpus to ensure that the kernel scheduler by default does not
> schedule OS processes on the cores that we've reserved for customer's
> virtual instances(containers/VMs).
> As per these 2 PRs(https://github.com/lxc/lxc/pull/1282,
> https://github.com/lxc/lxc/pull/1309), cpuset.cpus inside cpuset:/lxc is
> configured to have only those cpus that are not present in isolcpus. This
> effectively means setting lxc.cgroup.cpuset.cpus for a lxc container to any
> cpu that is part of isolcpus setting causes the container not to start.
> This is because cpuset cgroup allows cpuset.cpus to be set to a value that
> is a subset of the parent's value. Quoting from
> https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt
>
> For any cpuset, one can define child cpusets containing a subset
>    of the parents CPU and Memory Node resources.
>
> I've verified this behaviour
>
> root at fk-cloud-none-1080125:/home/iaas# cgcreate -g cpuset:/test
> root at fk-cloud-none-1080125:/home/iaas# echo 0-1 >
> /sys/fs/cgroup/cpuset/test/cpuset.cpus
> root at fk-cloud-none-1080125:/home/iaas# cgcreate -g cpuset:/test/container
> root at fk-cloud-none-1080125:/home/iaas# echo 0 >
> /sys/fs/cgroup/cpuset/test/container/cpuset.cpus
> root at fk-cloud-none-1080125:/home/iaas# echo 1 >
> /sys/fs/cgroup/cpuset/test/container/cpuset.cpus
> root at fk-cloud-none-1080125:/home/iaas# echo 2 >
> /sys/fs/cgroup/cpuset/test/container/cpuset.cpus
> bash: echo: write error: Permission denied
>
> In our case, we have 48 core boxes on which we reserve 4 cores for OS
> processes and remaining 44 cores for customers virtual instances.
> It was working fine for us until we upgraded from debian 8 to debian 9,
> i.e., kernel 3.16 to 4.9. In kernel 3.16 even if isolcpus was enabled, the
> file /sys/devices/system/cpu/isolated was missing and is available in 4.9
> through this commit
> https://github.com/torvalds/linux/commit/59f30abe94bff50636c8cad45207a01fdcb2ee49
>
> I had come across a recommended way of overcoming this issue in this
> comment
> <https://discuss.linuxcontainers.org/t/lxc-lxd-2-0-11-isolcpu-limits-cpu/1845/6>
> by stgraber where he had recommended that all system processes should be
> put inside a restricted cpuset cgroup and the remaining cores to be used
> for scheduling by lxc/lxd using a script. Is there alternate way in systemd
> to do this without writing an ad-hoc startup script.
>
> Also since we use both libvirt(qemu/kvm) and lxc(without libvirt), I was
> wondering why the behaviour of isolcpus is different in both these cases.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20180628/ff1af74b/attachment.html>