[lxc-users] Is this expected isolcpus behaviour with lxc

[Christian Brauner]

On Thu, Jun 28, 2018 at 11:07:38AM +0530, Emmanuel Livingstone wrote:
> For now to overcome this issue, we've added a patch to an internal mirror
> of lxc debian repo https://anonscm.debian.org/git/pkg-lxc/lxc.git
> If you guys think adding a config parameter to not mask out the isolcpus in
> cpuset:/lxc would be welcome, please let me know. I can work on a patch and
> submit the same.

Hm, the main problem with this is that there's no nice way of adding a
config item that would allow one to configure this. Otherwise I'd be
fine with it.

> 
> On Wed, Jun 6, 2018 at 9:34 AM Emmanuel Livingstone <
> livingstone.s.e at gmail.com> wrote:
> 
> > Hi,
> >
> > We use lxc along with qemu/kvm in our boxes to provide containers and VMs.
> > We rely on isolcpus to ensure that the kernel scheduler by default does not
> > schedule OS processes on the cores that we've reserved for customer's
> > virtual instances(containers/VMs).
> > As per these 2 PRs(https://github.com/lxc/lxc/pull/1282,
> > https://github.com/lxc/lxc/pull/1309), cpuset.cpus inside cpuset:/lxc is
> > configured to have only those cpus that are not present in isolcpus. This
> > effectively means setting lxc.cgroup.cpuset.cpus for a lxc container to any
> > cpu that is part of isolcpus setting causes the container not to start.
> > This is because cpuset cgroup allows cpuset.cpus to be set to a value that
> > is a subset of the parent's value. Quoting from
> > https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt
> >
> > For any cpuset, one can define child cpusets containing a subset
> >    of the parents CPU and Memory Node resources.
> >
> > I've verified this behaviour
> >
> > root at fk-cloud-none-1080125:/home/iaas# cgcreate -g cpuset:/test
> > root at fk-cloud-none-1080125:/home/iaas# echo 0-1 >
> > /sys/fs/cgroup/cpuset/test/cpuset.cpus
> > root at fk-cloud-none-1080125:/home/iaas# cgcreate -g cpuset:/test/container
> > root at fk-cloud-none-1080125:/home/iaas# echo 0 >
> > /sys/fs/cgroup/cpuset/test/container/cpuset.cpus
> > root at fk-cloud-none-1080125:/home/iaas# echo 1 >
> > /sys/fs/cgroup/cpuset/test/container/cpuset.cpus
> > root at fk-cloud-none-1080125:/home/iaas# echo 2 >
> > /sys/fs/cgroup/cpuset/test/container/cpuset.cpus
> > bash: echo: write error: Permission denied
> >
> > In our case, we have 48 core boxes on which we reserve 4 cores for OS
> > processes and remaining 44 cores for customers virtual instances.
> > It was working fine for us until we upgraded from debian 8 to debian 9,
> > i.e., kernel 3.16 to 4.9. In kernel 3.16 even if isolcpus was enabled, the
> > file /sys/devices/system/cpu/isolated was missing and is available in 4.9
> > through this commit
> > https://github.com/torvalds/linux/commit/59f30abe94bff50636c8cad45207a01fdcb2ee49
> >
> > I had come across a recommended way of overcoming this issue in this
> > comment
> > <https://discuss.linuxcontainers.org/t/lxc-lxd-2-0-11-isolcpu-limits-cpu/1845/6>
> > by stgraber where he had recommended that all system processes should be
> > put inside a restricted cpuset cgroup and the remaining cores to be used
> > for scheduling by lxc/lxd using a script. Is there alternate way in systemd
> > to do this without writing an ad-hoc startup script.
> >
> > Also since we use both libvirt(qemu/kvm) and lxc(without libvirt), I was
> > wondering why the behaviour of isolcpus is different in both these cases.
> >

> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users