[Lxc-users] can't restart container without rebooting entire host, because can't delete cgroups files, tasks is 0

Mon Nov 15 14:26:12 UTC 2010

Hello,

	please see:

ls -l
total 0
drwxr-xr-x 4 root root 0 2010-11-15 15:00 1285
drwxr-xr-x 4 root root 0 2010-11-15 15:00 1298
drwxr-xr-x 4 root root 0 2010-11-15 15:01 1322
drwxr-xr-x 4 root root 0 2010-11-15 15:01 1325
drwxr-xr-x 4 root root 0 2010-11-15 15:02 1335
drwxr-xr-x 4 root root 0 2010-11-15 15:09 1386
drwxr-xr-x 4 root root 0 2010-11-15 15:11 1401
drwxr-xr-x 4 root root 0 2010-11-15 15:12 1408
drwxr-xr-x 4 root root 0 2010-11-15 15:12 1411
drwxr-xr-x 3 root root 0 2010-11-15 15:17 1459
-r--r--r-- 1 root root 0 2010-11-15 15:02 cgroup.procs
-r--r--r-- 1 root root 0 2010-11-15 15:02 cpuacct.stat
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuacct.usage
-r--r--r-- 1 root root 0 2010-11-15 15:02 cpuacct.usage_percpu
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpu.rt_period_us
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpu.rt_runtime_us
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.cpu_exclusive
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.cpus
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.mem_exclusive
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.mem_hardwall
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.memory_migrate
-r--r--r-- 1 root root 0 2010-11-15 15:02 cpuset.memory_pressure
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.memory_spread_page
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.memory_spread_slab
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.mems
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.sched_load_balance
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.sched_relax_domain_level
-rw-r--r-- 1 root root 0 2010-11-15 15:02 cpu.shares
--w------- 1 root root 0 2010-11-15 15:02 devices.allow
--w------- 1 root root 0 2010-11-15 15:02 devices.deny
-r--r--r-- 1 root root 0 2010-11-15 15:02 devices.list
-rw-r--r-- 1 root root 0 2010-11-15 15:02 freezer.state
-rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.failcnt
--w------- 1 root root 0 2010-11-15 15:02 memory.force_empty
-rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.limit_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.max_usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.memsw.failcnt
-rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.memsw.limit_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.memsw.max_usage_in_bytes
-r--r--r-- 1 root root 0 2010-11-15 15:02 memory.memsw.usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.soft_limit_in_bytes
-r--r--r-- 1 root root 0 2010-11-15 15:02 memory.stat
-rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.swappiness
-r--r--r-- 1 root root 0 2010-11-15 15:02 memory.usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.use_hierarchy
-rw-r--r-- 1 root root 0 2010-11-15 15:02 net_cls.classid
-rw-r--r-- 1 root root 0 2010-11-15 15:02 notify_on_release
-rw-r--r-- 1 root root 0 2010-11-15 15:02 tasks
root at lnx-zl-teaspl:/cgroup/teas_www# ls -lR 1285
1285:
total 0
drwxr-xr-x 2 root root 0 2010-11-15 15:00 2
drwxr-xr-x 2 root root 0 2010-11-15 15:00 3
-r--r--r-- 1 root root 0 2010-11-15 15:00 cgroup.procs
-r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.stat
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage
-r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage_percpu
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_period_us
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_runtime_us
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpu_exclusive
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpus
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_exclusive
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_hardwall
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_migrate
-r--r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_pressure
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_page
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_slab
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mems
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_load_balance
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_relax_domain_level
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.shares
--w------- 1 root root 0 2010-11-15 15:00 devices.allow
--w------- 1 root root 0 2010-11-15 15:00 devices.deny
-r--r--r-- 1 root root 0 2010-11-15 15:00 devices.list
-rw-r--r-- 1 root root 0 2010-11-15 15:00 freezer.state
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.failcnt
--w------- 1 root root 0 2010-11-15 15:00 memory.force_empty
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.limit_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.max_usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.failcnt
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.limit_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.max_usage_in_bytes
-r--r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.soft_limit_in_bytes
-r--r--r-- 1 root root 0 2010-11-15 15:00 memory.stat
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.swappiness
-r--r--r-- 1 root root 0 2010-11-15 15:00 memory.usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.use_hierarchy
-rw-r--r-- 1 root root 0 2010-11-15 15:00 net_cls.classid
-rw-r--r-- 1 root root 0 2010-11-15 15:00 notify_on_release
-rw-r--r-- 1 root root 0 2010-11-15 15:00 tasks

1285/2:
total 0
-r--r--r-- 1 root root 0 2010-11-15 15:00 cgroup.procs
-r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.stat
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage
-r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage_percpu
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_period_us
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_runtime_us
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpu_exclusive
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpus
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_exclusive
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_hardwall
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_migrate
-r--r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_pressure
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_page
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_slab
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mems
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_load_balance
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_relax_domain_level
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.shares
--w------- 1 root root 0 2010-11-15 15:00 devices.allow
--w------- 1 root root 0 2010-11-15 15:00 devices.deny
-r--r--r-- 1 root root 0 2010-11-15 15:00 devices.list
-rw-r--r-- 1 root root 0 2010-11-15 15:00 freezer.state
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.failcnt
--w------- 1 root root 0 2010-11-15 15:00 memory.force_empty
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.limit_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.max_usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.failcnt
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.limit_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.max_usage_in_bytes
-r--r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.soft_limit_in_bytes
-r--r--r-- 1 root root 0 2010-11-15 15:00 memory.stat
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.swappiness
-r--r--r-- 1 root root 0 2010-11-15 15:00 memory.usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.use_hierarchy
-rw-r--r-- 1 root root 0 2010-11-15 15:00 net_cls.classid
-rw-r--r-- 1 root root 0 2010-11-15 15:00 notify_on_release
-rw-r--r-- 1 root root 0 2010-11-15 15:00 tasks

1285/3:
total 0
-r--r--r-- 1 root root 0 2010-11-15 15:00 cgroup.procs
-r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.stat
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage
-r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage_percpu
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_period_us
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_runtime_us
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpu_exclusive
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpus
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_exclusive
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_hardwall
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_migrate
-r--r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_pressure
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_page
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_slab
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mems
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_load_balance
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_relax_domain_level
-rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.shares
--w------- 1 root root 0 2010-11-15 15:00 devices.allow
--w------- 1 root root 0 2010-11-15 15:00 devices.deny
-r--r--r-- 1 root root 0 2010-11-15 15:00 devices.list
-rw-r--r-- 1 root root 0 2010-11-15 15:00 freezer.state
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.failcnt
--w------- 1 root root 0 2010-11-15 15:00 memory.force_empty
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.limit_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.max_usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.failcnt
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.limit_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.max_usage_in_bytes
-r--r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.soft_limit_in_bytes
-r--r--r-- 1 root root 0 2010-11-15 15:00 memory.stat
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.swappiness
-r--r--r-- 1 root root 0 2010-11-15 15:00 memory.usage_in_bytes
-rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.use_hierarchy
-rw-r--r-- 1 root root 0 2010-11-15 15:00 net_cls.classid
-rw-r--r-- 1 root root 0 2010-11-15 15:00 notify_on_release
-rw-r--r-- 1 root root 0 2010-11-15 15:00 tasks

It is contentof my cgroup directory with running LXC. There are
directories:

drwxr-xr-x 4 root root 0 2010-11-15 15:00 1285
drwxr-xr-x 4 root root 0 2010-11-15 15:00 1298
drwxr-xr-x 4 root root 0 2010-11-15 15:01 1322
drwxr-xr-x 4 root root 0 2010-11-15 15:01 1325
drwxr-xr-x 4 root root 0 2010-11-15 15:02 1335
drwxr-xr-x 4 root root 0 2010-11-15 15:09 1386
drwxr-xr-x 4 root root 0 2010-11-15 15:11 1401
drwxr-xr-x 4 root root 0 2010-11-15 15:12 1408
drwxr-xr-x 4 root root 0 2010-11-15 15:12 1411
drwxr-xr-x 3 root root 0 2010-11-15 15:17 1459

These PIDs are no in global proc filesystem
and still increasing.

Load of my machine is still higher and higher.

I will must restart computer ant the end. :-(

What can i do with it?

Miroslav

P.S.: Sorry for my bad english.

Dne 8.11.2010 15:25, Daniel Lezcano napsal(a):
> On 11/05/2010 09:27 PM, Serge E. Hallyn wrote:
>> Quoting Miroslav Lednicky (lednicky at avonet.cz):
>>> Hello,
>>>
>>> it is the same problem as I have may be:
>>>
>>> http://sourceforge.net/mailarchive/forum.php?thread_name=4CD297E2.2030806%40avonet.cz&forum_name=lxc-users
>>>
>>>
>>> I don't know what can I do with it.
>>>
>>> Server reboot is necessary ...
>> I fired off a container and did a bunch of ns_exec's under it to create
>> some cgroup subdirectories. When I powered off the container, lxc did
>> indeed complain about not being able to remove /cgroup/ubuntu1. I did
>>
>> find /cgroup/ubuntu1 -type d -print0 | xargs -0 rmdir
>>
>> which deleted the subdirectories and complainted about not being able
>> to remove ubuntu1 itself. Then I did 'rmdir /cgroup/ubuntu1' and it
>> worked.
>>
>> So the only problem with my find statement is that it doesn't work
>> depth-first, so you might have to do it a few times to get all of the
>> subdirectories.
>>
>> Daniel, I think it would be worthwhile for lxc, on exiting a container,
>> to do a depth-first search or all directories under the container's
>> cgroup, and rmdir them all. What do you think?
>
> Yep, agree.
>
>

-- 
Miroslav Lednicky, AVONET, s.r.o.