[Lxc-users] can't restart container without rebooting entire host, because can't delete cgroups files, tasks is 0

Mon Nov 15 15:17:26 UTC 2010

Dne 15.11.2010 15:56, Daniel Lezcano napsal(a):
> On 11/15/2010 03:26 PM, Miroslav Lednicky, AVONET, s.r.o. wrote:
>> Hello,
>>
>> please see:
>>
>> ls -l
>> total 0
>> drwxr-xr-x 4 root root 0 2010-11-15 15:00 1285
>> drwxr-xr-x 4 root root 0 2010-11-15 15:00 1298
>> drwxr-xr-x 4 root root 0 2010-11-15 15:01 1322
>> drwxr-xr-x 4 root root 0 2010-11-15 15:01 1325
>> drwxr-xr-x 4 root root 0 2010-11-15 15:02 1335
>> drwxr-xr-x 4 root root 0 2010-11-15 15:09 1386
>> drwxr-xr-x 4 root root 0 2010-11-15 15:11 1401
>> drwxr-xr-x 4 root root 0 2010-11-15 15:12 1408
>> drwxr-xr-x 4 root root 0 2010-11-15 15:12 1411
>> drwxr-xr-x 3 root root 0 2010-11-15 15:17 1459
>> -r--r--r-- 1 root root 0 2010-11-15 15:02 cgroup.procs
>> -r--r--r-- 1 root root 0 2010-11-15 15:02 cpuacct.stat
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuacct.usage
>> -r--r--r-- 1 root root 0 2010-11-15 15:02 cpuacct.usage_percpu
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpu.rt_period_us
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpu.rt_runtime_us
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.cpu_exclusive
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.cpus
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.mem_exclusive
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.mem_hardwall
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.memory_migrate
>> -r--r--r-- 1 root root 0 2010-11-15 15:02 cpuset.memory_pressure
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.memory_spread_page
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.memory_spread_slab
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.mems
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.sched_load_balance
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpuset.sched_relax_domain_level
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 cpu.shares
>> --w------- 1 root root 0 2010-11-15 15:02 devices.allow
>> --w------- 1 root root 0 2010-11-15 15:02 devices.deny
>> -r--r--r-- 1 root root 0 2010-11-15 15:02 devices.list
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 freezer.state
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.failcnt
>> --w------- 1 root root 0 2010-11-15 15:02 memory.force_empty
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.limit_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.max_usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.memsw.failcnt
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.memsw.limit_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.memsw.max_usage_in_bytes
>> -r--r--r-- 1 root root 0 2010-11-15 15:02 memory.memsw.usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.soft_limit_in_bytes
>> -r--r--r-- 1 root root 0 2010-11-15 15:02 memory.stat
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.swappiness
>> -r--r--r-- 1 root root 0 2010-11-15 15:02 memory.usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 memory.use_hierarchy
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 net_cls.classid
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 notify_on_release
>> -rw-r--r-- 1 root root 0 2010-11-15 15:02 tasks
>> root at lnx-zl-teaspl:/cgroup/teas_www# ls -lR 1285
>> 1285:
>> total 0
>> drwxr-xr-x 2 root root 0 2010-11-15 15:00 2
>> drwxr-xr-x 2 root root 0 2010-11-15 15:00 3
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cgroup.procs
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.stat
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage_percpu
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_period_us
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_runtime_us
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpu_exclusive
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpus
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_exclusive
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_hardwall
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_migrate
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_pressure
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_page
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_slab
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mems
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_load_balance
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_relax_domain_level
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.shares
>> --w------- 1 root root 0 2010-11-15 15:00 devices.allow
>> --w------- 1 root root 0 2010-11-15 15:00 devices.deny
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 devices.list
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 freezer.state
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.failcnt
>> --w------- 1 root root 0 2010-11-15 15:00 memory.force_empty
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.limit_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.max_usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.failcnt
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.limit_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.max_usage_in_bytes
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.soft_limit_in_bytes
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 memory.stat
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.swappiness
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 memory.usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.use_hierarchy
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 net_cls.classid
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 notify_on_release
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 tasks
>>
>> 1285/2:
>> total 0
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cgroup.procs
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.stat
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage_percpu
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_period_us
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_runtime_us
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpu_exclusive
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpus
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_exclusive
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_hardwall
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_migrate
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_pressure
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_page
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_slab
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mems
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_load_balance
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_relax_domain_level
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.shares
>> --w------- 1 root root 0 2010-11-15 15:00 devices.allow
>> --w------- 1 root root 0 2010-11-15 15:00 devices.deny
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 devices.list
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 freezer.state
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.failcnt
>> --w------- 1 root root 0 2010-11-15 15:00 memory.force_empty
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.limit_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.max_usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.failcnt
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.limit_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.max_usage_in_bytes
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.soft_limit_in_bytes
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 memory.stat
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.swappiness
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 memory.usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.use_hierarchy
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 net_cls.classid
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 notify_on_release
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 tasks
>>
>> 1285/3:
>> total 0
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cgroup.procs
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.stat
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cpuacct.usage_percpu
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_period_us
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.rt_runtime_us
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpu_exclusive
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.cpus
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_exclusive
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mem_hardwall
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_migrate
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_pressure
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_page
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.memory_spread_slab
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.mems
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_load_balance
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpuset.sched_relax_domain_level
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 cpu.shares
>> --w------- 1 root root 0 2010-11-15 15:00 devices.allow
>> --w------- 1 root root 0 2010-11-15 15:00 devices.deny
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 devices.list
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 freezer.state
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.failcnt
>> --w------- 1 root root 0 2010-11-15 15:00 memory.force_empty
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.limit_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.max_usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.failcnt
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.limit_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.max_usage_in_bytes
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 memory.memsw.usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.soft_limit_in_bytes
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 memory.stat
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.swappiness
>> -r--r--r-- 1 root root 0 2010-11-15 15:00 memory.usage_in_bytes
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 memory.use_hierarchy
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 net_cls.classid
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 notify_on_release
>> -rw-r--r-- 1 root root 0 2010-11-15 15:00 tasks
>>
>> It is contentof my cgroup directory with running LXC. There are
>> directories:
>>
>> drwxr-xr-x 4 root root 0 2010-11-15 15:00 1285
>> drwxr-xr-x 4 root root 0 2010-11-15 15:00 1298
>> drwxr-xr-x 4 root root 0 2010-11-15 15:01 1322
>> drwxr-xr-x 4 root root 0 2010-11-15 15:01 1325
>> drwxr-xr-x 4 root root 0 2010-11-15 15:02 1335
>> drwxr-xr-x 4 root root 0 2010-11-15 15:09 1386
>> drwxr-xr-x 4 root root 0 2010-11-15 15:11 1401
>> drwxr-xr-x 4 root root 0 2010-11-15 15:12 1408
>> drwxr-xr-x 4 root root 0 2010-11-15 15:12 1411
>> drwxr-xr-x 3 root root 0 2010-11-15 15:17 1459
>>
>> These PIDs are no in global proc filesystem
>> and still increasing.
>>
>> Load of my machine is still higher and higher.
>>
>> I will must restart computer ant the end. :-(
>>
>> What can i do with it?
>
> Ok, let's try to understand.
>
> Let's do it step-by-step :
>
> 1 - the topmost directory shows "freezer.state" and it shouldn't because
> you can not freeze the system (except if there is a recent change in the
> kernel)
>
> 2 - try to delete the 1285/2 directory, if you can't check the content
> of 1285/2/tasks and look for the process in the system: is it a process
> running inside a container ?

1285/2/tasks is empty

rm -r 1285/2
rm: cannot remove `1285/2/cpuset.memory_spread_slab': Operation not 
permitted
rm: cannot remove `1285/2/cpuset.memory_spread_page': Operation not 
permitted
rm: cannot remove `1285/2/cpuset.memory_pressure': Operation not permitted
rm: cannot remove `1285/2/cpuset.memory_migrate': Operation not permitted
rm: cannot remove `1285/2/cpuset.sched_relax_domain_level': Operation 
not permitted
rm: cannot remove `1285/2/cpuset.sched_load_balance': Operation not 
permitted
rm: cannot remove `1285/2/cpuset.mem_hardwall': Operation not permitted
rm: cannot remove `1285/2/cpuset.mem_exclusive': Operation not permitted
rm: cannot remove `1285/2/cpuset.cpu_exclusive': Operation not permitted
rm: cannot remove `1285/2/cpuset.mems': Operation not permitted
rm: cannot remove `1285/2/cpuset.cpus': Operation not permitted
rm: cannot remove `1285/2/cpu.rt_period_us': Operation not permitted
rm: cannot remove `1285/2/cpu.rt_runtime_us': Operation not permitted
rm: cannot remove `1285/2/cpu.shares': Operation not permitted
rm: cannot remove `1285/2/cpuacct.stat': Operation not permitted
rm: cannot remove `1285/2/cpuacct.usage_percpu': Operation not permitted
rm: cannot remove `1285/2/cpuacct.usage': Operation not permitted
rm: cannot remove `1285/2/memory.memsw.failcnt': Operation not permitted
rm: cannot remove `1285/2/memory.memsw.limit_in_bytes': Operation not 
permitted
rm: cannot remove `1285/2/memory.memsw.max_usage_in_bytes': Operation 
not permitted
rm: cannot remove `1285/2/memory.memsw.usage_in_bytes': Operation not 
permitted
rm: cannot remove `1285/2/memory.swappiness': Operation not permitted
rm: cannot remove `1285/2/memory.use_hierarchy': Operation not permitted
rm: cannot remove `1285/2/memory.force_empty': Operation not permitted
rm: cannot remove `1285/2/memory.stat': Operation not permitted
rm: cannot remove `1285/2/memory.failcnt': Operation not permitted
rm: cannot remove `1285/2/memory.soft_limit_in_bytes': Operation not 
permitted
rm: cannot remove `1285/2/memory.limit_in_bytes': Operation not permitted
rm: cannot remove `1285/2/memory.max_usage_in_bytes': Operation not 
permitted
rm: cannot remove `1285/2/memory.usage_in_bytes': Operation not permitted
rm: cannot remove `1285/2/devices.list': Operation not permitted
rm: cannot remove `1285/2/devices.deny': Operation not permitted
rm: cannot remove `1285/2/devices.allow': Operation not permitted
rm: cannot remove `1285/2/freezer.state': Operation not permitted
rm: cannot remove `1285/2/net_cls.classid': Operation not permitted
rm: cannot remove `1285/2/notify_on_release': Operation not permitted
rm: cannot remove `1285/2/cgroup.procs': Operation not permitted
rm: cannot remove `1285/2/tasks': Operation not permitted

> Oh, a dumb question : are you using the libvirt with lxc ?

No, i using lxc package (lxc-start, lxc-stop, lxc-console, ...)

I see, that vsftd running in container generate it sometimes for example.

Miroslav.

-- 
Miroslav Lednicky, AVONET, s.r.o.