[Lxc-users] Containers slow to start after 1600

[Benoit Lourdelet]

Hello,

I am running LXC 0.8.0 kernel 3.7.9 and try to start more than 1000 small containers : around 10MB of RAM per containers.

Starting around the first 1600 happens smoothy - I have a 32 virtual core machine - but then everything gets very slow :

up to a minute per contain creation.  Ultimately the server CPU goes to 100%.

I get this error  multiple time in  the syslog :


[ 2402.961711] INFO: task lxc-start:128486 blocked for more than 120 seconds.
[ 2402.961717] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2402.961724] lxc-start D ffffffff8180cc60 0 128486 1 0x00000000
[ 2402.961727] ffff883c30359cb0 0000000000000086 ffff883c2ea3c800 ffff883c2f550600
[ 2402.961734] ffff883c2d955c00 ffff883c30359fd8 ffff883c30359fd8 ffff883c30359fd8
[ 2402.961741] ffff881fd35e5c00 ffff883c2d955c00 ffff883c3533ec10 ffffffff81cac4e0
[ 2402.961747] Call Trace:
[ 2402.961753] [<ffffffff816dbfc9>] schedule+0x29/0x70
[ 2402.961758] [<ffffffff816dc27e>] schedule_preempt_disabled+0xe/0x10
[ 2402.961763] [<ffffffff816dadd7>] __mutex_lock_slowpath+0xd7/0x150
[ 2402.961768] [<ffffffff8158b911>] ? net_alloc_generic+0x21/0x30
[ 2402.961772] [<ffffffff816da9ea>] mutex_lock+0x2a/0x50
[ 2402.961777] [<ffffffff8158c044>] copy_net_ns+0x84/0x110
[ 2402.961782] [<ffffffff81081f4b>] create_new_namespaces+0xdb/0x180
[ 2402.961787] [<ffffffff8108210c>] copy_namespaces+0x8c/0xd0
[ 2402.961792] [<ffffffff81055ea0>] copy_process+0x970/0x1550
[ 2402.961796] [<ffffffff8119e542>] ? do_filp_open+0x42/0xa0
[ 2402.961801] [<ffffffff81056bc9>] do_fork+0xf9/0x340
[ 2402.961806] [<ffffffff81199de6>] ? final_putname+0x26/0x50
[ 2402.961811] [<ffffffff81199ff9>] ? putname+0x29/0x40
[ 2402.961816] [<ffffffff8101d498>] sys_clone+0x28/0x30
[ 2402.961819] [<ffffffff816e5c23>] stub_clone+0x13/0x20
[ 2402.961823] [<ffffffff816e5919>] ? system_call_fastpath+0x16/0x1b

regards

Benoit


Mu container config


lxc.utsname = lwb2001
lxc.rootfs = /var/lib/lxc/lwb2001/rootfs
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth6.3
lxc.network.name = eth2
lxc.network.hwaddr = 00:50:56:a8:03:03
lxc.network.ipv6 = 2007:4339:0:3::2/64
lxc.network.ipv4 = 192.168.1.1/24
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth7.3
lxc.network.name = eth1
lxc.network.ipv4 = 2.2.2.2/24
lxc.network.ipv6 = 2007:4339:0:12::2/64
lxc.network.hwaddr = 00:50:57:b8:00:01
lxc.devttydir = lxc
lxc.tty = 1
lxc.pts = 1024
lxc.arch = amd64
lxc.cap.drop = sys_module mac_admin mac_override
lxc.pivotdir = lxc_putold
lxc.mount.entry = proc     proc         proc    nodev,noexec,nosuid 0 0
lxc.mount.entry = sysfs           sys          sysfs defaults  0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/usr usr none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/lib lib none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/sbin sbin none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/etc etc none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/bin bin none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/backups var/backups none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/cache var/cache none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/lib/apt var/lib/apt none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/lib/dpkg var/lib/dpkg none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/local var/local none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/mail var/mail none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/opt var/opt none ro,bind 0 0
lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/spool var/spool none ro,bind 0 0
lxc.cgroup.cpuset.cpus = 0
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = b *:* m
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 254:0 rwm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 10:228 rwm
lxc.cgroup.devices.allow = c 10:232 rwm
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20130311/ae35be96/attachment.html>