<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Hello,
<div><br>
</div>
<div>I am running LXC 0.8.0 kernel 3.7.9 and try to start more than 1000 small containers : around 10MB of RAM per containers.</div>
<div><br>
</div>
<div>Starting around the first 1600 happens smoothy - I have a 32 virtual core machine - but then everything gets very slow :</div>
<div><br>
</div>
<div>up to a minute per contain creation. Ultimately the server CPU goes to 100%.</div>
<div><br>
</div>
<div>I get this error multiple time in the syslog :</div>
<div><br>
</div>
<div><!--?xml version="1.0" encoding="UTF-8" standalone="no"?-->
<div><span class="Apple-style-span" style="font-family: Arial; "><br class="Apple-interchange-newline">
</span><span class="Apple-style-span" style="font-family: Arial; ">[ 2402.961711] INFO: task lxc-start:128486 blocked for more than 120 seconds.</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961717] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961724] lxc-start D ffffffff8180cc60 0 128486 1 0x00000000</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961727] ffff883c30359cb0 0000000000000086 ffff883c2ea3c800 ffff883c2f550600</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961734] ffff883c2d955c00 ffff883c30359fd8 ffff883c30359fd8 ffff883c30359fd8</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961741] ffff881fd35e5c00 ffff883c2d955c00 ffff883c3533ec10 ffffffff81cac4e0</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961747] Call Trace:</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961753] [<ffffffff816dbfc9>] schedule+0x29/0x70</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961758] [<ffffffff816dc27e>] schedule_preempt_disabled+0xe/0x10</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961763] [<ffffffff816dadd7>] __mutex_lock_slowpath+0xd7/0x150</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961768] [<ffffffff8158b911>] ? net_alloc_generic+0x21/0x30</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961772] [<ffffffff816da9ea>] mutex_lock+0x2a/0x50</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961777] [<ffffffff8158c044>] copy_net_ns+0x84/0x110</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961782] [<ffffffff81081f4b>] create_new_namespaces+0xdb/0x180</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961787] [<ffffffff8108210c>] copy_namespaces+0x8c/0xd0</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961792] [<ffffffff81055ea0>] copy_process+0x970/0x1550</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961796] [<ffffffff8119e542>] ? do_filp_open+0x42/0xa0</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961801] [<ffffffff81056bc9>] do_fork+0xf9/0x340</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961806] [<ffffffff81199de6>] ? final_putname+0x26/0x50</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961811] [<ffffffff81199ff9>] ? putname+0x29/0x40</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961816] [<ffffffff8101d498>] sys_clone+0x28/0x30</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961819] [<ffffffff816e5c23>] stub_clone+0x13/0x20</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">[
2402.961823] [<ffffffff816e5919>] ? system_call_fastpath+0x16/0x1b</span></div>
</div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;"><br>
</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">regards</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;"><br>
</span></div>
<div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium;">Benoit</span></div>
<div><br>
</div>
<div><br>
</div>
<div>Mu container config</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>lxc.utsname = lwb2001</div>
<div>lxc.rootfs = /var/lib/lxc/lwb2001/rootfs</div>
<div>lxc.network.type = phys</div>
<div>lxc.network.flags = up</div>
<div>lxc.network.link = eth6.3 </div>
<div>lxc.network.name = eth2</div>
<div>lxc.network.hwaddr = 00:50:56:a8:03:03</div>
<div>lxc.network.ipv6 = 2007:4339:0:3::2/64</div>
<div>lxc.network.ipv4 = 192.168.1.1/24</div>
<div>lxc.network.type = phys</div>
<div>lxc.network.flags = up</div>
<div>lxc.network.link = eth7.3</div>
<div>lxc.network.name = eth1</div>
<div>lxc.network.ipv4 = 2.2.2.2/24</div>
<div>lxc.network.ipv6 = 2007:4339:0:12::2/64</div>
<div>lxc.network.hwaddr = 00:50:57:b8:00:01</div>
<div>lxc.devttydir = lxc</div>
<div>lxc.tty = 1</div>
<div>lxc.pts = 1024</div>
<div>lxc.arch = amd64</div>
<div>lxc.cap.drop = sys_module mac_admin mac_override</div>
<div>lxc.pivotdir = lxc_putold</div>
<div>lxc.mount.entry = proc proc proc nodev,noexec,nosuid 0 0</div>
<div>lxc.mount.entry = sysfs sys sysfs defaults 0 0</div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/usr usr none ro,bind 0 0 </div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/lib lib none ro,bind 0 0 </div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/sbin sbin none ro,bind 0 0 </div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/etc etc none ro,bind 0 0 </div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/bin bin none ro,bind 0 0</div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/backups var/backups none ro,bind 0 0</div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/cache var/cache none ro,bind 0 0</div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/lib/apt var/lib/apt none ro,bind 0 0</div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/lib/dpkg var/lib/dpkg none ro,bind 0 0</div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/local var/local none ro,bind 0 0</div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/mail var/mail none ro,bind 0 0</div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/opt var/opt none ro,bind 0 0</div>
<div>lxc.mount.entry = /var/lib/lxc/lwb2000/rootfs/var/spool var/spool none ro,bind 0 0</div>
<div>lxc.cgroup.cpuset.cpus = 0</div>
<div>lxc.cgroup.devices.deny = a</div>
<div>lxc.cgroup.devices.allow = c *:* m</div>
<div>lxc.cgroup.devices.allow = b *:* m</div>
<div>lxc.cgroup.devices.allow = c 1:3 rwm</div>
<div>lxc.cgroup.devices.allow = c 1:5 rwm</div>
<div>lxc.cgroup.devices.allow = c 5:1 rwm</div>
<div>lxc.cgroup.devices.allow = c 5:0 rwm</div>
<div>lxc.cgroup.devices.allow = c 1:9 rwm</div>
<div>lxc.cgroup.devices.allow = c 1:8 rwm</div>
<div>lxc.cgroup.devices.allow = c 136:* rwm</div>
<div>lxc.cgroup.devices.allow = c 5:2 rwm</div>
<div>lxc.cgroup.devices.allow = c 254:0 rwm</div>
<div>lxc.cgroup.devices.allow = c 10:229 rwm</div>
<div>lxc.cgroup.devices.allow = c 10:200 rwm</div>
<div>lxc.cgroup.devices.allow = c 1:7 rwm</div>
<div>lxc.cgroup.devices.allow = c 10:228 rwm</div>
<div>lxc.cgroup.devices.allow = c 10:232 rwm</div>
</div>
</body>
</html>