[Lxc-users] Issues with mass-starting and bridge

[Leon Waldman]

Hi,


Just to send a feedback... Looks like the problems were related to the 
virtualbox network layer.

I did the same setup on a Xen VM and till now, everything went fine 
(except for the fact that the VMs are rebooting after 20 minutes, but I 
think this is duo to low setting on the gc_thresh1 (it's on 128) and I'm 
starting to troubleshoot it now! :P).

Any way, thank very much for the insights! :)


On 10/08/2012 04:02 PM, Leon Waldman wrote:
> Hi all,
>
>
> I'm having a strange issue with the bridge on my lxc host.
>
>
> First, some background info:
> I'm running lxc version 0.75 on a CentOS 6.3 install with the standard 
> kernel (2.6.32-279.9.1.el6.i686).
> it's running inside a VirtualBox VM.
>
> I created 100 containers and named from 000 to 099. they are all with 
> sequential assigned IPs and macaddresses (from 10.0.1.0/8 till 
> 10.0.1.99/8 | 80:04:C0:AB:00:00 till 80:04:C0:AB:00:63).
>
>
> When starting them in "mass" with something like:
> for i in `range 0 99`; do echo $i; lxc-start -n $i -d; done
> after a random number of containers is started the br0 bridge just 
> stops to answer for anything. It just freezes (it's easy to note 
> because I'm connecting to the host through ssh using the same bridge).
>
> If I put a sleep 5 on the for above, things goes better, but then one 
> or two of the containers became a "remote bridge bomb". The bridge 
> stays working till a packet arrives to the "bomb" container, and them 
> again the entire bridge freezes. This container bomb is also random.
>
> if I remove the problematic container iface from the bridge, it 
> unfreezes. If I add it again and send a packet... It freezes again. If 
> I stop the container and re-start it... all goes fine and the issue 
> vanish.
>
>
> From my point of view, looks like when the virtual iface is created 
> during the container start (if under load or with too much concurrent 
> starts... I don't know), sometimes something goes wrong with the iface 
> and it mess with the entire bridge if this "messy" iface is attached.
>
>
> Do any one have any input on this (do this happened before? Is there a 
> limit for concurrent containers initializations? Is there a kernel 
> setting or some config that I could do to solve it)?
>
>
> In order to achieve what I need in production environment, I will need 
> to start and stop containers several times per minute. Do this sound 
> too problematic?
>
>
> If you guys need any additional information, please just let me know.
>
>
> Thanks in advance
>
> --
> *Leon Waldman* <http://leonwaldman.com/>
> Senior Linux/UNIX SysAdmin & Consultant.
> Back-End & Infrastructure Architect.
> View Leon Waldman's profile on LinkedIn 
> <http://br.linkedin.com/in/leonwaldman>
> <https://twitter.com/lewaldman>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121016/251c9d7d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 382 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121016/251c9d7d/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 2775 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121016/251c9d7d/attachment-0001.png>