[Lxc-users] seeing a network pause when starting and stopping LXCs - how do I stop this ?

Tue Dec 13 17:57:18 UTC 2011

I always hate replying to my own posts but I have stumbled onto some
interesting clarification as I've continued to play with this...

Below in-line.

On Tue, 2011-12-13 at 01:30 -0500, Michael H. Warfield wrote: 
> On Mon, 2011-12-12 at 08:43 +0100, Ulli Horlacher wrote: 
> > On Sun 2011-12-11 (19:48), Derek Simkowiak wrote:
> > 
> > >      The problem is not related to the setfd option.  It is caused by 
> > > the bridge acquired a new MAC address.  Libvirt already has a fix for 
> > > this, and there is a patch in the works for the LXC tools.
> 
> > I am wonder why I do not have this problem: I really often start new
> > containers and I do not have this patch, but no network freeze at all.
> 
> Look to your MAC addresses.  If the MAC address of the new container is
> greater than the MAC address of the bridge, you will never see this
> problem.  If it's less than the bridge MAC address, the MAC address will
> change and, if you have any active network traffic, you WILL see this
> problem.
> 
> The problem is old and related to a design decision wrt the bridges in
> the Linux kernel.  I don't really agree with where the decision went but
> I understand the problem and their concerns (IIRC).  The problem is the
> bridge NEEDS a MAC address but has no "intrinsic" address.  So, they
> started out by assigning the MAC address of the first device assigned to
> the bridge.  That worked but had a problem.  The bridge is an
> independent autonomous object within the kernel and must be independent
> of the connected devices.  Sooo, what happens to the MAC address if you
> delete that device from the bridge?  It can't stay the same.  Now, my
> memory is very fuzzy after about a decade (and Google searches are NOT
> helping my research) and anyone is welcome to correct me if I recall
> incorrect but...  The arbitrary decision was made to always use the
> lowest MAC address on the bridge.  The trouble is that they did that
> when they added devices.  The problem really only occurs when you delete
> devices and, even then, only when the MAC of the bridge matches the MAC
> of the device being deleted and there is no other device with that MAC
> address (multihomed SPARCs) on that bridge.  I would have chosen
> differently but this was not my shot to call and I didn't have any
> strong arguments to even insert into the discussion.  They chose to
> always use the lowest MAC address.  The choice was arbitrary but a
> choice had to be made.  No choice was a non-starter and all choices had
> some downside to them.

> There are actually TWO solutions to this problem.

> The first one people have already glommed onto.  You have to set your
> MAC address of your containers to be greater than the MAC address of the
> bridge.  Problems there are that 1) we don't have our own vendor code
> and 2) we have to make sure we're higher than ANY vendor code and 3) the
> "locally administered bit" is NOT the highest order bit in the MAC
> address (that really would have made it toooooo easy to fix but that's
> and even more ancient bad choice).  The good point is that we CAN auto
> assign anything we WANT as long as that locally administered bit is set
> and we can then set the vendor code as high as we want.  THAT works.
> Use FF:FF:F2:: and you are gold.  Play with the remaining bits all you
> want.

> There is another.  When creating the bridge, assign it a MAC address
> that's very LOW.  You can do that with ifconfig or ip.  It doesn't HAVE
> to have a MAC address that matches ANY interface attached to it.  That's
> a misconception.  It merely has to HAVE a MAC address.  The problem with
> this solution is that "technically" that MAC address is then "locally
> administered" so the locally administered bit SHOULD be set (00:02::)
> but then anything of the vendor codes from 00:00:00:: through 00:00:ff::
> would be less than that (00:01:: is the multicast bit).  Grrr...  But
> it's only 256 old vendors.  Using 00:00:00:: for the vendor code works
> and is sooo unlikely to conflict with Xerox Corporation (vendor code
> 00:00:00) or their Ethernet cards (I know I'll get HATE MAIL from
> purists on THAT comment) that it's really NOT worth worrying about.

According to this post, I may be mistaken that it doesn't have to be an
address of a connected interface.

http://backreference.org/2010/07/28/linux-bridge-mac-addresses-and-dynamic-ports/

In my testing I've certainly found that it does NOT have to be a MAC of
a connected interface for IPv6 to work.  My results for IPv4 have been
mixed where, in one case, it did not and, in another case, it did.  IPv6
worked in both cases but there are some peculiar filtering paths down in
the kernel bridging logic that could well be coming into play here.  It
may also have to do with the MAC address detection logic and spanning
tree algorithms.  So be cautious there.

But there was this remark:

-- 
Fortunately, there is a way to ensure that the bridge's MAC address is
fixed and never changes, thus entirely avoiding the problem. Thanks to
this thread
<http://www.spinics.net/lists/linux-ethernet-bridging/msg03615.html>
and this thread
<http://www.mail-archive.com/bridge@lists.linux-foundation.org/msg01084.html>
by people with similar problems, I found out that if the bridge's MAC
address is forced to a specific value, the bridge "remembers" that and
makes the address permanent.
-- 

The "sticky bridge address" patch appears to have been implemented in
2.6.27.

That threat also confirms that using a MAC address that is not a MAC
address of a connected interface causes the bridge to stop functioning
(but only for IPv4 from what I have determined).
http://www.mail-archive.com/bridge@lists.linux-foundation.org/msg01094.html

There's also a good explanation about WHY it works that way further down
in the thread (which does NOT explain why it still works for IPv6) which
makes me think it's related to the spanning tree algorithm behavior when
you have multiple bridges, but that's another story.

So, it sounds like this is an even easier solution.  On 2.6.27 and above
just use the ip command to assign the MAC address of the permanent
ethernet interface to the bridge and it becomes "sticky".

> You do have to be forewarned that tinkering with MAC addresses at
> runtime can be even MORE disasterous when dealing with IPv6 and autoconf
> addresses.  Assigning a fixed MAC address to a bridge should only be
> done at boot time and not changed after (one of the arguments against
> that earlier decision).  Which points out another PITA associated with
> that early decision regarding MAC addresses.  It makes a MESS out of
> IPv6!

> Regards,
> Mike

Regards,
Mike
-- 
Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20111213/a1838224/attachment.pgp>