[Lxc-users] seeing a network pause when starting and stopping LXCs - how do I stop this ?

[Michael H. Warfield]

On Mon, 2011-12-12 at 08:43 +0100, Ulli Horlacher wrote: 
> On Sun 2011-12-11 (19:48), Derek Simkowiak wrote:
> 
> >      The problem is not related to the setfd option.  It is caused by 
> > the bridge acquired a new MAC address.  Libvirt already has a fix for 
> > this, and there is a patch in the works for the LXC tools.

> I am wonder why I do not have this problem: I really often start new
> containers and I do not have this patch, but no network freeze at all.

Look to your MAC addresses.  If the MAC address of the new container is
greater than the MAC address of the bridge, you will never see this
problem.  If it's less than the bridge MAC address, the MAC address will
change and, if you have any active network traffic, you WILL see this
problem.

The problem is old and related to a design decision wrt the bridges in
the Linux kernel.  I don't really agree with where the decision went but
I understand the problem and their concerns (IIRC).  The problem is the
bridge NEEDS a MAC address but has no "intrinsic" address.  So, they
started out by assigning the MAC address of the first device assigned to
the bridge.  That worked but had a problem.  The bridge is an
independent autonomous object within the kernel and must be independent
of the connected devices.  Sooo, what happens to the MAC address if you
delete that device from the bridge?  It can't stay the same.  Now, my
memory is very fuzzy after about a decade (and Google searches are NOT
helping my research) and anyone is welcome to correct me if I recall
incorrect but...  The arbitrary decision was made to always use the
lowest MAC address on the bridge.  The trouble is that they did that
when they added devices.  The problem really only occurs when you delete
devices and, even then, only when the MAC of the bridge matches the MAC
of the device being deleted and there is no other device with that MAC
address (multihomed SPARCs) on that bridge.  I would have chosen
differently but this was not my shot to call and I didn't have any
strong arguments to even insert into the discussion.  They chose to
always use the lowest MAC address.  The choice was arbitrary but a
choice had to be made.  No choice was a non-starter and all choices had
some downside to them.

There are actually TWO solutions to this problem.

The first one people have already glommed onto.  You have to set your
MAC address of your containers to be greater than the MAC address of the
bridge.  Problems there are that 1) we don't have our own vendor code
and 2) we have to make sure we're higher than ANY vendor code and 3) the
"locally administered bit" is NOT the highest order bit in the MAC
address (that really would have made it toooooo easy to fix but that's
and even more ancient bad choice).  The good point is that we CAN auto
assign anything we WANT as long as that locally administered bit is set
and we can then set the vendor code as high as we want.  THAT works.
Use FF:FF:F2:: and you are gold.  Play with the remaining bits all you
want.

There is another.  When creating the bridge, assign it a MAC address
that's very LOW.  You can do that with ifconfig or ip.  It doesn't HAVE
to have a MAC address that matches ANY interface attached to it.  That's
a misconception.  It merely has to HAVE a MAC address.  The problem with
this solution is that "technically" that MAC address is then "locally
administered" so the locally administered bit SHOULD be set (00:02::)
but then anything of the vendor codes from 00:00:00:: through 00:00:ff::
would be less than that (00:01:: is the multicast bit).  Grrr...  But
it's only 256 old vendors.  Using 00:00:00:: for the vendor code works
and is sooo unlikely to conflict with Xerox Corporation (vendor code
00:00:00) or their Ethernet cards (I know I'll get HATE MAIL from
purists on THAT comment) that it's really NOT worth worrying about.

You do have to be forewarned that tinkering with MAC addresses at
runtime can be even MORE disasterous when dealing with IPv6 and autoconf
addresses.  Assigning a fixed MAC address to a bridge should only be
done at boot time and not changed after (one of the arguments against
that earlier decision).  Which points out another PITA associated with
that early decision regarding MAC addresses.  It makes a MESS out of
IPv6!

Regards,
Mike
-- 
Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20111213/b3bfee8a/attachment.pgp>