[lxc-devel] [ lxc-Bugs-3411497 ] random veth device MAC addresses cause bridge problems

[SourceForge.net]

Bugs item #3411497, was opened at 2011-09-19 10:18
Message generated for change (Comment added) made by mhwarfield
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=826303&aid=3411497&group_id=163076

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Gary Richards ()
Assigned to: Nobody/Anonymous (nobody)
Summary: random veth device MAC addresses cause bridge problems

Initial Comment:
Hi,

I've recently spent some time debugging an LXC test setup. The problem that we were having is that sometimes (not every time) that we started or stopped a container, the host networking would 'freeze' (for want of a better term) for anywhere from 10-30 seconds.

To cut a long story short, we eventually realised that the following was happening:
- LXC creates a veth device for our container on a random MAC address
- LXC adds the veth device into the bridge that we've specified in our configuration
- If the MAC address of the veth device in the host (not the MAC address of the virtual device in the container) is 'lower' than the MAC address of the bridge, the bridge would change to this new 'lower' MAC address. Obviously until everything else on the network made new arp requests, it would make it appear as if the networking for the host system had 'frozon'.
- The same thing happens when shutting down a container that had a 'lower' MAC address than the bridge original had setup. On removal of the veth device from the bridge, the bridge then realises that it no longer has a device with its MAC address and reconfigures itself with the next 'lowest' MAC address.

I looked into why i've never seen this problem with KVM virtual machines (when using similar techniques for their networking). I'm unsure if it's KVM/qemu itself that does this or something in libvirt, but a VM configured with a MAC address of 52:54:aa.bb.cc.dd would get its associated veth device created as fe:54:aa:bb:cc:dd.

I assume that they default to using fe:....... because it creates the device with a very 'high' numbered MAC address and therefore mostly guards against the problem that I describe above due to all VM veth interfaces coming up with a number that's pretty much guaranteed to be higher than any real device that you might have associated with a bridge.

For the time being, we worked around this issue by forcing the MAC address of the bridge to the MAC of the attached real network interface.

Gary

----------------------------------------------------------------------

Comment By: Michael H. Warfield (mhwarfield)
Date: 2011-09-19 21:07

Message:
This is documented and by design in the Linux kernel and applies to all
bridges.  To avoid it, you have to assign mac addresses higher than that of
the host address.  Using purely "random" mac addresses has always been a
crap shoot.  VMware, Virtual Box, and OpenVZ get around this by using a
"Vendor ID" field (most of the first 3 bytes) that's higher than most
assigned mac vendor codes.  I always pick a high vendor code and then
assign the lower 3 bytes to a random number (and make sure the 02 bit in
the first byte is set and the 01 byte is clear to conform to the mac
address standard of setting the locally administered bit and clearing the
multicast bit).  If you want LXC to assign "random" mac addresses, they
will have to be restricted mac addresses and can not be truly 48 bits of
random data.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=826303&aid=3411497&group_id=163076