[Lxc-users] lxcbr0 MAC addr issue

Wed Jun 5 17:50:23 UTC 2013

On Wed, 2013-06-05 at 11:26 -0500, Serge Hallyn wrote: 
> Quoting Michael H. Warfield (mhw at WittsEnd.com):
> > Crap...  Bumped the keyboard and this one got away from me prematurely.
> > 
> > On Wed, 2013-06-05 at 11:23 -0400, Michael H. Warfield wrote: 
> > > On Wed, 2013-06-05 at 15:17 +0000, Jäkel, Guido wrote: 
> > > > >yes and it does this.  The point is that lxcbr0 is not tied to any
> > > > >physical nic.  So the first container you start, however high the
> > > > >macaddr is, lxcbr0 takes its mac.  If the next container gets a
> > > > >lower macaddr, lxcbr0's macaddr drops.
> > 
> > > > This lxcbr0 is special to Ubuntu, right? And if not to a physical
> > > NIC, to what is this bridge connected to on the host?
> > 
> > > Not to the best of my knowledge.  It should be a simple bridge.
> > 
> > > What do you get for this command?
> > 
> > > brctl show
> > 
> > > A bridge doesn 
> > 
> > A bridge doesn't have to be attach to a device.  A bridge is its own
> > logical entity in the kernel to which you may attach devices.  You can
> > not "attach a bridge" to something else.  You can only attach something
> > else "to the bridge".  There's a difference.
> > 
> > In the case of a NATing configuration, you set up a bridge (name it
> > whatever you want) and attach the containers to it.  Then you use the
> > NAT modules to route between the bridge and the external interface while
> > NATing the addresses.  I use "lxcbr0" on my Fedora hosts.  It's just a
> > bridge.
> > 
> > I could see where Ubuntu might have some preconfigured setups for this
> > purpose where I have to set them up by hand in Fedora.  That's just a
> > matter of the distro specific support scripts.
> 
> Right.  And we *could* attach a dummy device with mac starting with
> something lower.

> BUT I just did some testing, and even as I watch lxcbr0's addr go down
> when starting a new container, my ssh to the container which had the
> higher macaddr doesn't hiccough.

Hmmm...  It would be interesting to see what you get from "arp -a" on
the host and the container before and after that.  It would also be
interesting to see what happens if you ssh to a container with the
higher address first and then bring up a container with a lower mac
address and see if it impacts the existing connection.

It really all depends on how the host is managing the arp table and, if
the MAC address changes, how does the bridge change impact the arp
table.

In the case of ssh'ing to a container from the host, the host would
still have the correct arp entry for the container which would
facilitate the delivery of the initial SYN.  The container would have an
incorrect entry but that should correct itself as soon as that new
packet arrives connecting from the host, invalidating the containers arp
table entry, I would think.

It should also correct the MAC table entries for the bridge (used by the
STP) if it originates from the interface that changed.  It works the
same way over an eitherswitch externally.  The moment a packet is sent
with a new "from" MAC address, the switch (bridge) remembers what port
(attachment) that MAC address was last seen on and enters it into it's
mac switch table.

What would be interesting to know is how the container sees it, since
his arp table entry for the host is then wrong if he initiates a packet
first, after the change.  It could be that the tap/bridge connection
from container to host bridge would clear that up quickly.

> Perhaps it'll be a problem when connected from an outside host (through
> port forwarding).  In that case I'll happily do the dummy if hack.  But
> it seems possible that it just isn't needed.  (And since the iptables rule
> is --to-destination an ip address, I'm thinking it won't be)

Yeah, since the host exposed IP and MAC are not changing, it shouldn't
be, in the external case with NAT as you describe.  The external systems
will only be referencing the external IP address for which the MAC
doesn't change and the ARP tables are perfectly coherent, externally.
So NAT w/ port forwarding to a container should be covered.

It's where the MAC address of the host interface changes (because it's
attached to a bridge) where you get into trouble in the external case.
But, we have that case covered now with the fixes that are in-place
since 0.8.0 (assuming the OP is on a recent enough version of the lxc
tools).

So both of those should be covered.

That really just leaves the host <-> container case (container <->
container should be a trivial non-issue).  Then I think it's an issue of
which side of the bridge is issuing the first packet after the MAC
address changes and what does the bridge logic do about it.  Seems like
a corner case to me.  I also don't know what role STP has in this game.
On my fedora system, my lxcbr0 bridge has STP disabled but the virbr0
bridge (used by libvirt and setup by default) has it enabled and I have
no idea why.  It's possible that these symptoms could vary depending on
that setting.  I've definitely heard it mentioned before.

Oh, and this is likely to get really ugly with IPv6.  With the host as
the putative IPv6 router for the containers, if that MAC address
changes, that's going to directly impact the RA (Router Advertisement)
and RD (Router Discovery) protocols with all their TTL's and what not.
Still doesn't impact the straight-across bridging like what I do (since
the router for the containers is the router for the host and the host
doesn't have to run radvd or quagga to advertise routes).  There's also
the hybrid case where you bridge IPv6 while you route and NAT IPv4 (man
ebtables - look for BROUTE chain).  I'm going to have to experiment with
that one a bit.  Only case that really worries me would be the case were
the router (host) link local address changes with the MAC address.

I don't think the OP is doing anything sophisticated on that level.  :-P

> -serge

> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. A cloud service to automate IT design, transition and operations
> 2. Dashboards that offer high-level views of enterprise services
> 3. A single system of record for all IT processes
> http://p.sf.net/sfu/servicenow-d2d-j
> _______________________________________________

Regards,
Mike
-- 
Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20130605/2da2af3c/attachment.pgp>