[lxc-users] Containers have network issues when their host uses a bonded interface

[Peter Steele]

I've configured a standard CentOS bridge/bond, the exact same setup that 
I use for creating VMs. VMs on different hosts communicate through the 
bridge without issues. Containers that use the identical bridge however 
cannot reliably connect to containers on different hosts. We've 
determined that it is some kind of ARP table issue, or at least appears 
to be. I have reproduced a scenario where container A on host X is 
unable to ping container B on host Y. If I connect to B from host Y and 
ping say the gateway of our subnet, suddenly container A on host X can 
ping container B. Here's a specific case. Container A on host X has the 
IP 172.16.110.106. Its arp table looks like this:

# arp -v -n
Address                  HWtype  HWaddress           Flags 
Mask            Iface
172.16.110.112           ether   00:16:3e:21:c6:c5 C                     
eth0
172.16.110.111           ether   00:16:3e:31:2a:f0 C                     
eth0
172.16.110.110           ether   00:16:3e:5c:62:a9 C                     
eth0
Entries: 4      Skipped: 0      Found: 4

Container B on host Y has the IP 172.16.110.113. Its arp table looks 
like this:

# arp -v -n
Address                  HWtype  HWaddress           Flags 
Mask            Iface
172.16.110.112           ether   00:16:3e:21:c6:c5 C                     
eth0
172.16.110.106           ether   00:16:3e:20:df:87 C                     
eth0
Entries: 2      Skipped: 0      Found: 2

If I try to ping 172.16.110.113 (container B) from 172.16.110.106 
(container A), I get a "Host Unreachable":

# ping 172.16.110.113
PING 172.16.110.113 (172.16.110.113) 56(84) bytes of data.
 From 172.16.110.106 icmp_seq=1 Destination Host Unreachable
 From 172.16.110.106 icmp_seq=2 Destination Host Unreachable
 From 172.16.110.106 icmp_seq=3 Destination Host Unreachable
 From 172.16.110.106 icmp_seq=4 Destination Host Unreachable
 From 172.16.110.106 icmp_seq=5 Destination Host Unreachable
...

If while this ping is running I connect to container B and ping the 
gateway (172.16.0.1), the ping running on container A suddenly starts 
working:

...
 From 172.16.110.106 icmp_seq=6 Destination Host Unreachable
 From 172.16.110.106 icmp_seq=7 Destination Host Unreachable
 From 172.16.110.106 icmp_seq=8 Destination Host Unreachable
64 bytes from 172.16.110.113: icmp_seq=57 ttl=64 time=993 ms
64 bytes from 172.16.110.113: icmp_seq=58 ttl=64 time=0.283 ms
64 bytes from 172.16.110.113: icmp_seq=59 ttl=64 time=0.274 ms
...

The arp table on container A now has an entry for container B:

# arp -v -n
Address                  HWtype  HWaddress           Flags 
Mask            Iface
172.16.110.112           ether   00:16:3e:21:c6:c5 C                     
eth0
172.16.110.111           ether   00:16:3e:31:2a:f0 C                     
eth0
172.16.110.113           ether   00:16:3e:65:2a:c5 C                     
eth0
172.16.110.110           ether   00:16:3e:5c:62:a9 C                     
eth0
Entries: 5      Skipped: 0      Found: 5

The arp table on container B of course now has an entry for the gateway:

# arp -v -n
Address                  HWtype  HWaddress           Flags 
Mask            Iface
172.16.110.112           ether   00:16:3e:21:c6:c5 C                     
eth0
172.16.110.106           ether   00:16:3e:20:df:87 C                     
eth0
172.16.0.1               ether   64:12:25:e3:d4:4c C                     
eth0
Entries: 3      Skipped: 0      Found: 3

We've been running VMs (KVM) with this identical bridged/bonded 
configuration for years and have never had an issue with a VM on one 
host being unable to communicate with a VM on another host.We've never 
had to force an arp table update of any kind. Why do containers behave 
in this manner?

Peter

On 09/10/2015 01:01 PM, Bostjan Skufca wrote:
> Hi Peter,
>
> since you mentioned you are using bridged interfaces, is my assumption
> correct that your containers's network connection is joined directly
> to this bridge and containers talk to the world direcly (L2) and not
> via routed (L3) network over host OS?
>
> Did you try using routed setup (using bond0 directly and creating
> dedicated br1just  for containers) and taking bridging functionality
> of the linux kernel out of the picture?
>
> I am very interested in your findings, as I have similar setup planned
> for deployment in the coming weeks (vlan trunk -> bond -> bridge ->
> containers).
>
> (I had similar issues on bonded interfaces on VMware, where tap-based
> OpenVPN server would not work at all. It had to do something with how
> vmware handles bonds and this not being compatible with L2 stuff
> coming out of VM. The second thing I remember very vaguely is that I
> tried using bridged interface inside container too, and it did not
> cooperate well with host's bridge, so I stopped using bridges inside
> containers altogether.)
>
> b.
>
>