[Lxc-users] multi-homed host

[Brian K. White]

Shouldn't I be able to have two different nics on a host, on two 
different, unrelated, public networks, and have two bridge devices on 
the host, and some containers on one bridge and some containers on the 
other bridge, and have all containers be able to talk to their 
respective internet connections regardless which nic happens to be the 
default gateway fro the host?

Host setup:

eth0 -> 10.0.0.x -> lan with other 10.0.0.x machines

eth1 -> br0 -> a.a.a.x -> public wan 1 , cable modem

eth2 -> br1 -> b.b.b.x -> public wan 2 , fios

ip forwarding is enabled

eth0 lan works fine.
The host talks to other 10.0.0.x boxes via this with no problem.

eth1/br0 works fine.
The hosts's default gateway is a.a.a.1
The host talks to the internet & vice/versa just fine via this.

eth2/br1 works fine from the hosts point of view.
other b.b.b.x machines are reached directly via this, not routing over 
eth1/br0.

Containers:

Containers with a.a.a.x ip's work fully and as expected.
They can reach the internet and the internet can reach them.
These containers have a.a.a.x ips and their default gw is a.a.a.1

Containers with b.b.b.x addresses do not work fully.
These have b.b.b.x ip's and default gw b.b.b.1
They can see the host and each other on the same host, and they can even 
see other neighboring b.b.b.x hosts, external to the host, but on the 
same physical local switch where traffic does not have to go out of the 
switch up to the b.b.b.1 default gateway.
(b.b.b.1 is on the other end of the fios line, not on premises and not 
owned or operated by me but by verizon)

None of the hosts nor the switch has any vlans or tagging other than the 
default vlan id is 1 in the switch when left undefined.
Software firewalls are disabled in the hosts and containers at least for 
now while still trying to figure this out.

What in the world could allow a container in the host talk outside the 
host well enough to talk to other neighboring hosts on the same switch, 
but but just not be able to reach the default gateway outside the 
switch? It's like the gateway has firewalled certain ip's and not 
others, but the ips actually work fine if put on a laptop directly or if 
the hosts default gateway and nameserver are switched over to the 
b.b.b.x network. Say the host br1 is b.b.b.50 and a container is 
b.b.b.60, and there is one single switch connecting 4 things
b.b.b.1 - default gateway on other end of uplink
b.b.b.40 - neighboring host, regular traditional server, single ip.
b.b.b.41 - neighboring host, regular traditional server, single ip.
b.b.b.50 - the host
b.b.b.51 - container 1 on host
b.b.b.52 - container 2 on host
All but the container are plugged into the same single switch, but .50 
and .51 are on the same bridge on the host.

The host .50 can ping and be pinged by all, itself, it's containers, 
neighboring hosts, containers inside neighboring hosts, and the gateway.

The container .51 can ping .50, .52, and .40 and .41, but not .1 !
How in the world can .51 reach across the hosts br1 and across the 
switch to .41, and yet not do exactly the same thing for .1 which is 
exactly the same number and forms of hops away ?

I've already called verizon tech support and they just said their equip 
ony reports all well, and I tested all ip's with a laptop directly on 
the b.b.b.x ethernet drop and they all worked fine that way , and 
swapped out my switch for another one just for the heck of it, so I'm 
down to config in my lxc hosts as the culprit.

About the only consistent pattern I can find is the hosts default 
gateway. The only the containers that work fully are the ones that 
happen to use the same gateway as the host, but if a bridge interface is 
just a "software switch" then why should the hosts default gateway 
setting matter at all to the containers ability to talk across it?

-- 
bkw