[Lxc-users] appropriate architecture for two sets of containers on one host

[Jäkel, Guido]

Dear Mike,

Don't put an IP on the second (or further) bridges. Think about this bridges configuration slot as an additional virtual interface card to connect your hosts IP stack with this network. Said that, you will not be surprised that you got two network interface devices and two default routes with your configuration.

And as you would do it with a plain machine with two network cards on two different networks, you'll get in trouble to route your outgoing traffic. And run into advanced problems, if there will be multiple routes to reach your host, especially if this traffic goes through a statefull firewall. 

I think you don't want (and I even would say you should not) to offer any services by the host. Therefore you should not need to reach the host from the VLANs you're using for the different groups of containers (; respectively you don't need an IP for the host in this subnets).

I would suggest to use a separate "management" network/VLAN. Then, just add a bridge for it and put the hosts IP to it to plug hosts IP stack to it. You'll have a simple default route to your router and it's up to it to provide/control the interconnection to other network ranges.


>I'm curious how you configured yours.

Because I use PXE and a NFS-rootfs for my hosts, I'm using two physical interfaces on it. The "eth0" is to access the host. It's attached to an untrunked "plain old" port on the switch because I can't find howto PXE and NFS-boot from a trunked VLAN. To provide access to our different VLANs for the containers, the "eth1" takes all the virtual vlan interfaces (named vlan#) and this the bridges (br###). This takes the outer side of the container veth's (named by the container name) and at inside the container you'll see the VLAN of interest unrolled on eth0. If you're know what you doing, you also may connect the container to more VLANs by adding additional veths. And because the containers rootfs and dataspace-fs is on a NFS mount done by host too, this network traffic goes through eth0. Therefore, there's no need to expose the storage architecture or other backside services to another VLAN than the "management" one.

For separation of concerns I also suggest to use DHCP to configure the containers. Many routers will provide a DHCP "relay agent" (i.e. a DHCP proxy) to spawn across networks; with this you don't need to make your DHCP server "multi-homed" to all the VLANs.


Greetings

Guido


>dropping the PVID from the switch.  But when I added another VLAN:
>
>+------------------------------------------------+
>|+----+                                          |
>||    |------------+ c1                          |
>|| c1 | eth0/.17.3 |---\                         |
>||    |------------+  +--------+   +---------+   |------------+
>|+----+               | br1.17 |---| eth1.17 |---| eth1/.17.2 |-----
>|+----+               +--------+   +---------+ | |------------+
>||    |------------+    |                      | |
>|| c2 | eth0/.17.4 |---/                       | |
>||    |------------+ c2                        | |
>|+----+                                        | |
>|                     +--------+   +---------+ | |
>|                     | br1.18 |---| eth1.18 |-/ |
>|                     +--------+   +---------+   |
>+------------------------------------------------+
>
>with
>
>---------------------------------------------
>iface eth1.18 inet manual
>
>auto br1.18
>iface br1.18 inet static
>	bridge_ports eth1.18
>	bridge_maxwait 0
>	bridge_fd 0
>	bridge_stp off
>	address 192.168.18.2
>	netmask 255.255.255.0
>	gateway 192.168.18.1
>	dns...
>
>iface eth1.17 inet manual
>
>auto br1.17
>iface br1.17 inet static
>	bridge_ports eth1.17
>	bridge_maxwait 0
>	bridge_fd 0
>	bridge_stp off
>	address 192.168.17.2
>	netmask 255.255.255.0
>	gateway 192.168.17.1
>	dns...
>---------------------------------------------
>
>in /etc/network/interfaces, I got two default routes:
>
>---------------------------------------------
>host$ ip route show
>192.168.18.0/24 dev br1.18  proto kernel  scope link  src 192.168.18.2
>192.168.17.0/24 dev br1.17  proto kernel  scope link  src 192.168.17.2
>default via 192.168.17.1 dev br1.17
>default via 192.168.18.1 dev br1.18
>---------------------------------------------
>