[lxc-users] LAN for LXD containers (with multiple LXD servers)?

[Ron Kelley]

(Long reply follows…)

Personally, I think you need to look at the big picture for such deployments.  From what I read below, you are asking, “how do I extend my layer-2 subnets between data centers such that container1 in Europe can talk with container6 in Asia, etc”.  If this is true, I think you need to look at deploying data center hardware (servers with multiple NICs, IPMI/DRAC/iLO interfaces) with proper L2/L3 routing (L2TP/IPSEC, etc).  And, you must look at how your failover services will work in this design.  It’s easy to get a couple of servers working with a simple design, but those simple designs tend to go to production very fast without proper testing and design.


All that said, here is one way I would tackle this type of request:

* Get servers with at least 3 NICs (preferably 5)
  * One iLO/DRAC/IPMI interface for out-of-band management
  * One for Container server management (ie: LXD1 IP 1.2.3.4) - use a second NIC for redundancy in a bonded configuration
  * One for Container hosting network (ie container1, container2, etc) - use a second NIC for redundancy and VLANs to separate traffic

* Get firewalls in each location with L2TP/IPSEC support (pfSense works great)
  * Extend your L2 networks between your sites with L2TP
  * Secure the connection with IPSSEC

* On your LXD servers, create 2 bonded NICs
  * One for container management (eth0, eth1)
  * One for hosting network (eth2, eth3)
  * Use VLANs on hosting network to separate traffic
  * Configure your containers with the appropriate VLAN tag (ie: 501)

Once the above is done, your containers can talk w/each other in different locations.  You can use firewall rules to allow/deny IP connections from your container VMs.  You can extend both your container management and hosting networks across the L2 tunnel allowing you to move VMs at will.  


General Notes:
-----------------------
* For server bonded connections, I use linux mode type 6;  works well, provides great throughput, requires no special configuration on directly-connected switches.
* On the LXD side, create multiple profiles with VLAN configurations.  Personally, I have 2 profiles: one for VLAN 501 and one for VLAN 502.  Local firewall provides security between container networks.
* Be mindful of the services you share across the tunnels.  Things like iSCSI, NFS, etc will kill your network performance because of the chatty type of traffic.


Some good references:
-----------------------
https://doc.pfsense.org/index.php/L2TP/IPsec
http://archive.openflow.org/wk/index.php/Tunneling_-_GRE/L2TP
http://www.networkworld.com/article/2163334/tech-primers/what-can-l2tp-do-for-your-network-.html

Caution: L2 networks have a lot of broadcast traffic.  If your site-to-site connections are slow, your entire extended L2 network will suffer.  Must find a way to suppress L2 broadcast/multicast between sites.


Hope this helps.  Happy to share my LXD configurations with anyone...

-Ron






On Sep 18, 2016, at 5:16 AM, Tomasz Chmielewski <mangoo at wpkg.org> wrote:

It's easy to create a "LAN" for LXD containers on a single LXD server - just attach them to the same bridge, use the same subnet (i.e. 10.10.10.0/24) - done. Containers can communicate with each other using their private IP address.

However, with more then one LXD server *not* in the same LAN (i.e. two LXD servers in different datacentres), the things get tricky.


Is anyone using such setups, with multiple LXD servers and containers being able to communicate with each other?


LXD1: IP 1.2.3.4, Europe    LXD2: IP 2.3.4.5, Asia
container1, 10.10.10.10     container4, 10.10.10.20
container2, 10.10.10.11     container5, 10.10.10.21
container3, 10.10.10.12     container6, 10.10.10.22


LXD3: IP 3.4.5.6, US
container7, 10.10.10.30
container8, 10.10.10.31
container8, 10.10.10.32


While I can imagine setting up many OpenVPN tunnels between all LXD servers (LXD1-LXD2, LXD1-LXD3, LXD2-LXD3) and constantly adjusting the routes as containers are stopped/started/migrated, it's a bit of a management nightmare. And even more so if the number of LXD servers grows.

Hints, discussion?


Tomasz Chmielewski
https://lxadm.com
_______________________________________________
lxc-users mailing list
lxc-users at lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users