[lxc-devel] Networking and LXD

[Tom Denham]

I'm going to need to digest this a bit more but my initial concern is that lxd relies on bridges for networking whereas Calico is more L3 focused.

The way Calico works is similar to what you described below
* Create a veth
* Put one end in the netns and leave the other in the host netns
* But don't connect the host end to a bridge
* Instead, we program rules into the linux routing tables to forward the packets to the correct destination (and program IPTABLES rules to enforce policy)

Make sense?

Tom

-----Original Message-----
From: Serge Hallyn [mailto:serge.hallyn at ubuntu.com] 
Sent: Thursday, September 24, 2015 8:43 AM
To: LXC development mailing-list
Cc: Tom Denham
Subject: Re: [lxc-devel] Networking and LXD

Quoting Tom Denham (Tom.Denham at metaswitch.com):
> Hi,
> Please point me in the right direction if this is the wrong forum for this question...
> 
> I work on Project Calico (http://www.projectcalico.org) and we provide networking for containers. I was just chatting with Dustin Kirkland at Container Summit and he suggested I should post here with my quesstions...
> 
> How would I go about integrating Calico with LXD? I've taken a quick look but I can't find any documentation. Can someone point me in the right direction?
> 
> Thanks
> Tom

Hi Tom,

I looked around at the website a bit, but all the info I saw was either very low level (how the host kernel does it) or very high level (how to use calicoctl specifically with docker with what appears to be intelligence baked into the tool).  I could look at the source, but the info I'm looking for (which may already be there and I just didn't find it) is how would I use this by hand?  So for instance if I simply create a new network namespace, how would I get calico networking in that?  As an example, with veth that would be

Terminal 1                                  | Terminal 2
=============================================================================
ip link add type veth
                                              sudo unshare -mn -- bash
                                              #  echo $$
                                              598 ip link set veth0 netns 598 brctl addif br0 veth1

And now the new netns in terminal 2 can use veth0 over the host's br0.

By default that is also how lxd happens to do its networking.  It sets up a bridge at boot time over which containers can talk.  The same sort of thing should be doable for calico, but I don't know how the container network actually gets set up there.

I also don't know whether and how calico's design would impact nesting.
We can run lxd inside lxd inside lxd (as Dustin demonstrated this week).
I know Stéphane has run bgp routers in nested containers before so I don't think calico should have any problems with nesting, but if it does it'd be nice if we can work around it.

Getting back to the general lxd network documentation, there is a bit in https://github.com/lxc/lxd/blob/master/specs/configuration.md and https://github.com/lxc/lxd/blob/master/specs/command-line-user-experience.md .
But network configuration is done by defining network devices for containers or container profiles (for multiple containers).  For instance, to add a nic to container c1, you would do

lxc config device add c1 eth1 nic nictype=bridged parent=lxcbr0

Then as the container starts, the above process basically happens using host bridge lxcbr0.

Hope this helps.

-serge