[lxc-users] LXD in EC2

[Brian Candler]

I am trying to get LXD containers to work inside an EC2 instance. I want 
each container to have its own VPC IP address (e.g. 10.0.0.x) so that it 
appears like another VM.

Here's what I've managed to find so far:

1. EC2 networking only allows traffic to/from IP addresses which have 
been explicitly assigned as secondary addresses to the instance.

This is relatively straightforward:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/MultipleIP.html

although there are limits:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

For example, a t2.medium instance allows up to three NICs each with 6 IP 
addresses

Note also that if you create a second NIC then the primary NIC loses its 
auto public IP address, so you have to use an Elastic IP if you want 
external connectivity.

2. I was able to create a bridge on a second NIC (leaving my primary NIC 
with its original config, so as not to lock myself out), and create LXD 
instances statically configured with the secondary addresses of the EC2 
instance.

Unfortunately, it seems that Amazon has static mappings of IP addresses 
to the instance's MAC address. I've been able to demonstrate this with 
tcpdump.  Indeed, if you send out ARP queries from the primary 
interface, EC2 proxy-ARP responds with the MAC address of the instance.

This means that the LXD container's MAC addresses are not learned, and 
traffic can't get to them :-(

So I'm wondering if anyone has come across this, and how they've dealt 
with it? Approaches I can think of:

- put the containers on a separate internal subnet (say lxdbr0), and add 
one-to-one NAT mappings using iptables. [Normally this would also 
require proxy ARP on the outside as well, but it seems EC2 is doing that 
for me already anyway]

- messing around with ebtables so that the MAC addresses of incoming 
(and possibly outgoing) packets are rewritten at layer 2

- any other suggestions?

Thanks,

Brian Candler.