[lxc-users] Networking issues with LXC containers in EC2

[Peter Steele]

On 01/12/2016 07:03 PM, Fajar A. Nugraha wrote:
> On Tue, Jan 12, 2016 at 9:29 PM, Peter Steele <pwsteele at gmail.com> wrote:
>> On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote:
>>> On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele <pwsteele at gmail.com> wrote:
>>>> I should have added that I have no issue running our software on a single
>>>> EC2 instance with containers running on that instance. We can assign
>>>> multiple IPs to the instance itself, as well as to the containers running
>>>> under the instance, and the containers can all communicate with each
>>>> other
>>>> as well as with the host.
>>>
>>> can the containers in that setup communicate with systems outside the
>>> host (e.g. access the internet)?
>>>
>>> if "no", then you might hit the multiple mac problem
>> Sadly the answer is no. They cannot even ping another host in the same
>> VPC...
> Looks like multiple mac problem. As in, EC2 only allows one mac from
> your interface.
>
> Proxyarp should work:
>
> (1) Make SURE your EC2 instances (I'd call them "host" from now on)
> supports multiple IPs (private or elastic/public IPs, depending on
> your needs). The easiest way is to add those IPs to your host
> interface, make sure that that new IP can be accessed (e.g. ping that
> IP from another host), and then remove it.
>
> (2) Enable proxy arp on the host
>
> echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
>
It turned out that proxy arp was indeed the solution here, but a few 
other parameters had to be set as well. I just need to run the following 
commands on each EC2 instance:

echo 1 > /proc/sys/net/ipv4/conf/br0/forwarding
echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp_pvlan
echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/br0/send_redirects

With these settings, my containers and hosts can all talk to each other 
just like they were all residing on the same subnet. An easy solution in 
the end.

Peter