[lxc-users] Configuration for a bridged LXC container as a proxy web server

[Itamar Gal]

Hey lxc gang,

Here's the short version of my problem.

I have two LXC containers: lxc-proxy and lxc-app. I want to put lxc-proxy
on br0 with a (static) public IP address and on lxcbr0 (with a private IP),
and I want to put lxc-server on lxcbr0. Then I want to run Apache on
lxc-proxy as a proxy server to relay requests to a web application on
lxc-app.

I'm tinkering with both the lxc container configuration on the host and the
network interface configuration in the proxy container and I can get
different parts of this setup working at different times, but unfortunately
I can't seem to get everything working at once.


-----


Now here's the longer version.

I'm a junior systems administrator working in a (very) small department
with a mostly-inherited environment; I'm new to Linux containers.

I have a server running Ubuntu 12.04.5 LTS (Precise Pangolin) with kernel
3.2.0-24-generic. I want to set up an LXC container to function as a proxy
web server with a public IP address and then have separate containers for
individual web applications on the private network. My thinking is that the
proxy server will be attached to the public interface as well as the lxc
virtual bridge (i.e. br0 and lxcbr0) and that the other containers will
only be attached to the virtual network (i.e. lxcbr0 only). If someone has
any advice regarding better ways to achieve this kind of separation then
I'd be interested in hearing that as well, but I'm primarily interested in
understanding what mistakes I'm making in trying to implement this solution.

There are two configuration files that I'm tinkering with. One is the lxc
configuration file (on the host) for the proxy server:

host:/var/lib/lxc/proxy-server/config

and the other is the network interface configuration file (in the
container):

proxy-server:/etc/network/interfaces

or equivalently:

host:/var/lib/lxc/proxy-server/rootfs/etc/network/interfaces

In order to put the proxy server on the public network I initially had the
following two configurations:

# lxc-proxy:/etc/network/interfaces - version 1

# The loopback network interface
auto lo
iface lo inet loopback

# The public network interface (i.e. br0 on the host)
auto eth0
iface eth0 inet static
    address 1.2.3.4
    netmask 255.255.255.224
    network 1.2.3.0
    broadcast 1.2.3.31
    gateway 1.2.3.1
    dns-nameservers 5.6.7.8 9.10.11.12
    dns-search some.domain.com

and:

# host:/var/lib/lxc/lxc-proxy/config - version 1

lxc.network.type = veth
lxc.network.link = br0
lxc.network.name = eth0

[...]

This seems to work, in that I can access the proxy server at its public
address (e.g. 1.2.3.4) both on the local network and from elsewhere on the
internet. Then I try to add the virtual interface as well; so I have:

# lxc-proxy:/etc/network/interfaces - version 2

# The loopback network interface
auto lo
iface lo inet loopback

# The public network interface (i.e. br0 on the host)
auto eth0
iface eth0 inet static
    address 1.2.3.4
    netmask 255.255.255.224
    network 1.2.3.0
    broadcast 1.2.3.31
    gateway 1.2.3.1
    dns-nameservers 5.6.7.8 9.10.11.12
    dns-search some.domain.com

# The virtual network interface (i.e. lxcbr0 on the host)
auto eth1
iface eth1 inet static
    address 10.0.3.2
    netmask 255.255.255.0
    network 10.0.3.255
    broadcast 10.0.3.255
    gateway 10.0.3.1
    dns-nameservers 5.6.7.8 9.10.11.12
    dns-search some.domain.com

and:

# host:/var/lib/lxc/lxc-proxy/config - version 2

# The public network interface
lxc.network.type=veth
lxc.network.link=br0
lxc.network.name = eth0

# The virtual network interface
lxc.network.type=veth
lxc.network.link=lxcbr0
lxc.network.name = eth1

[...]

Unfortunately with this configuration I can't even access the lxc-proxy
console (i.e. sudo lxc-console -n lxc-proxy); I get a blank screen.

After a little bit of web-searching I found some blog posts that suggest
avoiding the local /etc/network/interfaces files in the containers, and
performing all of the network configuration inside of the lxc configuration
files. My next iteration looks like this:


# lxc-proxy:/etc/network/interfaces - version 3

# The loopback network interface
auto lo
iface lo inet loopback

and:

# host:/var/lib/lxc/lxc-proxy/config - version 3

# The public network interface
lxc.network.type=veth
lxc.network.link=br0
lxc.network.ipv4=1.2.3.4/27 1.2.3.31
lxc.network.name = eth0
lxc.network.flags=up

# The virtual network interface
lxc.network.type=veth
lxc.network.link=lxcbr0
lxc.network.ipv4=10.0.3.2/24
lxc.network.name = eth1
lxc.network.flags=up

[...]

With this configuration I'm able to access the lxc-proxy console and
everything appears to be working on the local network; lxc-proxy is
accessible via its private and public ip addresses and the Apache proxy
server on lxc-proxy is able to communicate with the application running on
lxc-app. However lxc-proxy is not publicly accessible by its public ip.

At this point I'm pretty thoroughly confused and I figure this is a good
place to stop and ask for advice. I'm not sure why my second attempt broke
the lxc console or why my third attempt isn't making the container
publically accessible. Any insight would be greatly appreciated.

Thank you for your time!

Cheers,
Itamar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20151027/1e3ca472/attachment-0001.html>