[Lxc-users] Converting existing CentOS 6.x to container within Ubuntu 12.04 - can that be simple?

[Whit Blauvelt]

On Fri, Nov 02, 2012 at 05:35:54AM +0700, Fajar A. Nugraha wrote:

> Having said that, I recall some VPS providers enforcing the same
> limitation, so your condition is quite common. Please share whatever
> ended up working for you so others can benefit from it as well.

I've almost got it fully working. Taking some ideas from here:

http://www.activestate.com/blog/2011/10/virtualization-ec2-cloud-using-lxc

I've created a virtual bridge on the host, put a subnet on that bridge, and
connected the guest to the host through the subnet. I've also added a second
IP to the host's LAN interface, and used DNAT and SNAT in iptables to
connect that to the guest. 

The private bridge is on 192.168.3.0/24, and the LAN on 10.196.58.0/24, with
the second LAN IP on the host 10.196.58.117, and the IP of the guest on the
private bridge 192.168.3.134.

So my two iptables rules on the lxc host are simply:

iptables -t nat -A PREROUTING -d 10.196.58.117 -j DNAT --to-destination 192.168.3.134
iptables -t nat -A POSTROUTING -s 192.168.3.134 -j SNAT --to 10.196.58.117

which results in these rules in the nat table:

Chain PREROUTING (policy ACCEPT 2676 packets, 427K bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1       73  3672 DNAT       all  --  *      *       0.0.0.0/0            10.196.58.117        to:192.168.3.134

Chain POSTROUTING (policy ACCEPT 37927 packets, 2358K bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1     1621  121K SNAT       all  --  *      *       192.168.3.134        0.0.0.0/0            to:10.196.58.117

That's working to a large extent, but not completely. From outside I can SSH
into the guest at the host's 10.196.58.117 address. And from the guest, I
can ping out to anywhere. But from the guest, I am not able to SSH to
anywhere. I can't mount filesystems from other systems either. In some cases
SSH attempts will simply give a "Host key verification failed" message -
this to hosts which the lxc host can SSH to with no problem. In others SSH
just hangs. File system mounts fail with "mount error(13): Permission
denied" - this with mount commands which work perfectly on the lxc host.

So the solution so far is fine for an lxc container that's simply going to
be a server, without needing to mount any filesystems hosted on other
systems or SSH to other systems. It's working within the restrictions of the
VMware host that's underneath this, which allows multiple IPs on its guests
(such as the lxc host here) but not multiple MAC addresses. The MAC address
restriction prevents the lxc guest from sharing the host's bridge set up on
the host's VMware-LAN-facing interface.

My real-world use for this setup requires mounting filesystems which are
outside the guest. If there's a way to mount them to the lxc host and make
them available to the guest I haven't found it. SSH'ing out from the lxc
guest isn't a requirement. But I suspect that and the filesystem mounts are
failing for a common reason.

Ideas?

Thanks,
Whit