[Lxc-users] Lean containers and ro-mounts

[Daniel Lezcano]

On 10/06/2010 09:46 AM, Helmut Lichtenberg wrote:
> Hi,
> I have a question concerning the practical use of lxc containers.
>
> Currently we have lots of complete virtual machines (kvm and xen) for services
> like fileserver/samba, dhcp/cups, ldap/kerberos, terminalserver,
> computecluster, etc. on about 10 hardware machines (mostly recent 2-socket
> quadcores). This serves about 60 concurrent users in our research institute.
>
> As hardware gets more and more powerful, I would like to reduce the number of
> virtual machines (to make my life easier :^).
> Currently we reach the state to provide most of the services for the users
> like file- and terminalservices on *one* hardware machine (2-socket sixcore
> with X5680 CPU @ 3.33GHz), apart from any redundancy.
>
> My plan is to create really tiny containers to separate the services, and
> concentrate most of then on one machine.
>
> In a test container, I readonly bind-mounted the directories
>     /usr
>     /bin
>     /sbin
>     /lib
>     /lib32
>     /var/lib/dpkg
> into lxc which leads to about 200MB size of the remaining lxc-tree.
>
> The problem appeared, that this does not work out of the box.
> Simple programs like atd don't work as e.g. the directories
> /var/spool/cron/atjobs/ and /var/spool/cron/atspool/ do not exist, just as
> /etc/init.d/atd. They have been created on the host during installation.
>
> This is just a simple example, but in general, most of the programs in /usr,
> /bin, etc. have configurations in /etc and leave traces in /var during
> installation.
>
> Does anybody use such an appealing setup and how can one handle this problem?
>    

Maybe you can use unionfs or aufs on top of /var/spool and /etc/init.d, no ?
Another alternative, more experimental, would be to install a distro in 
an btrfs image file. You can create a snapshot for each container and 
use this snapshot as the rootfs for a container. As this filesystem is a 
COW, the 'atd' container will have its private data in /var/spool. 
Moreover, the initial image can be used as a backup installation as it 
will be never modified.