[Lxc-users] Container size minialisation

[Zhu Yanhai]

2011/12/13 Zhu Yanhai <zhu.yanhai at gmail.com>:
> Hi,
>
> 2011/12/13 Derek Simkowiak <derek at simkowiak.net>:
>>> I'm trying to compose a system, where lxc containers behave like virtual
>>> hosts for a web server.
>>
>>     It is possible to use shared, read-only mounts between the host and
>> containers.  However, you will need to carefully consider your security
>> requirements and maintenance procedures.
>>
>>     If you are using shared mounts, then upgrading software on one container
>> will update files in the other containers.  That could be a problem, because
>> software updates usually include a post-install script that is executed at
>> the end of an upgrade (like a postinst file in the .deb package).  These
>> scripts may restart a service, update a configuration file in /etc/, or even
>> prompt the user for some input.
>>
>>     So what will happen when you update the LAMP files on your shared mount,
>> but your LXC containers don't do a corresponding server restart, or config
>> migration?  Things will probably break at some point.
>>
>>     Also, there are several security risks to consider.  A shared /home/
>> directory would also (implicitly) share everyone's .ssh/authorized_keys
>> files (which will grant OpenSSH access to all your containers).  You would
>> also need to be sure that all SSL and SSH host certs are independently
>> managed.  Using a single certificate for many hosts is not secure.  Apache
>> and OpenSSH keep their certs under /etc/, but Tomcat does not iirc.
>>
>>     Also, UIDs and GIDs are shared on the filesystem, so a root user in any
>> container would be able to alter any file in any other container (unless
>> it's a read-only mount from an external fstab file, and the "sys_admin"
>> capability is dropped in your lxc.conf).  What's worse, if you have
>> different /etc/passwd or /etc/group files in the containers, then group id
>> "121" might be the group "admin" in one container, but something else
>> entirely in another container.  The shared filesystem only stores the
>> integer group ID, not the actual group membership or resulting sudo
>> permissions.
>>
>>     Because of these complications, I have decided to give each LXC
>> container its own, full filesystem.  Unfortunately that "wastes" a few
>> hundred megs of disk space for each container, because the files are mostly
>> redundant in /usr/, /var/, etc.  However, disk space is very cheap, and the
>> value of having a standalone container is more than worth it to me.
>>
>>
>>
>>> Has anyone any experience with this technique?
>>
>>     I include a sample configuration for a shared filesystem with my
>> container creation script.  It is disabled by default, but you can read
>> through the configuration to get an idea.  You can download it from here:
>>
>> http://derek.simkowiak.net/lxc-ubuntu-x/
>>
>>
>> Thanks,
>> Derek Simkowiak
>>
>>
>> On 12/12/2011 09:47 AM, István Király - LaKing wrote:
>>
>> Hi folks.
>>
>> I'm trying to compose a system, where lxc containers behave like virtual
>> hosts for a web server.
>>
>> As next step I would like to minimize container size. My question is, what
>> the best, most elegant and fail proof  technique for that?
>>
>> At this moment I'm thinking of a "master container" and "slave containers"
>> where the /usr folder for example in the slave containers is a mount from
>> the master container. That gives a significant size drop already, from 400
>> to 40 megabytes.
>>
>> I would like to keep the containers really minimal. 4 megabyte should be
>> small enough.
>>
>> Lets say only some important files in /etc ....
>>
>> Has anyone any experience with this technique?
>>
>> Thank you for sharing.
>>
>> greetings,
>> István Király
>>
>>
>> LaKing at D250.hu
>>
>> D250 Laboratories
>> www.D250.hu
>>
>> ------------------------------------------------------------------------------
>> Learn Windows Azure Live!  Tuesday, Dec 13, 2011
>> Microsoft is holding a special Learn Windows Azure training event for
>> developers. It will provide a great way to learn Windows Azure and what it
>> provides. You can attend the event by watching it streamed LIVE online.
>> Learn more at http://p.sf.net/sfu/ms-windowsazure
>> _______________________________________________
>> Lxc-users mailing list
>> Lxc-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/lxc-users
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Learn Windows Azure Live!  Tuesday, Dec 13, 2011
>> Microsoft is holding a special Learn Windows Azure training event for
>> developers. It will provide a great way to learn Windows Azure and what it
>> provides. You can attend the event by watching it streamed LIVE online.
>> Learn more at http://p.sf.net/sfu/ms-windowsazure
>> _______________________________________________
>> Lxc-users mailing list
>> Lxc-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/lxc-users
>>
>
> It's not only an issue about disk space, it's about page cache cost.
> If the under level files are shared, the OS see them as unique inodes,
> so the OS allocate only one copy for each one inode. However if we
> copy them as separate directories, each of them will occupy one copy
> in the page cache, resulting in large cost of memory.
> I think a better approach here is a file-level copy-on-write trick in
> filesystem or some other proper layers. Initially all files are shared
> in the host, and copy one as a private item at the first time it is
> touched in one containers.
>
> --
> Thanks,
> Zhu Yanhai

P.S.  Although we can setup KSM for the duplicated page cache, like
what we did for XEN/KVM machines, they will of course introduce extra
computing overhead, and footprint, which is not good for such light
weight containers as LXC.

-zyh