[lxc-users] lxc-execute with read-only rootfs

Thu Dec 19 14:41:14 UTC 2013

Hi Stephane,

Thanks for following-up with me. I actually have one last question.
Because I also have to compile students' codes and would like to do it
within the container, a read-only rootfs won't do the trick. I am
thinking of using an overlayfs as suggested by Cal and as is done with
ephemeral containers. Do you know what's the best way of setting up a
size quota for the oupperdir in ubuntu? Also, I read somewhere that
for ephemeral containers, the upperdir changes where stored in memory.
But I did not see anything special when I looked at the
lxc-start-ephemeral python source code. All I see is a call to
tempfile.mkdtemp to create the temporary directory for the upperdir.
Am I missing something?

Thanks,

Antonin

2013/12/19 Stéphane Graber <stgraber at ubuntu.com>:
> On Wed, Dec 18, 2013 at 06:29:57PM -0800, Antonin Bas wrote:
>> Thanks Cal. I will look into this.
>> However, for now, I have found a quick fix. I just added "lxc.pivotdir
>> = /mnt" to my config file. The pivotdir is used when pivoting the
>> original root file system. It seems that in older versions of lxc, the
>> default value was /mnt but this was changed since (in my case it was
>> /usr/lib/x86_64-linux-gnu/
>> lxc/lxc_putold by default). Nothing is actually written in this
>> directory, so it can work even in read-only mode. However, if this
>> location does not exist lxc will try to create it, which will fail.
>> mnt/ exists so no problem.
>> I hope I did not break anything by changing it to /mnt. So far it
>> seems to be working.
>
> You should be fine.
>
> The reason we set it to something other than our previous default of
> /mnt was issues when you had some mounts defined in the container's
> fstab that were also using /mnt.
> Apparently in your setup that's not the case so you won't have any problem.
>
>> Note that I still have the following error:
>>
>> lxc-execute: Read-only file system - error unlinking
>> /usr/lib/x86_64-linux-gnu/lxc/
>> dev/kmsg
>>
>> but it does not provide the container from spawning.
>>
>> Antonin
>>
>> 2013/12/18 Cal Leeming [Simplicity Media Ltd]
>> <cal.leeming at simplicitymedialtd.co.uk>:
>> > It looks like something being executed is expecting your rootfs to be
>> > writeable, this could be something in lxc-execute or it could be /bin/bash
>> > itself (I wouldn't know without strace'ing both or looking at the source).
>> >
>> > You could try adding the necessary /dev, /proc and /sys mounts, which may be
>> > enough to get it to run without error (add the following into your lxc
>> > config);
>> > http://pastebin.com/ZAcXn926
>> >
>> > However any applications expecting a write-able FS are going to fail hard -
>> > for example, anything relying on /tmp, or /var/log, and numerous other
>> > places. You could in theory bind mount a ramfs to those locations, but it
>> > would be easier, in my opinion, to use a stacked file system.
>> >
>> > Here is an example of using overlayfs;
>> > http://askubuntu.com/questions/109413/how-do-i-use-overlayfs
>> >
>> > In short, you'd build your container rootfs and make whatever changes you
>> > wanted to make, once you're happy with it you then do;
>> >
>> > $ mount -t overlayfs -o
>> > lowerdir=/your.rootfs.here,upperdir=/your.discarded.changes.here overlayfs
>> > /your.new.mount.here
>> > lxc.rootfs = /your.new.mount.here
>> >
>> > There is also a previous discussion on this;
>> > http://osdir.com/ml/lxc-chroot-linux-containers/2011-07/msg00019.html
>> > https://www.redhat.com/archives/libvirt-users/2011-December/msg00024.html
>> > http://s3hh.wordpress.com/2011/09/22/sharing-mounts-with-a-container/
>> > https://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg02190.html
>> >
>> > Hope this helps
>> >
>> > Cal
>> >
>> >
>> > On Thu, Dec 19, 2013 at 1:51 AM, Antonin Bas <antoninb at stanford.edu> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I guess that could do it. But I don't understand while it would not be
>> >> possible to have a read-only rootfs (i.e. is what I am seeing the
>> >> expected behaviour?).
>> >> What would the configuration look like if I decided to use overlayfs?
>> >> Is it easy to use with lxc-execute? Because I just want to run an
>> >> application, and I don't want to pay the overhead of
>> >> lxc-start-ephemeral.
>> >>
>> >> Thanks,
>> >>
>> >> Antonin
>> >>
>> >> 2013/12/18 Cal Leeming [Simplicity Media Ltd]
>> >> <cal.leeming at simplicitymedialtd.co.uk>:
>> >> > Would it not be better to use a stacked file system, such as overlayfs
>> >> > or
>> >> > aufs, then discard the changes?
>> >> >
>> >> > Cal
>> >> >
>> >> >
>> >> > On Thu, Dec 19, 2013 at 12:49 AM, Antonin Bas <antoninb at stanford.edu>
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I am trying to run an application container with lxc-execute. I am
>> >> >> going to run "untrusted" student codes in this container and I want
>> >> >> the root file system to be shared with the host but read-only. I
>> >> >> thought this would be as easy as using the following configuration
>> >> >> file:
>> >> >>
>> >> >> # Container with new network withtout network devices
>> >> >> lxc.utsname = omega
>> >> >> lxc.network.type = empty
>> >> >> lxc.network.flags = up
>> >> >>
>> >> >> lxc.rootfs = /tmp/guest/rootfs
>> >> >> lxc.mount.entry=/ /tmp/guest/rootfs/ none ro,bind 0 0
>> >> >>
>> >> >> However, when I run `sudo lxc-execute -n test -f grader.conf
>> >> >> --logpriority=DEBUG -- /bin/bash`, I get the following message:
>> >> >>
>> >> >> lxc-execute: Read-only file system - error unlinking
>> >> >> /usr/lib/x86_64-linux-gnu/lxc/dev/kmsg
>> >> >>
>> >> >> lxc-execute: failed to setup kmsg for 'test'
>> >> >> lxc-execute: Read-only file system - failed to create directory
>> >> >> '/usr/lib/x86_64-linux-gnu/lxc/lxc_putold'
>> >> >>
>> >> >> lxc-execute: Read-only file system - failed to create pivotdir
>> >> >> '/usr/lib/x86_64-linux-gnu/lxc/lxc_putold'
>> >> >> lxc-execute: failed to setup pivot root
>> >> >> lxc-execute: failed to set rootfs for 'test'
>> >> >> lxc-execute: failed to setup the container
>> >> >> lxc-execute: invalid sequence number 1. expected 2
>> >> >> lxc-execute: failed to spawn 'test'
>> >> >>
>> >> >>
>> >> >> Is it possible to have some insight on what the problem is here?
>> >> >> I am using Ubuntu 13.10, and my lxc is the one from the official repo
>> >> >> (1.0.0.alpha1).
>> >> >>
>> >> >> Thanks you in advance for your help,
>> >> >>
>> >> >> Antonin
>> >> >> _______________________________________________
>> >> >> lxc-users mailing list
>> >> >> lxc-users at lists.linuxcontainers.org
>> >> >> http://lists.linuxcontainers.org/listinfo/lxc-users
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > lxc-users mailing list
>> >> > lxc-users at lists.linuxcontainers.org
>> >> > http://lists.linuxcontainers.org/listinfo/lxc-users
>> >> _______________________________________________
>> >> lxc-users mailing list
>> >> lxc-users at lists.linuxcontainers.org
>> >> http://lists.linuxcontainers.org/listinfo/lxc-users
>> >
>> >
>> >
>> > _______________________________________________
>> > lxc-users mailing list
>> > lxc-users at lists.linuxcontainers.org
>> > http://lists.linuxcontainers.org/listinfo/lxc-users
>> _______________________________________________
>> lxc-users mailing list
>> lxc-users at lists.linuxcontainers.org
>> http://lists.linuxcontainers.org/listinfo/lxc-users
>
> --
> Stéphane Graber
> Ubuntu developer
> http://www.ubuntu.com
>
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users