[lxc-users] lxc-execute with read-only rootfs

Thu Dec 19 14:56:20 UTC 2013

On Thu, Dec 19, 2013 at 06:41:14AM -0800, Antonin Bas wrote:
> Hi Stephane,
> 
> Thanks for following-up with me. I actually have one last question.
> Because I also have to compile students' codes and would like to do it
> within the container, a read-only rootfs won't do the trick. I am
> thinking of using an overlayfs as suggested by Cal and as is done with
> ephemeral containers. Do you know what's the best way of setting up a
> size quota for the oupperdir in ubuntu? Also, I read somewhere that
> for ephemeral containers, the upperdir changes where stored in memory.
> But I did not see anything special when I looked at the
> lxc-start-ephemeral python source code. All I see is a call to
> tempfile.mkdtemp to create the temporary directory for the upperdir.
> Am I missing something?

lxc-start-ephemeral will mount a tmpfs if passed the right options.
In which case, changes are stored in RAM and lost on exit.

tmpfs can be size limited, so you could set that mount option and use
that as a quota.

For the other filesystems it's usually trickier, I believe none have
proper support for path-based quotas (though I'm not too familiar with
btrfs and zfs).
So usually your best bet is to either use LVM with one size limited LV
per container or use loop-mounted partition images, in both case you'll
basically end up with one partition of whatever size you choose for the
container rootfs. Increasing the size may be a bit tricky though...

I really wish we'd get more decent quota supports in the existing
filesystems as the existing user/group quotas have clearly shown their
limitations...

> 
> Thanks,
> 
> Antonin
> 
> 2013/12/19 Stéphane Graber <stgraber at ubuntu.com>:
> > On Wed, Dec 18, 2013 at 06:29:57PM -0800, Antonin Bas wrote:
> >> Thanks Cal. I will look into this.
> >> However, for now, I have found a quick fix. I just added "lxc.pivotdir
> >> = /mnt" to my config file. The pivotdir is used when pivoting the
> >> original root file system. It seems that in older versions of lxc, the
> >> default value was /mnt but this was changed since (in my case it was
> >> /usr/lib/x86_64-linux-gnu/
> >> lxc/lxc_putold by default). Nothing is actually written in this
> >> directory, so it can work even in read-only mode. However, if this
> >> location does not exist lxc will try to create it, which will fail.
> >> mnt/ exists so no problem.
> >> I hope I did not break anything by changing it to /mnt. So far it
> >> seems to be working.
> >
> > You should be fine.
> >
> > The reason we set it to something other than our previous default of
> > /mnt was issues when you had some mounts defined in the container's
> > fstab that were also using /mnt.
> > Apparently in your setup that's not the case so you won't have any problem.
> >
> >> Note that I still have the following error:
> >>
> >> lxc-execute: Read-only file system - error unlinking
> >> /usr/lib/x86_64-linux-gnu/lxc/
> >> dev/kmsg
> >>
> >> but it does not provide the container from spawning.
> >>
> >> Antonin
> >>
> >> 2013/12/18 Cal Leeming [Simplicity Media Ltd]
> >> <cal.leeming at simplicitymedialtd.co.uk>:
> >> > It looks like something being executed is expecting your rootfs to be
> >> > writeable, this could be something in lxc-execute or it could be /bin/bash
> >> > itself (I wouldn't know without strace'ing both or looking at the source).
> >> >
> >> > You could try adding the necessary /dev, /proc and /sys mounts, which may be
> >> > enough to get it to run without error (add the following into your lxc
> >> > config);
> >> > http://pastebin.com/ZAcXn926
> >> >
> >> > However any applications expecting a write-able FS are going to fail hard -
> >> > for example, anything relying on /tmp, or /var/log, and numerous other
> >> > places. You could in theory bind mount a ramfs to those locations, but it
> >> > would be easier, in my opinion, to use a stacked file system.
> >> >
> >> > Here is an example of using overlayfs;
> >> > http://askubuntu.com/questions/109413/how-do-i-use-overlayfs
> >> >
> >> > In short, you'd build your container rootfs and make whatever changes you
> >> > wanted to make, once you're happy with it you then do;
> >> >
> >> > $ mount -t overlayfs -o
> >> > lowerdir=/your.rootfs.here,upperdir=/your.discarded.changes.here overlayfs
> >> > /your.new.mount.here
> >> > lxc.rootfs = /your.new.mount.here
> >> >
> >> > There is also a previous discussion on this;
> >> > http://osdir.com/ml/lxc-chroot-linux-containers/2011-07/msg00019.html
> >> > https://www.redhat.com/archives/libvirt-users/2011-December/msg00024.html
> >> > http://s3hh.wordpress.com/2011/09/22/sharing-mounts-with-a-container/
> >> > https://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg02190.html
> >> >
> >> > Hope this helps
> >> >
> >> > Cal
> >> >
> >> >
> >> > On Thu, Dec 19, 2013 at 1:51 AM, Antonin Bas <antoninb at stanford.edu> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> I guess that could do it. But I don't understand while it would not be
> >> >> possible to have a read-only rootfs (i.e. is what I am seeing the
> >> >> expected behaviour?).
> >> >> What would the configuration look like if I decided to use overlayfs?
> >> >> Is it easy to use with lxc-execute? Because I just want to run an
> >> >> application, and I don't want to pay the overhead of
> >> >> lxc-start-ephemeral.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Antonin
> >> >>
> >> >> 2013/12/18 Cal Leeming [Simplicity Media Ltd]
> >> >> <cal.leeming at simplicitymedialtd.co.uk>:
> >> >> > Would it not be better to use a stacked file system, such as overlayfs
> >> >> > or
> >> >> > aufs, then discard the changes?
> >> >> >
> >> >> > Cal
> >> >> >
> >> >> >
> >> >> > On Thu, Dec 19, 2013 at 12:49 AM, Antonin Bas <antoninb at stanford.edu>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi,
> >> >> >>
> >> >> >> I am trying to run an application container with lxc-execute. I am
> >> >> >> going to run "untrusted" student codes in this container and I want
> >> >> >> the root file system to be shared with the host but read-only. I
> >> >> >> thought this would be as easy as using the following configuration
> >> >> >> file:
> >> >> >>
> >> >> >> # Container with new network withtout network devices
> >> >> >> lxc.utsname = omega
> >> >> >> lxc.network.type = empty
> >> >> >> lxc.network.flags = up
> >> >> >>
> >> >> >> lxc.rootfs = /tmp/guest/rootfs
> >> >> >> lxc.mount.entry=/ /tmp/guest/rootfs/ none ro,bind 0 0
> >> >> >>
> >> >> >> However, when I run `sudo lxc-execute -n test -f grader.conf
> >> >> >> --logpriority=DEBUG -- /bin/bash`, I get the following message:
> >> >> >>
> >> >> >> lxc-execute: Read-only file system - error unlinking
> >> >> >> /usr/lib/x86_64-linux-gnu/lxc/dev/kmsg
> >> >> >>
> >> >> >> lxc-execute: failed to setup kmsg for 'test'
> >> >> >> lxc-execute: Read-only file system - failed to create directory
> >> >> >> '/usr/lib/x86_64-linux-gnu/lxc/lxc_putold'
> >> >> >>
> >> >> >> lxc-execute: Read-only file system - failed to create pivotdir
> >> >> >> '/usr/lib/x86_64-linux-gnu/lxc/lxc_putold'
> >> >> >> lxc-execute: failed to setup pivot root
> >> >> >> lxc-execute: failed to set rootfs for 'test'
> >> >> >> lxc-execute: failed to setup the container
> >> >> >> lxc-execute: invalid sequence number 1. expected 2
> >> >> >> lxc-execute: failed to spawn 'test'
> >> >> >>
> >> >> >>
> >> >> >> Is it possible to have some insight on what the problem is here?
> >> >> >> I am using Ubuntu 13.10, and my lxc is the one from the official repo
> >> >> >> (1.0.0.alpha1).
> >> >> >>
> >> >> >> Thanks you in advance for your help,
> >> >> >>
> >> >> >> Antonin
> >> >> >> _______________________________________________
> >> >> >> lxc-users mailing list
> >> >> >> lxc-users at lists.linuxcontainers.org
> >> >> >> http://lists.linuxcontainers.org/listinfo/lxc-users
> >> >> >
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > lxc-users mailing list
> >> >> > lxc-users at lists.linuxcontainers.org
> >> >> > http://lists.linuxcontainers.org/listinfo/lxc-users
> >> >> _______________________________________________
> >> >> lxc-users mailing list
> >> >> lxc-users at lists.linuxcontainers.org
> >> >> http://lists.linuxcontainers.org/listinfo/lxc-users
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > lxc-users mailing list
> >> > lxc-users at lists.linuxcontainers.org
> >> > http://lists.linuxcontainers.org/listinfo/lxc-users
> >> _______________________________________________
> >> lxc-users mailing list
> >> lxc-users at lists.linuxcontainers.org
> >> http://lists.linuxcontainers.org/listinfo/lxc-users
> >
> > --
> > Stéphane Graber
> > Ubuntu developer
> > http://www.ubuntu.com
> >
> > _______________________________________________
> > lxc-users mailing list
> > lxc-users at lists.linuxcontainers.org
> > http://lists.linuxcontainers.org/listinfo/lxc-users
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users

-- 
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20131219/5381aacb/attachment.pgp>