[lxc-devel] lxc-start: Invalid argument - pivot_root syscall failed

[Serge Hallyn]

Quoting Peter Volkov (pva at gentoo.org):
> В Пт, 18/10/2013 в 00:22 +0400, Peter Volkov пишет:
> > В Чт, 10/10/2013 в 14:47 -0500, Serge Hallyn пишет:
> > > Quoting Peter Volkov (pva at gentoo.org):
> > > > I'm using 1.0.0.alpha1 although I've tried with 0.8.0 also and I'm
> > > > unable to start container with the following error:
> > > > 
> > > > lxc-start: Invalid argument - pivot_root syscall failed
> > > > lxc-start: failed to setup pivot root
> > > > lxc-start: failed to set rootfs for 'repos'
> > > > lxc-start: failed to setup the container
> > > > lxc-start: invalid sequence number 1. expected 2
> > > > lxc-start: failed to spawn 'repos'
> > > > 
> > > > I've tried mount --make-private on all mount point I've thought of with
> > > > no luck.
> > > > Also I've tried lxc.autodev = 1 also no luck and I guess this is
> > > > relevant with systemd while this systems uses openrc as init system.
> > > > 
> > > > Container's conf file:
> > > > 
> > > > lxc.arch = amd64
> > > > lxc.utsname = repos
> > > > lxc.rootfs = /virt/lxc/repos
> > > > 
> > > > Distribution Gentoo. Same config works fine on another gentoo system.
> > > > Although systems are completely different I think important differences
> > > > are:
> > > > 1. init system: on laptop I'm using systemd while on server openrc
> > > > 2. on server I have full system inside ram (system resides inside
> > > > initramfs and after boot root stays in RAM on rootfs)
> > > 
> > > I think that's the problem.  I could be wrong, but I think it's
> > > refusing ecause your root doesn't have a parent, i.e. isn't
> > > mounted somewhere.
> > > 
> > > I suspect we want detect_shared_rootfs() updated to check for
> > > your rootfs being mount #1, and also return 1 in that case
> > > (meaning we will set up an environment in which you can in
> > > fact pivot_root).  
> > > 
> > > Is such a patch something you could write and test?
> > 
> > Well, it's not that easy, unfortunately. For tests I just modified
> > detect_shared_rootfs to return 1, so it'll detect that / is shared.
> > Tried and lxc-start failed with:
> > 
> > lxc-start: Invalid argument - failed to mount /usr/local/lib/lxc/rootfs
> > bind
> > lxc-start: Failed to chroot into slave /
> > 
> > and really, if I try manually mount it fails:
> > # mount -o bind /usr/local/lib/lxc/rootfs /usr/local/lib/lxc/rootfs
> > mount: wrong fs type, bad option, bad superblock
> > on /usr/local/lib64/lxc/rootfs,
> >        missing codepage or helper program, or other error
> >        In some cases useful info is found in syslog - try
> >        dmesg | tail or so
> > 
> > Ok, not a problem, I've added followint in config:
> > lxc.rootfs.mount = /virt/lxc/pivot_root
> > /virt/lxc/pivot_root are not on rootfs partion so mount -o bind works as
> > it should:
> > # mount -o bind /virt/lxc/pivot_root /virt/lxc/pivot_root
> > # 
> > 
> > Tried again and now lxc-start fails:
> >  # lxc-start -f repos.conf -n repos -l DEBUG -o lxc-start-debug 
> > lxc-start: Invalid argument - Failed to rbind mount /
> > to /virt/lxc/pivot_root/root
> > lxc-start: Failed to chroot into slave /
> > lxc-start: failed to setup rootfs for 'repos'
> > lxc-start: failed to setup the container
> > lxc-start: invalid sequence number 1. expected 2
> > lxc-start: failed to spawn 'repos'
> > 
> > Why does it needs to mount /? I've read the comments before
> > chroot_into_slave() but I don't understand why we need to do all of
> > that. Was there any discussion? 
> 
> Well finally I found problem. In
> Documentation/filesystems/ramfs-rootfs-initramfs.txt it is written:
> 
> Rootfs is a special instance of ramfs (or tmpfs, if that's enabled),
> which is   
> always present in 2.6 systems.  You can't unmount rootfs for
> approximately the  
> same reason you can't kill the init process; rather than having special
> code    
> to check for and handle an empty list, it's smaller and simpler for the
> kernel  
> to just make sure certain lists can't become empty. 
> 
> So mount --bind is not supposed to work on rootfs. Then I've wrote small
> init script that creates real 'tmpfs' (as opposed to 'rootfs'), moves
> system root there and then switch_root there. Now lxc-start works!
> Cool! :)

Might be worth having lxc check /proc/self/mountinfo to determine if
this is the case and give the user some meaningful info.

-serge