[Lxc-users] [lxc-devel] [systemd-devel] Unable to run systemd in an LXC / cgroup container.

[Serge Hallyn]

Quoting Michael H. Warfield (mhw at WittsEnd.com):
> On Sun, 2012-10-28 at 18:52 +0100, Serge Hallyn wrote:
> > Quoting Michael H. Warfield (mhw at WittsEnd.com):
> > > On Sat, 2012-10-27 at 13:51 -0400, Michael H. Warfield wrote:
> > > > On Sat, 2012-10-27 at 13:40 -0400, Michael H. Warfield wrote:
> > > > > /me erasing everything at this point and taking off the systemd crew,
> > > > > since this will have no relevance to them...
> > > > > 
> > > > > Testing the hook feature out using git rev (finally got it built)...
> > > > > 
> > > > > I added this line to my config...
> > > > > 
> > > > > lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
> > > > > lxc.hook.mount = /var/lib/lxc/Plover/mount
> > > > > 
> > > > > In /var/lib/lxc/Plover/mount I have this:
> > > > > 
> > > > > -- 
> > > > > rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
> > > > > -- 
> > > > 
> > > > > (This is just testing out the concepts.
> > > > 
> > > > > If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
> > > > > mounting takes place and lxc.hook.mount takes place after the mount.
> > > > 
> > > > > Problem is, the result of that rsync is not showing up in the mounted
> > > > > tmpfs file system but is showing up in the underlying parent file system
> > > > > as if it were run pre-mount.  Something not right here...
> > > 
> > > > I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
> > > > script in the container) which then works but that then requires the
> > > > template and the command to be in the container.  Suboptimal to say the
> > > > least.  But it gives me a way to test this tmpfs thing out.
> > > 
> > > > I also noticed that the .start hook runs, it appears, after caps are
> > > > dropped and I see a lot of bitching about mknod on certain devices.  I
> > > > had to thrown an exit 0 into that script so it would continue in spite
> > > > of the errors but, now, I can refine my template based on what it could
> > > > create.
> > > 
> > > Crap.  I've got a catch-22 here...  This is going to take some work.
> 
> > Hey,
> 
> > I've got a rather minimal patch (appended below) to add the support for
> > mounting and populating a minimal /dev working.  (A few hours were wasted
> > due to my not knowing that upstart was going to issue mounted-dev even though
> > /dev was mounted before upstart started - and the mounted-dev hook deletes
> > and recreates all consoles.  GAH)
> 
> I am happy to report that, after manually editing my git head branch to

Sorry, it was against the ubuntu quantal package.  I've been in the air
without onboard wifi, so working with what i had at hand.

> patch in the failed hunks, I was able to build it and test it and my
> Fedora 17 systemd based container fired right up after adding the
> lxc.autodev = 1 parameter to the config file.  Yeah!!!!
> 
> I did run into one gotcha, but one I can live with.  I had been bind
> mounting the private root file system to another directory and then
> using that as the rootfs like this:
> 
> -- 
> lxc.rootfs = /srv/lxc/rootfs
> lxc.mount.entry=/srv/lxc/private/Alcove /srv/lxc/rootfs none bind.shared 0 0
> lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
> 
> lxc.autodev = 1
> -- 
> 
> This did not work and I got the startup error that it can not mount
> to /dev because it doesn't exist.

Hm, yeah.  If you do need to play a game like this, you might be best
off using a pre-mount hook for that.

> I believe I can see why...  You're doing the autodev populate prior to
> any of the mounts being performed, so that "private" root file system is
> not bound to the directory at that time.
> 
> Drop that bind mount for the rootfs and this then worked like a charm:
> 
> -- 
> lxc.rootfs = /srv/lxc/private/Alcove
> lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
> 
> lxc.autodev = 1
> -- 
> 
> I think that rootfs directory bind was an effort to more fully match the
> OpenVZ behavior but also trying to deal with some of the read-only
> problems were where having in the past with shutdowns.  If it won't
> work, it won't work and I won't miss it.
> 
> I did see some errors setting up that dev...
> 
> -- 
> [root at forest mhw]# lxc-start -n Alcove
> lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
> lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
> lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
> lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
> lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
> lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
> systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
> 
> Welcome to Fedora 17 (Beefy Miracle)!
> 
> -- 
> 
> Not sure what that's all about but, since systemd isn't going to start
> getty's on the tty? interfaces anyways, it probably doesn't make much
> difference.

Oh, I see.  Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files.  I need to fix that.

Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?