[lxc-users] Unable to Start Unprivileged Containers on Debian / Jessie

[Serge Hallyn]

Quoting Michael H. Warfield (mhw at WittsEnd.com):
> On Tue, 2014-09-30 at 19:39 -0400, Michael H. Warfield wrote:
> > On Tue, 2014-09-30 at 23:56 +0100, Chris wrote:
> > > On 30/09/14 19:28, Michael H. Warfield wrote:
> > > >> I haven't looked a whole lot into the premade containers, my gut feeling
> > > >> was that I didn't want to download a whole operating system from this
> > > >> project, and that I'd be a lot more comfortable taking distribution that
> > > >> I trust, and making the template manually. This way I know everything
> > > >> extra that's going into it.
> > > > Our templates are pretty barebones.  Very minimal.  You'll have to add
> > > > just about anything you would really want to make a useful container.
> > > I should definitely take a closer look sometime.
> > > >> It's running Debian Jessie. LXC 1.0.5-3 from package management. And
> > > >> systemd 208-8 also from package management.
> > > > OK... THAT explains a LOT!  That systemd option is why you're running
> > > > into this problem and you're about to have far worse.
> > > >
> > > >> I took a config from an existing container and modified it for what I
> > > >> thought would work for an unprivileged container. I've attached the
> > > >> config for osmium. I've also attached the latest trace output from the
> > > >> lxc-start, as I've fixed a few slight errors in the config since then.
> > > > You're going to have to make some additional changes...  Make sure you
> > > > add "lxc.kmsg = 0" to your container or systemd.journald is going to eat
> > > > your CPU time for lunch (and be sure to flush
> > > > your /dev/.lxc/user/osmium* directory).  There's also some adjustments
> > > > that need to be made for mgetty consoles and such.  You also need to
> > > > link the shutdown unit to the SIGPWR service to allow lxc to shut the
> > > > container down gracefully.  You might take a look at the Oracle or
> > > > Fedora templates for some guidance there.
> > > Will definitely come back to this once it starts up, thank you for the 
> > > advice.
> > > >> osmium at cadmium:~$ find /dev/.lxc/user -ls
> > > >>     9668    0 drwxrwxrwt   3 root     root           60 Sep 30 15:38
> > > >> /dev/.lxc/user
> > > >>    11109    0 drwxr-xr-x   3 427680   427680         60 Sep 30 15:38
> > > >> /dev/.lxc/user/osmium.3c68b3f0c5eeec7d
> > > >>    11110    0 drwxr-xr-x   2 427680   427680         40 Sep 30 15:38
> > > >> /dev/.lxc/user/osmium.3c68b3f0c5eeec7d/pts
> > > > Bingo!
> > > >
> > > > Ok...  So it appears that lxc-start did manage to create your dev
> > > > directory properly under the host /dev/.lxc/user.
> > > >
> > > > Now I see the real problem...
> > > >
> > > > The same code that creates that directory creates the symlink
> > > > in /home/osmium/.local/share/lxc/osmium.  But, the /dev/ directory is
> > > > owned by "427680:427680" while the directory containing the symlink is
> > > > own by "osmium:osmium" and you then have a permission denied because
> > > > 427680:427680 doesn't have write permissions
> > > > to /home/osmium/.local/share/lxc/osmium.
> > > >
> > > > That's a (the!) problem.  I'm just not sure if chown/chgrp is the
> > > > correct answer or if you need to add some group membership and add group
> > > > write permissions with appropriate host auth secondary groups.  Either
> > > > way, it's that permission problem that biting you in the rear end.
> > > >
> > > OK, yes. This was that problem. Fixing it has progressed startup a 
> > > little further. It didn't like the lxc.mount.entry for devpts, so I 
> > > threw that out for the time being also. Now it's still stuck at 
> > > 'populating dev' though. I've attached the latest trace in case you help 
> > > me again.
> > > 
> > > osmium at cadmium:~$ lxc-start -n osmium -l trace -o /tmp/xxx7
> > > lxc-start: Operation not permitted - Error creating null
> 
> > Ouch.  Looks like it's attempting to do a mknod which is not allowed and
> > can not be allowed as it's prohibited in the kernel to any non-priv
> > users.  That's a security issue that can not be changed.  That's going
> > to be a problem for any autodev containers running under non-priv users.
> > Serge and/or Stéphane will have to respond to that one.
> 
> > That was actually a case I was considering when I was discussing the
> > whole devtmpfs thing with Serge back at LinuxPlumbers last year.
> > Non-priv users can not create devices (mknod), period, end of
> > discussion.  But they might be able to create hard links to them within
> > devtmpfs or do bind mounts, which might work under the current setup.
> > That was the intent at least.  I don't recall the hard link device code
> > being done (though we do bind mounts on some devices) so I'm not sure
> > what's transpiring at this point in the code.
> 
> Maybe I did test that out last year.  IAC...  Just retested it.  Damn
> it, oh yeah...  Hardlinks to device nodes in devtmpfs are prohibited to
> non-priv users even when permissions would permit.  That option is also
> non viable.  Nice idea.  Guess I know why none of us pursued it now.

... But that's why we bind mount.  Sorry, I've still not dived deeper,
into the code, but something here isn't making sense to me.  The
unprivileged user cannot create the /dev/.lxc/user-whatever directory to
which he was trying to symlink, so what should create that?  I assume
that in fact you want the persistent container-/dev for an unpriv user
to actually come from ~/.local/share/lxc/container/root.dev ?  Then,
yes, inside there when starting the container, for any file which exists
under ~/.local/share/lxc/container/root.dev lxc should bind-mount from
the host's /dev.

-serge