[lxc-users] Unable to Start Unprivileged Containers on Debian / Jessie

[Michael H. Warfield]

On Wed, 2014-10-01 at 15:05 +0000, Serge Hallyn wrote:
> Quoting Michael H. Warfield (mhw at WittsEnd.com):
> > On Wed, 2014-10-01 at 13:53 +0000, Serge Hallyn wrote:
> > > Quoting Michael H. Warfield (mhw at WittsEnd.com):
> > > > On Tue, 2014-09-30 at 19:39 -0400, Michael H. Warfield wrote:
> > > > > On Tue, 2014-09-30 at 23:56 +0100, Chris wrote:
> > > > > > On 30/09/14 19:28, Michael H. Warfield wrote:
> > > > > > >> I haven't looked a whole lot into the premade containers, my gut feeling
> > > > > > >> was that I didn't want to download a whole operating system from this
> > > > > > >> project, and that I'd be a lot more comfortable taking distribution that
> > > > > > >> I trust, and making the template manually. This way I know everything
> > > > > > >> extra that's going into it.
> > > > > > > Our templates are pretty barebones.  Very minimal.  You'll have to add
> > > > > > > just about anything you would really want to make a useful container.
> > > > > > I should definitely take a closer look sometime.
> > > > > > >> It's running Debian Jessie. LXC 1.0.5-3 from package management. And
> > > > > > >> systemd 208-8 also from package management.
> > > > > > > OK... THAT explains a LOT!  That systemd option is why you're running
> > > > > > > into this problem and you're about to have far worse.
> > > > > > >
> > > > > > >> I took a config from an existing container and modified it for what I
> > > > > > >> thought would work for an unprivileged container. I've attached the
> > > > > > >> config for osmium. I've also attached the latest trace output from the
> > > > > > >> lxc-start, as I've fixed a few slight errors in the config since then.
> > > > > > > You're going to have to make some additional changes...  Make sure you
> > > > > > > add "lxc.kmsg = 0" to your container or systemd.journald is going to eat
> > > > > > > your CPU time for lunch (and be sure to flush
> > > > > > > your /dev/.lxc/user/osmium* directory).  There's also some adjustments
> > > > > > > that need to be made for mgetty consoles and such.  You also need to
> > > > > > > link the shutdown unit to the SIGPWR service to allow lxc to shut the
> > > > > > > container down gracefully.  You might take a look at the Oracle or
> > > > > > > Fedora templates for some guidance there.
> > > > > > Will definitely come back to this once it starts up, thank you for the 
> > > > > > advice.
> > > > > > >> osmium at cadmium:~$ find /dev/.lxc/user -ls
> > > > > > >>     9668    0 drwxrwxrwt   3 root     root           60 Sep 30 15:38
> > > > > > >> /dev/.lxc/user
> > > > > > >>    11109    0 drwxr-xr-x   3 427680   427680         60 Sep 30 15:38
> > > > > > >> /dev/.lxc/user/osmium.3c68b3f0c5eeec7d
> > > > > > >>    11110    0 drwxr-xr-x   2 427680   427680         40 Sep 30 15:38
> > > > > > >> /dev/.lxc/user/osmium.3c68b3f0c5eeec7d/pts
> > > > > > > Bingo!
> > > > > > >
> > > > > > > Ok...  So it appears that lxc-start did manage to create your dev
> > > > > > > directory properly under the host /dev/.lxc/user.
> > > > > > >
> > > > > > > Now I see the real problem...
> > > > > > >
> > > > > > > The same code that creates that directory creates the symlink
> > > > > > > in /home/osmium/.local/share/lxc/osmium.  But, the /dev/ directory is
> > > > > > > owned by "427680:427680" while the directory containing the symlink is
> > > > > > > own by "osmium:osmium" and you then have a permission denied because
> > > > > > > 427680:427680 doesn't have write permissions
> > > > > > > to /home/osmium/.local/share/lxc/osmium.
> > > > > > >
> > > > > > > That's a (the!) problem.  I'm just not sure if chown/chgrp is the
> > > > > > > correct answer or if you need to add some group membership and add group
> > > > > > > write permissions with appropriate host auth secondary groups.  Either
> > > > > > > way, it's that permission problem that biting you in the rear end.
> > > > > > >
> > > > > > OK, yes. This was that problem. Fixing it has progressed startup a 
> > > > > > little further. It didn't like the lxc.mount.entry for devpts, so I 
> > > > > > threw that out for the time being also. Now it's still stuck at 
> > > > > > 'populating dev' though. I've attached the latest trace in case you help 
> > > > > > me again.
> > > > > > 
> > > > > > osmium at cadmium:~$ lxc-start -n osmium -l trace -o /tmp/xxx7
> > > > > > lxc-start: Operation not permitted - Error creating null
> > > > 
> > > > > Ouch.  Looks like it's attempting to do a mknod which is not allowed and
> > > > > can not be allowed as it's prohibited in the kernel to any non-priv
> > > > > users.  That's a security issue that can not be changed.  That's going
> > > > > to be a problem for any autodev containers running under non-priv users.
> > > > > Serge and/or Stéphane will have to respond to that one.
> > > > 
> > > > > That was actually a case I was considering when I was discussing the
> > > > > whole devtmpfs thing with Serge back at LinuxPlumbers last year.
> > > > > Non-priv users can not create devices (mknod), period, end of
> > > > > discussion.  But they might be able to create hard links to them within
> > > > > devtmpfs or do bind mounts, which might work under the current setup.
> > > > > That was the intent at least.  I don't recall the hard link device code
> > > > > being done (though we do bind mounts on some devices) so I'm not sure
> > > > > what's transpiring at this point in the code.
> > > > 
> > > > Maybe I did test that out last year.  IAC...  Just retested it.  Damn
> > > > it, oh yeah...  Hardlinks to device nodes in devtmpfs are prohibited to
> > > > non-priv users even when permissions would permit.  That option is also
> > > > non viable.  Nice idea.  Guess I know why none of us pursued it now.
> > 
> > > ... But that's why we bind mount.  Sorry, I've still not dived deeper,
> > > into the code, but something here isn't making sense to me.  The
> > > unprivileged user cannot create the /dev/.lxc/user-whatever directory to
> > > which he was trying to symlink, so what should create that?
> > 
> > Actually, he can.  /dev/.lxc/user is set up mode 1777 much like /tmp
> > specifically for that purpose.  So an unpriv user gets a

> Ah, that's what I was trying to ask the other day.  So what does that?

> > directory /dev/.lxc/user/{container-hash} owned by him and restricted to
> > him while a priv user gets /dev/.lxc/{container-hash}.  That's done in
> > the startup helper script that set up the parent /dev/.lxc directory.

> This is a host init script?

More or less, yes.  It's lxc-devsetup in /usr/libexec/lxc for systemd
systems and it's a PreExec script for the lxc-containers.service unit.
Since it's only really required for autodev and, todate, we've only been
using autodev for systemd cases (which I understand is about to change)
it was only required in the systemd case.  That may need to change if
we're going to be using autodev with upstart or sysvinit.

> > > I assume
> > > that in fact you want the persistent container-/dev for an unpriv user
> > > to actually come from ~/.local/share/lxc/container/root.dev ?  Then,
> > > yes, inside there when starting the container, for any file which exists
> > > under ~/.local/share/lxc/container/root.dev lxc should bind-mount from
> > > the host's /dev.
> > 
> > I don't think the autodev populating code currently does bind mounts.
> > If not, it probably should.  Probably should look into that.  :-P

> Yeah it should be very simple, creat + mount in setup_autodev.

Pretty much what I figure.  Looks like I have a few tweeks to my latest
patch for kmsg with autodev as well (which can not have that symlink
in /dev for systemd or journald wigs out on the CPU).  Looks like
Stephane wasn't real thrilled with my earlier patch and the kmsg stuff
in particular, but it's got to be in there for runtime changes.

Regards,
Mike
-- 
Michael H. Warfield (AI4NB) | (770) 978-7061 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 465 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20141001/910c79be/attachment.sig>