[lxc-users] Unable to Start Unprivileged Containers on Debian / Jessie

[Serge Hallyn]

Quoting Michael H. Warfield (mhw at WittsEnd.com):
> On Wed, 2014-10-01 at 13:53 +0000, Serge Hallyn wrote:
> > Quoting Michael H. Warfield (mhw at WittsEnd.com):
> > > On Tue, 2014-09-30 at 19:39 -0400, Michael H. Warfield wrote:
> > > > On Tue, 2014-09-30 at 23:56 +0100, Chris wrote:
> > > > > On 30/09/14 19:28, Michael H. Warfield wrote:
> > > > > >> I haven't looked a whole lot into the premade containers, my gut feeling
> > > > > >> was that I didn't want to download a whole operating system from this
> > > > > >> project, and that I'd be a lot more comfortable taking distribution that
> > > > > >> I trust, and making the template manually. This way I know everything
> > > > > >> extra that's going into it.
> > > > > > Our templates are pretty barebones.  Very minimal.  You'll have to add
> > > > > > just about anything you would really want to make a useful container.
> > > > > I should definitely take a closer look sometime.
> > > > > >> It's running Debian Jessie. LXC 1.0.5-3 from package management. And
> > > > > >> systemd 208-8 also from package management.
> > > > > > OK... THAT explains a LOT!  That systemd option is why you're running
> > > > > > into this problem and you're about to have far worse.
> > > > > >
> > > > > >> I took a config from an existing container and modified it for what I
> > > > > >> thought would work for an unprivileged container. I've attached the
> > > > > >> config for osmium. I've also attached the latest trace output from the
> > > > > >> lxc-start, as I've fixed a few slight errors in the config since then.
> > > > > > You're going to have to make some additional changes...  Make sure you
> > > > > > add "lxc.kmsg = 0" to your container or systemd.journald is going to eat
> > > > > > your CPU time for lunch (and be sure to flush
> > > > > > your /dev/.lxc/user/osmium* directory).  There's also some adjustments
> > > > > > that need to be made for mgetty consoles and such.  You also need to
> > > > > > link the shutdown unit to the SIGPWR service to allow lxc to shut the
> > > > > > container down gracefully.  You might take a look at the Oracle or
> > > > > > Fedora templates for some guidance there.
> > > > > Will definitely come back to this once it starts up, thank you for the 
> > > > > advice.
> > > > > >> osmium at cadmium:~$ find /dev/.lxc/user -ls
> > > > > >>     9668    0 drwxrwxrwt   3 root     root           60 Sep 30 15:38
> > > > > >> /dev/.lxc/user
> > > > > >>    11109    0 drwxr-xr-x   3 427680   427680         60 Sep 30 15:38
> > > > > >> /dev/.lxc/user/osmium.3c68b3f0c5eeec7d
> > > > > >>    11110    0 drwxr-xr-x   2 427680   427680         40 Sep 30 15:38
> > > > > >> /dev/.lxc/user/osmium.3c68b3f0c5eeec7d/pts
> > > > > > Bingo!
> > > > > >
> > > > > > Ok...  So it appears that lxc-start did manage to create your dev
> > > > > > directory properly under the host /dev/.lxc/user.
> > > > > >
> > > > > > Now I see the real problem...
> > > > > >
> > > > > > The same code that creates that directory creates the symlink
> > > > > > in /home/osmium/.local/share/lxc/osmium.  But, the /dev/ directory is
> > > > > > owned by "427680:427680" while the directory containing the symlink is
> > > > > > own by "osmium:osmium" and you then have a permission denied because
> > > > > > 427680:427680 doesn't have write permissions
> > > > > > to /home/osmium/.local/share/lxc/osmium.
> > > > > >
> > > > > > That's a (the!) problem.  I'm just not sure if chown/chgrp is the
> > > > > > correct answer or if you need to add some group membership and add group
> > > > > > write permissions with appropriate host auth secondary groups.  Either
> > > > > > way, it's that permission problem that biting you in the rear end.
> > > > > >
> > > > > OK, yes. This was that problem. Fixing it has progressed startup a 
> > > > > little further. It didn't like the lxc.mount.entry for devpts, so I 
> > > > > threw that out for the time being also. Now it's still stuck at 
> > > > > 'populating dev' though. I've attached the latest trace in case you help 
> > > > > me again.
> > > > > 
> > > > > osmium at cadmium:~$ lxc-start -n osmium -l trace -o /tmp/xxx7
> > > > > lxc-start: Operation not permitted - Error creating null
> > > 
> > > > Ouch.  Looks like it's attempting to do a mknod which is not allowed and
> > > > can not be allowed as it's prohibited in the kernel to any non-priv
> > > > users.  That's a security issue that can not be changed.  That's going
> > > > to be a problem for any autodev containers running under non-priv users.
> > > > Serge and/or Stéphane will have to respond to that one.
> > > 
> > > > That was actually a case I was considering when I was discussing the
> > > > whole devtmpfs thing with Serge back at LinuxPlumbers last year.
> > > > Non-priv users can not create devices (mknod), period, end of
> > > > discussion.  But they might be able to create hard links to them within
> > > > devtmpfs or do bind mounts, which might work under the current setup.
> > > > That was the intent at least.  I don't recall the hard link device code
> > > > being done (though we do bind mounts on some devices) so I'm not sure
> > > > what's transpiring at this point in the code.
> > > 
> > > Maybe I did test that out last year.  IAC...  Just retested it.  Damn
> > > it, oh yeah...  Hardlinks to device nodes in devtmpfs are prohibited to
> > > non-priv users even when permissions would permit.  That option is also
> > > non viable.  Nice idea.  Guess I know why none of us pursued it now.
> 
> > ... But that's why we bind mount.  Sorry, I've still not dived deeper,
> > into the code, but something here isn't making sense to me.  The
> > unprivileged user cannot create the /dev/.lxc/user-whatever directory to
> > which he was trying to symlink, so what should create that?
> 
> Actually, he can.  /dev/.lxc/user is set up mode 1777 much like /tmp
> specifically for that purpose.  So an unpriv user gets a

Ah, that's what I was trying to ask the other day.  So what does that?

> directory /dev/.lxc/user/{container-hash} owned by him and restricted to
> him while a priv user gets /dev/.lxc/{container-hash}.  That's done in
> the startup helper script that set up the parent /dev/.lxc directory.

This is a host init script?

> > I assume
> > that in fact you want the persistent container-/dev for an unpriv user
> > to actually come from ~/.local/share/lxc/container/root.dev ?  Then,
> > yes, inside there when starting the container, for any file which exists
> > under ~/.local/share/lxc/container/root.dev lxc should bind-mount from
> > the host's /dev.
> 
> I don't think the autodev populating code currently does bind mounts.
> If not, it probably should.  Probably should look into that.  :-P

Yeah it should be very simple, creat + mount in setup_autodev.