[lxc-users] Something changed between 1.1.2 and 1.1.4 for unprivileged containers?

[Fajar A. Nugraha]

On Fri, Oct 16, 2015 at 6:17 PM, Dirk Geschke <dirk at lug-erding.de> wrote:
> Hi Fajar,
>
>> > Yes, it exists. But I think it is only used, if the containers are
>> > started by the user root?
>>
>> So you see, first it mounts the rootfs, then other supporting files/dirs.
>>
>> My guess is you somehow haven't mounted /sys and /dev on the container
>> (which lxc should do with default config), so that deeper mounts (e.g.
>> "/sys/fs/fuse/connections", "/dev/tty") fails.
>>
>> Do you use custom lxc config file? Does it work if you create a NEW
>> container using the download template?
>
> that's a good hint, but I use the default debian settings and a few
> bind mounts.
>
> But I tried to start it with -F and this works???
>
> Strange, I restarted it again without -F and it fails. Then I added
> -F again and it works (beside some minor issues like no direct root
> login possible on console).
>
> Hmm, what's the difference here? Is there a must for a console in
> ordert to start?
>
> Do you have an idea, what's going wrong?


Well, for one thing, listing the relevant part of your setup should
help. That includes what your container config looks like, including
the "few bind mounts".

Another thing, actually doing the suggestions and reporting the result
will also help. I still haven't seen whether you can successfully run
a NEW unpriv container that was created AFTER you install lxc-1.1.4.

FWIW, I just tested this combo:
- debian jessie
- cgmanager 0.39-2ubuntu2~ubuntu15.04.1~ppa1
- lxc (and friends, like liblxc1) 1.1.4-0ubuntu1~ubuntu15.04.1~ppa1
- lxcfs 0.10-0ubuntu2~ubuntu15.04.1~ppa1

cgmanager, lxc, and lxcfs were rebuilt from sources on
https://launchpad.net/~ubuntu-lxc/+archive/ubuntu/lxc-stable. I was
actually surprised that most of them can be rebuilt cleanly on jessie
(with the exception of lxc, needed to change dep from llibgnutls-dev
to llibgnutls28-dev). Most dependencies (e.g. apparmor, python) were
automatically provided by jessie's repo. "uidmap" was installed
manually because it's not listed as dependency, but needed to run
unpriv containers.

Customizations:
# cat /etc/sysctl.d/50-lxc.conf
# enable unpriv container
kernel.unprivileged_userns_clone = 1

# cat /etc/lxc/lxc-usernet
# USERNAME TYPE BRIDGE COUNT
user    veth    lxcbr0  10

# cat /home/user/.config/lxc/lxc.conf
lxc.lxcpath = /home/user/lxc
lxc.default_config = /home/user/lxc/lxc.default.conf

# cat /home/user/lxc/lxc.default.conf
lxc.include = /etc/lxc/default.conf
lxc.id_map = u 0 624288 65536
lxc.id_map = g 0 624288 65536
lxc.aa_allow_incomplete = 1


## as root, create cgroup for "user".
## Needed because debian still use old systemd without ubuntu
pam-systemd patches.
# cgm movepidabs all / $$
# cgm create all /users/user
# cgm chown all /users/user $(id -u user) $(id -g user)

## as "user", move pid to the newly-created cgroup
## Needed because debian still use old systemd without ubuntu
pam-systemd patches.
$ cgm movepidabs all /users/user $$
$ cat /proc/self/cgroup
8:perf_event:/users/user
7:blkio:/users/user
6:net_cls,net_prio:/users/user
5:freezer:/users/user
4:devices:/users/user
3:cpu,cpuacct:/users/user
2:cpuset:/users/user
1:name=systemd:/users/user

## as "user", create and start the container, and see boot progress by
attaching to console "0"
## cleanup leftover cgroups first (e.g. from old containers)
$ cgm prune all
$ lxc-create  -n jessie -t download -- -d debian -r jessie -a amd64
$ lxc-start -n jessie;lxc-console -n jessie -t 0


There's some delay on the container start, but after some time systemd
runs all the way to display login prompt. Tested login and apt-get
update, works. There are some problems (e.g. if I install dbus, then I
can't login anymore) due to old systemd version on the container. Note
that it works fine if I use sysvinit-core instead of systemd inside
the container.

>From INSIDE the container:

# cat /proc/self/uid_map
         0     624288      65536
# cat /proc/self/cgroup
8:perf_event:/users/user/lxc/jessie
7:blkio:/users/user/lxc/jessie
6:net_cls,net_prio:/users/user/lxc/jessie
5:freezer:/users/user/lxc/jessie
4:devices:/users/user/lxc/jessie
3:cpu,cpuacct:/users/user/lxc/jessie
2:cpuset:/users/user/lxc/jessie
1:name=systemd:/users/user/lxc/jessie/system.slice/console-getty.service


So you see, unpriv containers can start just fine with lxc-1.1.4 on
debian (with some workarounds). I can't comment more on your setup
because I don't know what your enviroment looks like.

-- 
Fajar