[lxc-users] unable to start containers ("Permission denied - Failed to mount")

[Tomasz Chmielewski]

It just broke for me on two servers again, more or less at the same 
time:

root at backup01 ~ # ls -l /data/lxd
total 0
drwx------ 1 root root 198 Jan 24 03:34 containers
(...)

Both servers are running Ubuntu 18.04 with LXD from snap:

lxd   3.9        9919  stable    canonical✓  -

And storage on a btrfs device:

root at lxd05 ~ # lxc storage list
+---------+-------------+--------+-----------+---------+
|  NAME   | DESCRIPTION | DRIVER |  SOURCE   | USED BY |
+---------+-------------+--------+-----------+---------+
| default |             | btrfs  | /data/lxd | 16      |
+---------+-------------+--------+-----------+---------+


root at backup01 ~ # lxc storage list
+---------+-------------+--------+-----------+---------+
|  NAME   | DESCRIPTION | DRIVER |  SOURCE   | USED BY |
+---------+-------------+--------+-----------+---------+
| default |             | btrfs  | /data/lxd | 44      |
+---------+-------------+--------+-----------+---------+


Not sure what's causing, but it's yet another time I'm seeing it.


Tomasz



On 2018-09-24 22:43, Christian Brauner wrote:
> On Mon, Sep 24, 2018 at 03:40:57PM +0200, Tomasz Chmielewski wrote:
>> Turns out something changed the permissions on "containers" directory:
> 
> Odd, the new storage snapshot api performs an on-disk upgrade but it
> shouldn't touch the containers directory...
> //cc Stéphane
> 
> Christian
> 
>> 
>> 
>> # lxc storage list
>> +---------+-------------+--------+-----------+---------+
>> |  NAME   | DESCRIPTION | DRIVER |  SOURCE   | USED BY |
>> +---------+-------------+--------+-----------+---------+
>> | default |             | btrfs  | /data/lxd | 12      |
>> +---------+-------------+--------+-----------+---------+
>> 
>> 
>> # ls -l /data/lxd
>> total 0
>> drwxr-xr-x 1 root root  90 Sep 24 13:05 archives
>> drwx------ 1 root root 518 Sep 24 13:12 containers <--------- here
>> drwx--x--x 1 root root   0 Mar 28 16:14 custom
>> drwx------ 1 root root   0 Sep 21 06:05 images
>> drwx------ 1 root root   0 Sep 24 05:48 snapshots
>> 
>> 
>> This fixed it:
>> 
>> chmod 711 /data/lxd/containers/
>> 
>> 
>> I'm 99% sure we did not change the permissions on that directory...
>> 
>> 
>> Tomasz
>> 
>> 
>> 
>> On 2018-09-24 15:32, Tomasz Chmielewski wrote:
>> > I'm not able to start any container today.
>> >
>> > # lxc start preprod-app
>> > Error: Failed to run: /snap/lxd/current/bin/lxd forkstart preprod-app
>> > /var/snap/lxd/common/lxd/containers
>> > /var/snap/lxd/common/lxd/logs/preprod-app/lxc.conf:
>> > Try `lxc info --show-log preprod-app` for more info
>> >
>> >
>> > # lxc info --show-log preprod-app
>> > Name: preprod-app
>> > Remote: unix://
>> > Architecture: x86_64
>> > Created: 2018/09/05 15:01 UTC
>> > Status: Stopped
>> > Type: persistent
>> > Profiles: default
>> >
>> > Log:
>> >
>> > lxc preprod-app 20180924132438.883 WARN     conf -
>> > conf.c:lxc_map_ids:2917 - newuidmap binary is missing
>> > lxc preprod-app 20180924132438.883 WARN     conf -
>> > conf.c:lxc_map_ids:2923 - newgidmap binary is missing
>> > lxc preprod-app 20180924132438.887 WARN     conf -
>> > conf.c:lxc_map_ids:2917 - newuidmap binary is missing
>> > lxc preprod-app 20180924132438.887 WARN     conf -
>> > conf.c:lxc_map_ids:2923 - newgidmap binary is missing
>> > lxc preprod-app 20180924132438.917 ERROR    dir -
>> > storage/dir.c:dir_mount:195 - Permission denied - Failed to mount
>> > "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" on
>> > "/var/snap/lxd/common/lxc/"
>> > lxc preprod-app 20180924132438.917 ERROR    conf -
>> > conf.c:lxc_mount_rootfs:1337 - Failed to mount rootfs
>> > "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" onto
>> > "/var/snap/lxd/common/lxc/" with options "(null)"
>> > lxc preprod-app 20180924132438.917 ERROR    conf -
>> > conf.c:lxc_setup_rootfs_prepare_root:3446 - Failed to setup rootfs for
>> > lxc preprod-app 20180924132438.917 ERROR    conf -
>> > conf.c:lxc_setup:3510 - Failed to setup rootfs
>> > lxc preprod-app 20180924132438.917 ERROR    start -
>> > start.c:do_start:1234 - Failed to setup container "preprod-app"
>> > lxc preprod-app 20180924132438.918 ERROR    sync -
>> > sync.c:__sync_wait:59 - An error occurred in another process (expected
>> > sequence number 5)
>> > lxc preprod-app 20180924132439.235 ERROR    start -
>> > start.c:__lxc_start:1910 - Failed to spawn container "preprod-app"
>> > lxc preprod-app 20180924132439.235 ERROR    lxccontainer -
>> > lxccontainer.c:wait_on_daemonized_start:840 - Received container state
>> > "ABORTING" instead of "RUNNING"
>> > lxc preprod-app 20180924132439.963 WARN     conf -
>> > conf.c:lxc_map_ids:2917 - newuidmap binary is missing
>> > lxc preprod-app 20180924132439.101 WARN     conf -
>> > conf.c:lxc_map_ids:2923 - newgidmap binary is missing
>> > lxc 20180924132439.380 WARN     commands -
>> > commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to
>> > receive response for command "get_state"
>> >
>> >
>> > # snap list
>> > Name  Version  Rev   Tracking  Publisher   Notes
>> > core  16-2.35  5328  stable    canonical✓  core
>> > lxd   3.5      8774  stable    canonical✓  -
>> >
>> >
>> > This is on Ubuntu 18.04.