[lxc-users] unable to start containers ("Permission denied - Failed to mount")

[Christian Brauner]

On Mon, Sep 24, 2018 at 03:40:57PM +0200, Tomasz Chmielewski wrote:
> Turns out something changed the permissions on "containers" directory:

Odd, the new storage snapshot api performs an on-disk upgrade but it
shouldn't touch the containers directory...
//cc Stéphane

Christian

> 
> 
> # lxc storage list
> +---------+-------------+--------+-----------+---------+
> |  NAME   | DESCRIPTION | DRIVER |  SOURCE   | USED BY |
> +---------+-------------+--------+-----------+---------+
> | default |             | btrfs  | /data/lxd | 12      |
> +---------+-------------+--------+-----------+---------+
> 
> 
> # ls -l /data/lxd
> total 0
> drwxr-xr-x 1 root root  90 Sep 24 13:05 archives
> drwx------ 1 root root 518 Sep 24 13:12 containers <--------- here
> drwx--x--x 1 root root   0 Mar 28 16:14 custom
> drwx------ 1 root root   0 Sep 21 06:05 images
> drwx------ 1 root root   0 Sep 24 05:48 snapshots
> 
> 
> This fixed it:
> 
> chmod 711 /data/lxd/containers/
> 
> 
> I'm 99% sure we did not change the permissions on that directory...
> 
> 
> Tomasz
> 
> 
> 
> On 2018-09-24 15:32, Tomasz Chmielewski wrote:
> > I'm not able to start any container today.
> > 
> > # lxc start preprod-app
> > Error: Failed to run: /snap/lxd/current/bin/lxd forkstart preprod-app
> > /var/snap/lxd/common/lxd/containers
> > /var/snap/lxd/common/lxd/logs/preprod-app/lxc.conf:
> > Try `lxc info --show-log preprod-app` for more info
> > 
> > 
> > # lxc info --show-log preprod-app
> > Name: preprod-app
> > Remote: unix://
> > Architecture: x86_64
> > Created: 2018/09/05 15:01 UTC
> > Status: Stopped
> > Type: persistent
> > Profiles: default
> > 
> > Log:
> > 
> > lxc preprod-app 20180924132438.883 WARN     conf -
> > conf.c:lxc_map_ids:2917 - newuidmap binary is missing
> > lxc preprod-app 20180924132438.883 WARN     conf -
> > conf.c:lxc_map_ids:2923 - newgidmap binary is missing
> > lxc preprod-app 20180924132438.887 WARN     conf -
> > conf.c:lxc_map_ids:2917 - newuidmap binary is missing
> > lxc preprod-app 20180924132438.887 WARN     conf -
> > conf.c:lxc_map_ids:2923 - newgidmap binary is missing
> > lxc preprod-app 20180924132438.917 ERROR    dir -
> > storage/dir.c:dir_mount:195 - Permission denied - Failed to mount
> > "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" on
> > "/var/snap/lxd/common/lxc/"
> > lxc preprod-app 20180924132438.917 ERROR    conf -
> > conf.c:lxc_mount_rootfs:1337 - Failed to mount rootfs
> > "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" onto
> > "/var/snap/lxd/common/lxc/" with options "(null)"
> > lxc preprod-app 20180924132438.917 ERROR    conf -
> > conf.c:lxc_setup_rootfs_prepare_root:3446 - Failed to setup rootfs for
> > lxc preprod-app 20180924132438.917 ERROR    conf -
> > conf.c:lxc_setup:3510 - Failed to setup rootfs
> > lxc preprod-app 20180924132438.917 ERROR    start -
> > start.c:do_start:1234 - Failed to setup container "preprod-app"
> > lxc preprod-app 20180924132438.918 ERROR    sync -
> > sync.c:__sync_wait:59 - An error occurred in another process (expected
> > sequence number 5)
> > lxc preprod-app 20180924132439.235 ERROR    start -
> > start.c:__lxc_start:1910 - Failed to spawn container "preprod-app"
> > lxc preprod-app 20180924132439.235 ERROR    lxccontainer -
> > lxccontainer.c:wait_on_daemonized_start:840 - Received container state
> > "ABORTING" instead of "RUNNING"
> > lxc preprod-app 20180924132439.963 WARN     conf -
> > conf.c:lxc_map_ids:2917 - newuidmap binary is missing
> > lxc preprod-app 20180924132439.101 WARN     conf -
> > conf.c:lxc_map_ids:2923 - newgidmap binary is missing
> > lxc 20180924132439.380 WARN     commands -
> > commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to
> > receive response for command "get_state"
> > 
> > 
> > # snap list
> > Name  Version  Rev   Tracking  Publisher   Notes
> > core  16-2.35  5328  stable    canonical✓  core
> > lxd   3.5      8774  stable    canonical✓  -
> > 
> > 
> > This is on Ubuntu 18.04.
> > 
> > 
> > Tomasz Chmielewski
> > https://lxadm.com
> > _______________________________________________
> > lxc-users mailing list
> > lxc-users at lists.linuxcontainers.org
> > http://lists.linuxcontainers.org/listinfo/lxc-users
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users