[lxc-users] Migrated to 2.9 - Catch 22 with Containers w/o Storage Pools

Sun Feb 19 10:05:08 UTC 2017

On 02/19/2017 08:06 AM, Stéphane Graber wrote:
> On Sun, Feb 19, 2017 at 12:40:19AM -0600, Neil Bowers wrote:
>> Since migrating to 2.9 I've found myself in a catch-22 situation where
>> somehow I ended up with a couple of containers that were not assigned to a
>> storage pool upon creation.
>>
>> As a side note, there should be an option to set a default storage pool
>> value to be used with the 'lxc launch' when the -s parameter isn't
>> specified - would've saved me a lot of time.
>>
>> Consider the following:
>>
>> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
>> |       NAME       |  STATE  |         IPV4         |      CREATED AT
>>   |     LAST USED AT     | STORAGE POOL |  PROFILES   |
>> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
>> | free-ray         | ERROR   |                      |
>>   |                      |              |             |
>> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
>> | intimate-civet   | RUNNING | 192.168.1.245 (eth0) | 2017/02/19 05:24 UTC
>> | 2017/02/19 05:24 UTC | lxc-vg       | default     |
>> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
>> | minecraft-msm    | STOPPED |                      | 2017/02/19 00:33 UTC
>> |                      | vmpool       | default-msm |
>> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
>> | topical-antelope | ERROR   |                      |
>>   |                      |              |             |
>> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
>> | warm-bluejay     | ERROR   |                      |
>>   |                      |              |             |
>> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
>>
>> The containers free-ray, tropical-antelope and warm-bluejay were created
>> without the use of the -s parameter, so therefore no storage pool was
>> assigned.
>>
>> Upon trying to delete these containers:
>>
>> └─▪ lxc delete free-ray
>> error: No storage pool specified.
>>
>> Not even the --force option works:
>>
>> lxc delete free-ray --force
>> error: No storage pool specified.
>>
>> After a lot of digging with the one successful container, it appears that,
>> starting with 2.9, every container must have a storage volume of the type
>> 'container' with the same name as the container itself.  So I thought,
>> maybe I could create the storage volume and then attach it to the
>> container, and then be able to delete it - wrong:
>>
>>   lxc storage volume create lxc-vg container/free-ray
>> error: Currently not allowed to create storage volumes of type container.
>>
>> Currently the only volume type that we're allowed to create is 'custom' -
>> so I gave that a try:
>>
>> lxc storage volume create lxc-vg free-ray
>> Storage volume free-ray created
>>
>> But then when I attempt to attach this new volume to 'free-ray', I once
>> again encounter:
>>
>> lxc storage volume attach lxc-vg free-ray free-ray root /
>> error: No storage pool specified.
>>
>> And now I'm stuck - I have three containers that can't be deleted because
>> they're not tied to a storage pool and no way to tie them to one, and a
>> fourth container that is tied to a now-deleted storage pool that can't be
>> deleted either.
>>
>> Short of blowing away the LXD/LXC install and starting over, is there any
>> way to fix this?
> Hey there,
>
> There are a bunch of different issues we're tracking at the moment
> around the storage work in LXD 2.9.
>
> For better tracking, a Github issue would be preferred for those.
>
>
> You don't have to pass "-s" for the container to get storage associated
> with it and -s wasn't a valid option until LXD 2.9. I suspect your
> broken containers were created before the upgrade to LXD 2.9.
>
> With LXD 2.9, the container root must come from a storage pool. Either
> directly attached to a particular container as is the case when -s is
> passed, or inherited through a profile.
>
>
> The upgrade code should have updated your default profile so that this
> all works, making new container creation work and in theory should have
> also made all your existing containers keep working.
>
> That last part is what's quite a bit buggy now and Christian and I are
> actively tracking and fixing those issues as quickly as we can
> (considering it's the weekend) and will be releasing a LXD 2.9.2 as soon
> as we've got a good set of fixes.
>
>
> As you pointed out, another bug that's present in 2.9.1 is that it's
> virtualy impossible to modify or even read anything from a broken
> container, making fixing such a container, near impossible.
>
>
>
> Your best bet to get back to working order is:
>   - Run "lxc profile show default" and confirm that you see a root device
>     with a pool property listed there.
>   - If that's the case, then chances are your broken containers don't
>     have the default profile attached to them.
>   - Since you can't change them to fix that by either adding the default
>     profile or adding a valid root device to them, we'll need to do some DB
>     surgery:
>      - Install "sqlite3" if you don't have it already
>      - Run: sqlite3 /var/lib/lxd/lxd.db 'SELECT id FROM profiles WHERE name="default"';
>      - For each broken container, do:
>        - Run: sqlite3 /var/lib/lxd/lxd.db 'SELECT id FROM containers WHERE name="NAME"';
>        - Run: sqlite3 /var/lib/lxd/lxd.db 'INSERT INTO containers_profiles (container_id, profile_id, apply_order) VALUES (PROFILE-ID, CONTAINER-ID, 1);'
>
> Here's an example I reproduced on one of my systems:
>
> ```
> root at athos:~# lxc info samba-fs01
> error: No storage pool specified.
>
> root at athos:~# lxc profile show default
> config: {}
> description: ""
> devices:
>    eth0:
>      nictype: bridged
>      parent: lxdbr0
>      type: nic
>    root:
>      path: /
>      pool: default
>      type: disk
> name: default
>
> root at athos:~# sqlite3 /var/lib/lxd/lxd.db 'SELECT id FROM profiles WHERE name="default"';
> 1
>
> root at athos:~# sqlite3 /var/lib/lxd/lxd.db 'SELECT id FROM containers WHERE name="samba-fs01"';
> 5
>
> root at athos:~# sqlite3 /var/lib/lxd/lxd.db 'INSERT INTO containers_profiles (container_id, profile_id, apply_order) VALUES (5, 1, 1);'
>
> root at athos:~# lxc info samba-fs01
> Name: samba-fs01
> Remote: unix:/var/lib/lxd/unix.socket
> Architecture: x86_64
> Status: Running
> Type: persistent
> Profiles: default
> Pid: 25679
> Ips:
>    eth0: inet6   2607:f2c0:f00f:2720:216:3eff:fe13:1f33  vethX1NNLH
>    eth0: inet6   fe80::216:3eff:fe13:1f33    vethX1NNLH
>    lo:   inet    127.0.0.1
>    lo:   inet6   ::1
> Resources:
>    Processes: 26
>    Disk usage:
>      root: 510.17MB
>    CPU usage:
>      CPU usage (in seconds): 27615
>    Memory usage:
>      Memory (current): 124.02MB
>      Memory (peak): 232.74MB
>    Network usage:
>      lo:
>        Bytes received: 478.36kB
>        Bytes sent: 478.36kB
>        Packets received: 8164
>        Packets sent: 8164
>      eth0:
>        Bytes received: 210.80GB
>        Bytes sent: 8.67GB
>        Packets received: 107995716
>        Packets sent: 85171306
> ```
>
> Hope this works for you too.

I just want to indicate, that I experience the same on one server and 
not on a very similar one. Strange.

tamas