[lxc-users] Migrated to 2.9 - Catch 22 with Containers w/o Storage Pools
Stéphane Graber
stgraber at ubuntu.com
Sun Feb 19 07:06:28 UTC 2017
On Sun, Feb 19, 2017 at 12:40:19AM -0600, Neil Bowers wrote:
> Since migrating to 2.9 I've found myself in a catch-22 situation where
> somehow I ended up with a couple of containers that were not assigned to a
> storage pool upon creation.
>
> As a side note, there should be an option to set a default storage pool
> value to be used with the 'lxc launch' when the -s parameter isn't
> specified - would've saved me a lot of time.
>
> Consider the following:
>
> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
> | NAME | STATE | IPV4 | CREATED AT
> | LAST USED AT | STORAGE POOL | PROFILES |
> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
> | free-ray | ERROR | |
> | | | |
> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
> | intimate-civet | RUNNING | 192.168.1.245 (eth0) | 2017/02/19 05:24 UTC
> | 2017/02/19 05:24 UTC | lxc-vg | default |
> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
> | minecraft-msm | STOPPED | | 2017/02/19 00:33 UTC
> | | vmpool | default-msm |
> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
> | topical-antelope | ERROR | |
> | | | |
> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
> | warm-bluejay | ERROR | |
> | | | |
> +------------------+---------+----------------------+----------------------+----------------------+--------------+-------------+
>
> The containers free-ray, tropical-antelope and warm-bluejay were created
> without the use of the -s parameter, so therefore no storage pool was
> assigned.
>
> Upon trying to delete these containers:
>
> └─▪ lxc delete free-ray
> error: No storage pool specified.
>
> Not even the --force option works:
>
> lxc delete free-ray --force
> error: No storage pool specified.
>
> After a lot of digging with the one successful container, it appears that,
> starting with 2.9, every container must have a storage volume of the type
> 'container' with the same name as the container itself. So I thought,
> maybe I could create the storage volume and then attach it to the
> container, and then be able to delete it - wrong:
>
> lxc storage volume create lxc-vg container/free-ray
> error: Currently not allowed to create storage volumes of type container.
>
> Currently the only volume type that we're allowed to create is 'custom' -
> so I gave that a try:
>
> lxc storage volume create lxc-vg free-ray
> Storage volume free-ray created
>
> But then when I attempt to attach this new volume to 'free-ray', I once
> again encounter:
>
> lxc storage volume attach lxc-vg free-ray free-ray root /
> error: No storage pool specified.
>
> And now I'm stuck - I have three containers that can't be deleted because
> they're not tied to a storage pool and no way to tie them to one, and a
> fourth container that is tied to a now-deleted storage pool that can't be
> deleted either.
>
> Short of blowing away the LXD/LXC install and starting over, is there any
> way to fix this?
Hey there,
There are a bunch of different issues we're tracking at the moment
around the storage work in LXD 2.9.
For better tracking, a Github issue would be preferred for those.
You don't have to pass "-s" for the container to get storage associated
with it and -s wasn't a valid option until LXD 2.9. I suspect your
broken containers were created before the upgrade to LXD 2.9.
With LXD 2.9, the container root must come from a storage pool. Either
directly attached to a particular container as is the case when -s is
passed, or inherited through a profile.
The upgrade code should have updated your default profile so that this
all works, making new container creation work and in theory should have
also made all your existing containers keep working.
That last part is what's quite a bit buggy now and Christian and I are
actively tracking and fixing those issues as quickly as we can
(considering it's the weekend) and will be releasing a LXD 2.9.2 as soon
as we've got a good set of fixes.
As you pointed out, another bug that's present in 2.9.1 is that it's
virtualy impossible to modify or even read anything from a broken
container, making fixing such a container, near impossible.
Your best bet to get back to working order is:
- Run "lxc profile show default" and confirm that you see a root device
with a pool property listed there.
- If that's the case, then chances are your broken containers don't
have the default profile attached to them.
- Since you can't change them to fix that by either adding the default
profile or adding a valid root device to them, we'll need to do some DB
surgery:
- Install "sqlite3" if you don't have it already
- Run: sqlite3 /var/lib/lxd/lxd.db 'SELECT id FROM profiles WHERE name="default"';
- For each broken container, do:
- Run: sqlite3 /var/lib/lxd/lxd.db 'SELECT id FROM containers WHERE name="NAME"';
- Run: sqlite3 /var/lib/lxd/lxd.db 'INSERT INTO containers_profiles (container_id, profile_id, apply_order) VALUES (PROFILE-ID, CONTAINER-ID, 1);'
Here's an example I reproduced on one of my systems:
```
root at athos:~# lxc info samba-fs01
error: No storage pool specified.
root at athos:~# lxc profile show default
config: {}
description: ""
devices:
eth0:
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
name: default
root at athos:~# sqlite3 /var/lib/lxd/lxd.db 'SELECT id FROM profiles WHERE name="default"';
1
root at athos:~# sqlite3 /var/lib/lxd/lxd.db 'SELECT id FROM containers WHERE name="samba-fs01"';
5
root at athos:~# sqlite3 /var/lib/lxd/lxd.db 'INSERT INTO containers_profiles (container_id, profile_id, apply_order) VALUES (5, 1, 1);'
root at athos:~# lxc info samba-fs01
Name: samba-fs01
Remote: unix:/var/lib/lxd/unix.socket
Architecture: x86_64
Status: Running
Type: persistent
Profiles: default
Pid: 25679
Ips:
eth0: inet6 2607:f2c0:f00f:2720:216:3eff:fe13:1f33 vethX1NNLH
eth0: inet6 fe80::216:3eff:fe13:1f33 vethX1NNLH
lo: inet 127.0.0.1
lo: inet6 ::1
Resources:
Processes: 26
Disk usage:
root: 510.17MB
CPU usage:
CPU usage (in seconds): 27615
Memory usage:
Memory (current): 124.02MB
Memory (peak): 232.74MB
Network usage:
lo:
Bytes received: 478.36kB
Bytes sent: 478.36kB
Packets received: 8164
Packets sent: 8164
eth0:
Bytes received: 210.80GB
Bytes sent: 8.67GB
Packets received: 107995716
Packets sent: 85171306
```
Hope this works for you too.
--
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20170219/39f9175d/attachment.sig>
More information about the lxc-users
mailing list