[lxc-users] lxc list fails to show containers after upgrade to lxd 0.22 / lxc 1.1.5

Sun Nov 22 04:37:03 UTC 2015

I was getting a bit desperate to recover my containers as I had invested a
lot of time configuring them.  I had backups of the containers (on ZFS with
snapshots and replicas) but that did me no good as long as lxc could not
"see" them.

It occurred to me maybe I could swap out 'dummy' containers with my real
ones.  So I tried the following, which seems to have worked - no issues
with the 'recovered' containers as yet.

I created a dummy template [template] using the same image as my valuable
containers. I used the template to create new containers [newName].

LXC could see and start the new container.  I stopped the container and
deleted it, then copied one of my existing containers in its place and
renamed it to the new name. I applied the correct profile. Then, whew!, lxc
could see and start the container and I had my container back, albeit with
a new name.  After tidying up the hostname of the container, it was good to
go. I repeated the exercise for all the containers I needed to recover.

I still have the ghost names of the containers in lxd.db but I am not
touching that.

$ lxc launch images:501a7fb84afe [template]
$ lxc stop [template]
$ lxc copy [template] [newName]
$ sudo rm -r /var/lib/lxd/containers/[newName]
$ sudo cp -R --preserve=all /backup/location/containers/[backup]
/var/lib/lxd/containers
$ sudo mv /var/lib/lxd/containers/[backup] /var/lib/lxd/containers/[newName]
$ lxc profile apply [newName] [profileName]
$ lxc start [newName]
$ lxc exec [newName] -- /bin/bash
# nano /etc/hosts
# nano /etc/hostname

Apologies to Stéphane and the LXD/LXC team for what must appear an
appalling solution.

On Thu, Nov 19, 2015 at 11:30 PM, Matthew McKennirey <mmckennirey at gmail.com>
wrote:

> Thanks for the rapid response. I rebooted the server, then tried
> again. Host is Ubuntu 15.04 fully patched, kernel 3.19.0-25 (with later
> kernels lxd will not start because of a permission issue with proc)
>
> $ sudo uname -a
> Linux zeus 3.19.0-25-generic #26-Ubuntu SMP Fri Jul 24 21:17:31 UTC 2015
> x86_64 x86_64 x86_64 GNU/Linux
>
> lxc is 1.1.5-0ubuntu3~ubuntu
> lxd is 0.22-0ubuntu2~ubuntu1
>
> I am running all client commands on the host machine
> $ lxc --version
> 0.22
> $ lxd --version
> 0.22
>
> Containers are present on the file system
> $ ls -l /var/lib/lxd/containers/
> -rwxrwxrwx 1 root   root   27854 Nov 14 23:20 lxc-monitord.log
> drwxrwxrwx 3 100000 100000     5 Feb 14  2015 ub1404_sg
> drwx--x--x 3 root   root       3 Aug 16 14:29 ubuntu14-java-utils
> drwx--x--x 3 100000 100000     3 Aug 18 23:12 ubuntu-build
> drwxr-xr-x 3 root   root       3 Oct 18 17:24 ubuntu-build-clone
> drwx--x--x 4 100000 100000     5 Aug 16 14:09 ubuntu-sg
> drwx--x--x 3 root   root       3 Aug 16 15:38 ubuntu-sg-base
> drwxr-xr-x 3 100000 100000     3 Oct 18 18:25 ubuntu-sg-tst
> drwxr-xr-x 3 100000 100000     3 Oct 18 17:30 ubuntu-tst
> drwxr-xr-x 3 root   root       3 Oct 18 17:18 ubuntu-web
>
> Containers are in lxd.db (see prior message)
>
> After the reboot I had trouble starting lxd
>
> $ sudo cat /var/log/syslog
> kernel: [ 6575.572166] init: lxd main process (5391) terminated with
> status 1
> kernel: [ 6575.572179] init: lxd main process ended, respawning
> kernel: [ 6575.588459] init: lxd main process (5396) terminated with
> status 1
> kernel: [ 6575.588474] init: lxd main process ended, respawning
> kernel: [ 6575.604184] init: lxd main process (5401) terminated with
> status 1
> kernel: [ 6575.604198] init: lxd respawning too fast, stopped
>
> $ sudo cat /var/log/upstart/lxd.log
> error: mkdir /var/lib/lxd/images: file exists
> error: mkdir /var/lib/lxd/images: file exists
> error: mkdir /var/lib/lxd/images: file exists
> error: mkdir /var/lib/lxd/images: file exists
>
> I'm really not sure why lxd was trying to create the images folder at
> start-up.
> Anyway I renamed the folder and lxd would start
>
> $ sudo mv /var/lib/lxd/images /var/lib/lxd/images-back
>
> $ sudo service lxd start --debug -v
> $ lxc list --debug
> DBUG[11-19|22:52:29] Fingering the daemon
> DBUG[11-19|22:52:29] Raw response:
> {"type":"sync","status":"Success","status_code":200,"metadata":{"api_compat":1,"auth":"trusted","config":{"images.remote_cache_expiry":"10"},"environment":{"addresses":[],"architectures":[2,1],"driver":"lxc","driver_version":"1.1.5","kernel":"Linux","kernel_architecture":"x86_64","kernel_version":"3.19.0-25-generic","server":"lxd","server_pid":5613,"server_version":"0.22","storage":"dir","storage_version":""}}}
>
> DBUG[11-19|22:52:29] Pong received
> DBUG[11-19|22:52:29] Raw response:
> {"type":"sync","status":"Success","status_code":200,"metadata":[]}
>
> +------+-------+------+------+-----------+-----------+
> | NAME | STATE | IPV4 | IPV6 | EPHEMERAL | SNAPSHOTS |
> +------+-------+------+------+-----------+-----------+
> +------+-------+------+------+-----------+-----------+
>
> Still no joy.
>
> $ sudo cat /var/log/lxd/lxd.log
> t=2015-11-19T22:45:51-0500 lvl=info msg="LXD is starting" path=/var/lib/lxd
> t=2015-11-19T22:45:51-0500 lvl=info msg="Default uid/gid map:"
> t=2015-11-19T22:45:51-0500 lvl=info msg=" - u 0 100000 65536"
> t=2015-11-19T22:45:51-0500 lvl=info msg=" - g 0 100000 65536"
> t=2015-11-19T22:45:51-0500 lvl=info msg=Init driver=storage/dir
> t=2015-11-19T22:45:51-0500 lvl=info msg="Looking for existing
> certificates" cert=/var/lib/lxd/server.crt key=/var/lib/lxd/server.key
> t=2015-11-19T22:45:51-0500 lvl=info msg="LXD isn't socket activated"
> t=2015-11-19T22:45:51-0500 lvl=info msg="REST API daemon:"
> t=2015-11-19T22:45:51-0500 lvl=info msg=" - binding socket"
> socket=/var/lib/lxd/unix.socket
> t=2015-11-19T22:46:09-0500 lvl=info msg=handling method=GET url=/1.0 ip=@
> t=2015-11-19T22:46:09-0500 lvl=info msg=handling method=GET
> url="/1.0/containers?recursion=1" ip=@
> t=2015-11-19T22:48:20-0500 lvl=info msg=handling method=GET url=/1.0 ip=@
> t=2015-11-19T22:48:20-0500 lvl=info msg=handling method=GET
> url="/1.0/containers?recursion=1" ip=@
> t=2015-11-19T22:49:06-0500 lvl=info msg=handling method=GET url=/1.0 ip=@
> t=2015-11-19T22:49:06-0500 lvl=info msg=handling method=GET
> url="/1.0/containers?recursion=1" ip=@
> t=2015-11-19T22:52:29-0500 lvl=info msg=handling ip=@ method=GET url=/1.0
> t=2015-11-19T22:52:29-0500 lvl=info msg=handling method=GET
> url="/1.0/containers?recursion=1" ip=@
>
> thanks very much
>
>
>
> From: "Stéphane Graber" <stgraber at ubuntu.com>
> To: LXC users mailing-list <lxc-users at lists.linuxcontainers.org>
> Cc:
> Date: Tue, 17 Nov 2015 22:09:17 -0500
> Subject: Re: [lxc-users] lxc list fails to show containers after upgrade
> to lxd 0.22 / lxc 1.1.5
> Can you confirm your client and server are running the exact same
> version of LXD?
>
> The fact that the containers appear to be properly in the database
> typically indicates an API mismatch between the client and the server.
>
> Our API isn't stable yet (it will be when we tag LXD 2.0 early next
> year), so mismatches can give you random results.
>
>
> If "lxc version" and "lxd --version" match, then I'd suggest running
> both of them with --debug, that should give us more of an idea of what's
> going on.
>
> On Tue, Nov 17, 2015 at 09:57:50PM -0500, Matthew McKennirey wrote:
> > I am running LXD/LXC on two hosts, each with multiple containers.
> > I upgraded one host to the latest versions of LXD /LXC (had been on lxd
> > 0.21 / lxc 1.1.4)
> >
> > After the upgrade. lxc list shows no containers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20151121/185ae187/attachment.html>