[lxc-users] lxc list fails to show containers after upgrade to lxd 0.22 / lxc 1.1.5
Stéphane Graber
stgraber at ubuntu.com
Fri Nov 20 04:55:20 UTC 2015
On Thu, Nov 19, 2015 at 11:51:55PM -0500, Stéphane Graber wrote:
> Hi,
>
> None of that really makes much sense...
>
> LXD should absolutely work on recent 3.19 kernels, in fact, our whole
> upstream testing is done on the latest 3.19.
>
> The mkdir error is also very odd as the call that appears to be failing
> there is a MkdirAll which specifically shouldn't fail if the file
> exists. A corrupt filesystem may explain that but you'd see more sign of
> that in your syslog and kernel log...
>
> It looks like something really odd is going on with your /var/lib/lxd,
> either at the filesystem level or maybe at the mount level.
>
> Having /var/lib/lxd be over-mounted or something similar happening to it
> may explain why the filesystem looks fine and contains all your data but
> LXD isn't able to see it somehow.
>
>
> To try and debug this further, can you attach:
> - Output of: dmesg
> - Output of: cat /proc/self/mountinfo
> - Do:
> - sudo systemctl stop lxd
> - pkill -9 lxd
> - lxd --debug --group lxd
> In a separate shell:
> - lxc list --debug
> Then give me the output of both lxd and lxc separately please.
Serge just pointed out to me (on IRC) that you seem to be using upstart
on Ubuntu 15.04 which is not the default setup. Things should still work
though as I use LXD on Ubuntu 14.04 with upstart without any problem.
It does change my instructions slightly though:
To try and debug this further, can you attach:
- Output of: dmesg
- Output of: cat /proc/self/mountinfo
- Do:
- sudo stop lxd
- lxd --debug --group lxd
In a separate shell:
- lxc list --debug
Then give me the output of both lxd and lxc separately please.
>
> Thanks
>
> On Thu, Nov 19, 2015 at 11:30:46PM -0500, Matthew McKennirey wrote:
> > Thanks for the rapid response. I rebooted the server, then tried
> > again. Host is Ubuntu 15.04 fully patched, kernel 3.19.0-25 (with later
> > kernels lxd will not start because of a permission issue with proc)
> >
> > $ sudo uname -a
> > Linux zeus 3.19.0-25-generic #26-Ubuntu SMP Fri Jul 24 21:17:31 UTC 2015
> > x86_64 x86_64 x86_64 GNU/Linux
> >
> > lxc is 1.1.5-0ubuntu3~ubuntu
> > lxd is 0.22-0ubuntu2~ubuntu1
> >
> > I am running all client commands on the host machine
> > $ lxc --version
> > 0.22
> > $ lxd --version
> > 0.22
> >
> > Containers are present on the file system
> > $ ls -l /var/lib/lxd/containers/
> > -rwxrwxrwx 1 root root 27854 Nov 14 23:20 lxc-monitord.log
> > drwxrwxrwx 3 100000 100000 5 Feb 14 2015 ub1404_sg
> > drwx--x--x 3 root root 3 Aug 16 14:29 ubuntu14-java-utils
> > drwx--x--x 3 100000 100000 3 Aug 18 23:12 ubuntu-build
> > drwxr-xr-x 3 root root 3 Oct 18 17:24 ubuntu-build-clone
> > drwx--x--x 4 100000 100000 5 Aug 16 14:09 ubuntu-sg
> > drwx--x--x 3 root root 3 Aug 16 15:38 ubuntu-sg-base
> > drwxr-xr-x 3 100000 100000 3 Oct 18 18:25 ubuntu-sg-tst
> > drwxr-xr-x 3 100000 100000 3 Oct 18 17:30 ubuntu-tst
> > drwxr-xr-x 3 root root 3 Oct 18 17:18 ubuntu-web
> >
> > Containers are in lxd.db (see prior message)
> >
> > After the reboot I had trouble starting lxd
> >
> > $ sudo cat /var/log/syslog
> > kernel: [ 6575.572166] init: lxd main process (5391) terminated with status
> > 1
> > kernel: [ 6575.572179] init: lxd main process ended, respawning
> > kernel: [ 6575.588459] init: lxd main process (5396) terminated with status
> > 1
> > kernel: [ 6575.588474] init: lxd main process ended, respawning
> > kernel: [ 6575.604184] init: lxd main process (5401) terminated with status
> > 1
> > kernel: [ 6575.604198] init: lxd respawning too fast, stopped
> >
> > $ sudo cat /var/log/upstart/lxd.log
> > error: mkdir /var/lib/lxd/images: file exists
> > error: mkdir /var/lib/lxd/images: file exists
> > error: mkdir /var/lib/lxd/images: file exists
> > error: mkdir /var/lib/lxd/images: file exists
> >
> > I'm really not sure why lxd was trying to create the images folder at
> > start-up.
> > Anyway I renamed the folder and lxd would start
> >
> > $ sudo mv /var/lib/lxd/images /var/lib/lxd/images-back
> >
> > $ sudo service lxd start --debug -v
> > $ lxc list --debug
> > DBUG[11-19|22:52:29] Fingering the daemon
> > DBUG[11-19|22:52:29] Raw response:
> > {"type":"sync","status":"Success","status_code":200,"metadata":{"api_compat":1,"auth":"trusted","config":{"images.remote_cache_expiry":"10"},"environment":{"addresses":[],"architectures":[2,1],"driver":"lxc","driver_version":"1.1.5","kernel":"Linux","kernel_architecture":"x86_64","kernel_version":"3.19.0-25-generic","server":"lxd","server_pid":5613,"server_version":"0.22","storage":"dir","storage_version":""}}}
> >
> > DBUG[11-19|22:52:29] Pong received
> > DBUG[11-19|22:52:29] Raw response:
> > {"type":"sync","status":"Success","status_code":200,"metadata":[]}
> >
> > +------+-------+------+------+-----------+-----------+
> > | NAME | STATE | IPV4 | IPV6 | EPHEMERAL | SNAPSHOTS |
> > +------+-------+------+------+-----------+-----------+
> > +------+-------+------+------+-----------+-----------+
> >
> > Still no joy.
> >
> > $ sudo cat /var/log/lxd/lxd.log
> > t=2015-11-19T22:45:51-0500 lvl=info msg="LXD is starting" path=/var/lib/lxd
> > t=2015-11-19T22:45:51-0500 lvl=info msg="Default uid/gid map:"
> > t=2015-11-19T22:45:51-0500 lvl=info msg=" - u 0 100000 65536"
> > t=2015-11-19T22:45:51-0500 lvl=info msg=" - g 0 100000 65536"
> > t=2015-11-19T22:45:51-0500 lvl=info msg=Init driver=storage/dir
> > t=2015-11-19T22:45:51-0500 lvl=info msg="Looking for existing certificates"
> > cert=/var/lib/lxd/server.crt key=/var/lib/lxd/server.key
> > t=2015-11-19T22:45:51-0500 lvl=info msg="LXD isn't socket activated"
> > t=2015-11-19T22:45:51-0500 lvl=info msg="REST API daemon:"
> > t=2015-11-19T22:45:51-0500 lvl=info msg=" - binding socket"
> > socket=/var/lib/lxd/unix.socket
> > t=2015-11-19T22:46:09-0500 lvl=info msg=handling method=GET url=/1.0 ip=@
> > t=2015-11-19T22:46:09-0500 lvl=info msg=handling method=GET
> > url="/1.0/containers?recursion=1" ip=@
> > t=2015-11-19T22:48:20-0500 lvl=info msg=handling method=GET url=/1.0 ip=@
> > t=2015-11-19T22:48:20-0500 lvl=info msg=handling method=GET
> > url="/1.0/containers?recursion=1" ip=@
> > t=2015-11-19T22:49:06-0500 lvl=info msg=handling method=GET url=/1.0 ip=@
> > t=2015-11-19T22:49:06-0500 lvl=info msg=handling method=GET
> > url="/1.0/containers?recursion=1" ip=@
> > t=2015-11-19T22:52:29-0500 lvl=info msg=handling ip=@ method=GET url=/1.0
> > t=2015-11-19T22:52:29-0500 lvl=info msg=handling method=GET
> > url="/1.0/containers?recursion=1" ip=@
> >
> > thanks very much
> >
> >
> >
> > From: "Stéphane Graber" <stgraber at ubuntu.com>
> > To: LXC users mailing-list <lxc-users at lists.linuxcontainers.org>
> > Cc:
> > Date: Tue, 17 Nov 2015 22:09:17 -0500
> > Subject: Re: [lxc-users] lxc list fails to show containers after upgrade to
> > lxd 0.22 / lxc 1.1.5
> > Can you confirm your client and server are running the exact same
> > version of LXD?
> >
> > The fact that the containers appear to be properly in the database
> > typically indicates an API mismatch between the client and the server.
> >
> > Our API isn't stable yet (it will be when we tag LXD 2.0 early next
> > year), so mismatches can give you random results.
> >
> >
> > If "lxc version" and "lxd --version" match, then I'd suggest running
> > both of them with --debug, that should give us more of an idea of what's
> > going on.
> >
> > On Tue, Nov 17, 2015 at 09:57:50PM -0500, Matthew McKennirey wrote:
> > > I am running LXD/LXC on two hosts, each with multiple containers.
> > > I upgraded one host to the latest versions of LXD /LXC (had been on lxd
> > > 0.21 / lxc 1.1.4)
> > >
> > > After the upgrade. lxc list shows no containers
>
> > _______________________________________________
> > lxc-users mailing list
> > lxc-users at lists.linuxcontainers.org
> > http://lists.linuxcontainers.org/listinfo/lxc-users
>
>
> --
> Stéphane Graber
> Ubuntu developer
> http://www.ubuntu.com
--
Stéphane
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20151119/fadfb603/attachment-0001.sig>
More information about the lxc-users
mailing list