<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Thank you Fajar.</p>
<p>I have made some progress, but would still value help (fresh
detail at the bottom).<br>
</p>
<br>
<div class="moz-cite-prefix">On 14/08/18 19:32, Fajar A. Nugraha
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAG1y0scqxtO=CoOv+rA8=VpXHeaoz-Vi0ELzvQd7pXMd9+i2aA@mail.gmail.com">
<meta http-equiv="Context-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Aug 14, 2018 at 1:54 PM, Tony
Lewis <span dir="ltr"><<a
href="mailto:tony@lewistribe.com" target="_blank"
moz-do-not-send="true">tony@lewistribe.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote">Apologies in advance for the
bump, but does anyone have an insights on this?
<div class="gmail-HOEnZb">
<div class="gmail-h5"><br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Did you install lxd before using source instead of
snap?</div>
</div>
</div>
</div>
</blockquote>
<br>
It turns out there were some residual config files left over from
the package-based install. No binaries, just various files in
/etc. I cleaned them up.<br>
<br>
<blockquote type="cite"
cite="mid:CAG1y0scqxtO=CoOv+rA8=VpXHeaoz-Vi0ELzvQd7pXMd9+i2aA@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><br>
<div>What does /var/snap/lxd/common/lxd/logs/lxd.log say?
Does it have any error?</div>
</div>
</div>
</div>
</blockquote>
<br>
Not much of interest that I can see. Here it is from a reboot
today:<br>
<br>
lvl=info msg="LXD 3.3 is starting in normal mode"
path=/var/snap/lxd/common/lxd t=2018-08-15T01:25:20+0000<br>
lvl=info msg="Kernel uid/gid map:" t=2018-08-15T01:25:20+0000<br>
lvl=info msg=" - u 0 0 4294967295" t=2018-08-15T01:25:20+0000<br>
lvl=info msg=" - g 0 0 4294967295" t=2018-08-15T01:25:20+0000<br>
lvl=info msg="Configured LXD uid/gid map:"
t=2018-08-15T01:25:20+0000<br>
lvl=info msg=" - u 0 1000000 1000000000" t=2018-08-15T01:25:20+0000<br>
lvl=info msg=" - g 0 1000000 1000000000" t=2018-08-15T01:25:20+0000<br>
lvl=warn msg="CGroup memory swap accounting is disabled, swap limits
will be ignored." t=2018-08-15T01:25:20+0000<br>
lvl=info msg="Initializing local database"
t=2018-08-15T01:25:20+0000<br>
lvl=info msg="Initializing database gateway"
t=2018-08-15T01:25:20+0000<br>
address= id=1 lvl=info msg="Start database node"
t=2018-08-15T01:25:20+0000<br>
lvl=info msg="Raft: Restored from snapshot 1-23922-1534296032171"
t=2018-08-15T01:25:20+0000<br>
lvl=info msg="Raft: Initial configuration (index=1):
[{Suffrage:Voter ID:1 Address:0}]" t=2018-08-15T01:25:20+0000<br>
lvl=info msg="Raft: Node at 0 [Leader] entering Leader state"
t=2018-08-15T01:25:20+0000<br>
lvl=info msg="LXD isn't socket activated" t=2018-08-15T01:25:20+0000<br>
lvl=info msg="Starting /dev/lxd handler:" t=2018-08-15T01:25:20+0000<br>
lvl=info msg=" - binding devlxd socket"
socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2018-08-15T01:25:20+0000<br>
lvl=info msg="REST API daemon:" t=2018-08-15T01:25:20+0000<br>
lvl=info msg=" - binding Unix socket"
socket=/var/snap/lxd/common/lxd/unix.socket
t=2018-08-15T01:25:20+0000<br>
lvl=info msg="Initializing global database"
t=2018-08-15T01:25:20+0000<br>
lvl=info msg="Initializing storage pools" t=2018-08-15T01:25:21+0000<br>
lvl=info msg="Initializing networks" t=2018-08-15T01:25:21+0000<br>
lvl=info msg="Loading configuration" t=2018-08-15T01:25:22+0000<br>
lvl=info msg="Connected to MAAS controller"
t=2018-08-15T01:25:22+0000<br>
lvl=info msg="Pruning expired images" t=2018-08-15T01:25:22+0000<br>
lvl=info msg="Done pruning expired images"
t=2018-08-15T01:25:22+0000<br>
lvl=info msg="Updating instance types" t=2018-08-15T01:25:22+0000<br>
lvl=info msg="Expiring log files" t=2018-08-15T01:25:22+0000<br>
lvl=info msg="Done expiring log files" t=2018-08-15T01:25:22+0000<br>
lvl=info msg="Updating images" t=2018-08-15T01:25:22+0000<br>
lvl=info msg="Done updating images" t=2018-08-15T01:25:22+0000<br>
lvl=warn msg="Unable to update backup.yaml at this time"
name=backuptests t=2018-08-15T01:25:23+0000<br>
lvl=info msg="Done updating instance types"
t=2018-08-15T01:25:35+0000<br>
<br>
<br>
<br>
<blockquote type="cite"
cite="mid:CAG1y0scqxtO=CoOv+rA8=VpXHeaoz-Vi0ELzvQd7pXMd9+i2aA@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>My GUESS is that you have /usr/bin/lxd and
/snap/bin/lxd, which interfere with each other. If that's
not it, then my next guess is that there's probably some
group issue, like <a
href="https://github.com/lxc/lxd/issues/1861#issuecomment-206507631"
moz-do-not-send="true">https://github.com/lxc/lxd/issues/1861#issuecomment-206507631</a>
. In any case lxd.log might have more info.</div>
</div>
</div>
</div>
</blockquote>
<br>
Thank you. There is no lxd binary anywhere other than three snap
versions, and only one of those is running. The old service was
still there, now removed, but it was showing failure because there
was no binary to start.<br>
<br>
Progress:<br>
<br>
I know that lxd is starting, but my containers still don't start.
When I try to stop the service I see the following in systemctl:<br>
<br>
# systemctl stop snap.lxd.daemon<br>
# systemctl status snap.lxd.daemon<br>
● snap.lxd.daemon.service - Service for snap application lxd.daemon<br>
Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service;
enabled; vendor preset: enabled)<br>
Active: inactive (dead) since Wed 2018-08-15 11:40:51 AEST; 2s
ago<br>
Process: 6761 ExecStop=/usr/bin/snap run --command=stop lxd.daemon
(code=exited, status=0/SUCCESS)<br>
Process: 5432 ExecStart=/usr/bin/snap run lxd.daemon (code=killed,
signal=TERM)<br>
Main PID: 5432 (code=killed, signal=TERM)<br>
<br>
Aug 15 11:40:47 server systemd[1]: Stopping Service for snap
application lxd.daemon...<br>
Aug 15 11:40:47 server /usr/bin/snap[6761]: cmd.go:105: DEBUG:
restarting into "/snap/core/current/usr/bin/snap"<br>
Aug 15 11:40:47 server snap[6777]: cmd.go:105: DEBUG: restarting
into "/snap/core/current/usr/bin/snap"<br>
Aug 15 11:40:47 server snap[6761]: error: no changes found<br>
Aug 15 11:40:50 server snap[6761]: => Stop reason is: host
shutdown<br>
Aug 15 11:40:50 server snap[6761]: => Stopping LXD (with
container shutdown)<br>
Aug 15 11:40:50 server snap[6761]: lxd: error while loading shared
libraries: liblxc.so.1: cannot open shared object file: No such file
or directory<br>
Aug 15 11:40:50 server snap[6761]: => Stopping LXCFS<br>
Aug 15 11:40:51 server snap[5432]: => LXD is ready<br>
Aug 15 11:40:51 server systemd[1]: Stopped Service for snap
application lxd.daemon.<br>
<br>
A key line is: lxd: error while loading shared libraries:
liblxc.so.1: cannot open shared object file: No such file or
directory<br>
<br>
The library is present in what looks to be the right places in the
snap directories, but not anywhere else:<br>
<br>
# find /snap -name liblxc.so.1 -print<br>
/snap/lxd/7651/lib/liblxc.so.1<br>
/snap/lxd/7792/lib/liblxc.so.1<br>
/snap/lxd/8011/lib/liblxc.so.1<br>
<br>
But when being launched, the daemon does not attempt to load from
the snap directories:<br>
<br>
# strace -f -F -etrace=file /usr/bin/snap run --command=stop
lxd.daemon 2>&1 | grep liblxc<br>
[pid 4964] open("/lib/x86_64-linux-gnu/tls/x86_64/liblxc.so.1",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/lib/x86_64-linux-gnu/tls/liblxc.so.1",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/lib/x86_64-linux-gnu/x86_64/liblxc.so.1",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/lib/x86_64-linux-gnu/liblxc.so.1",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/usr/lib/x86_64-linux-gnu/tls/x86_64/liblxc.so.1",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/usr/lib/x86_64-linux-gnu/tls/liblxc.so.1",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/usr/lib/x86_64-linux-gnu/x86_64/liblxc.so.1",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/usr/lib/x86_64-linux-gnu/liblxc.so.1",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/lib/tls/x86_64/liblxc.so.1", O_RDONLY|O_CLOEXEC)
= -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/lib/tls/liblxc.so.1", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)<br>
[pid 4964] open("/lib/x86_64/liblxc.so.1", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)<br>
[pid 4964] open("/lib/liblxc.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT
(No such file or directory)<br>
[pid 4964] open("/usr/lib/tls/x86_64/liblxc.so.1",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/usr/lib/tls/liblxc.so.1", O_RDONLY|O_CLOEXEC) =
-1 ENOENT (No such file or directory)<br>
[pid 4964] open("/usr/lib/x86_64/liblxc.so.1", O_RDONLY|O_CLOEXEC)
= -1 ENOENT (No such file or directory)<br>
[pid 4964] open("/usr/lib/liblxc.so.1", O_RDONLY|O_CLOEXEC) = -1
ENOENT (No such file or directory)<br>
lxd: error while loading shared libraries: liblxc.so.1: cannot open
shared object file: No such file or directory<br>
<br>
Strangely, even if I copy /snap/lxd/8011/lib/liblxc.so.1 into /lib,
the file is not found (strace reports no such file or directory). I
can't explain this, and I've checked and rechecked this.<br>
<br>
If I kill the daemon itself, I can restart it using systemctl and my
containers will start. However I cannot gracefully stop containers
(lxc stop <container> just hangs) nor can I gracefully stop
lxd (same missing library error).<br>
<br>
Any thoughts?<br>
<br>
Tony<br>
<br>
</body>
</html>