<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
On 13-09-18 05:57, Fajar A. Nugraha wrote:<br>
<blockquote type="cite"
cite="mid:CAG1y0sfnvcVpoa91akeudmg3REdQ0SO1YT_eGz+LVU+DEue=vg@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, Sep 12, 2018 at 9:33 PM,
Kees Bakker <span dir="ltr"><<a
href="mailto:keesb@ghs.com" target="_blank"
moz-do-not-send="true">keesb@ghs.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Hey,<br>
<br>
This with a LXD/LXC on a Ubuntu 18.04 server. Storage
is done<br>
with LVM. It was installed as a cluster with just one
node.<br>
It was also added as remote for three other LXD
servers (all Ubuntu 16.04<br>
and LXD 2.0.x). These old servers have BTRFS storage.<br>
<br>
</blockquote>
<div><br>
</div>
<div>Only added as remote? not lxd clustering (<a
href="https://lxd.readthedocs.io/en/latest/clustering/"
moz-do-not-send="true">https://lxd.readthedocs.io/en/latest/clustering/</a>)?</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Yes, only as remote.<br>
<br>
<blockquote type="cite"
cite="mid:CAG1y0sfnvcVpoa91akeudmg3REdQ0SO1YT_eGz+LVU+DEue=vg@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Suddenly I cannot do any lxc command anymore. They all
give<br>
<br>
Error: failed to begin transaction: database is
locked<br>
<br>
In /var/log/lxd/lxd.log it prints the following
message every 10 seconds<br>
<br>
lvl=warn msg="Failed to get current raft nodes:
failed to fetch raft server address: failed to begin
transaction: database is locked"
t=2018-09-12T16:28:44+0200<br>
<br>
Extra information. This afternoon I have upgraded one
of the "old" servers<br>
to LXD 3.0 (from xenial-backports). This was triggered
by the problems we<br>
have with a container in ERROR state and a kworker at
100% cpu load.<br>
<span class="gmail-HOEnZb"></span></blockquote>
<div><br>
</div>
<div><br>
</div>
<div>Do package versions on upgraded servers match? i.e.
all lxd, liblxc1, etc all 3.0 from xenial-backports,
without any 2.x or ppa packages mixed in?</div>
<div><br>
</div>
<div>Have you restart lxd on the upgraded server?</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
Not manually, no. The upgrade wasn't totally smooth. It ran into a
timeout setting<br>
up some lxc network config.<br>
<br>
Then did a reboot, and the shutdown was hanging for something with
ebtables (new<br>
package because of the move to 3.0). I forced a powerdown and
luckily the server<br>
came up normal.<br>
<br>
After that I noticed the problem described above. Restarting the lxd
server<br>
solve it, and it is back to normal. (( I didn't know for sure that
the LXD server can<br>
be restarted without killing the containers. But it worked. ))<br>
<br>
Here are a few lines from lxd.log at the time it started giving the
problem.<br>
<br>
lvl=info msg="Raft: Snapshot to 597621 complete"
t=2018-09-12T15:47:01+0200<br>
lvl=info msg="Raft: Starting snapshot up to 597696"
t=2018-09-12T15:52:15+0200<br>
lvl=info msg="Raft: Compacting logs from 597494 to 597568"
t=2018-09-12T15:52:16+0200<br>
lvl=info msg="Raft: Snapshot to 597696 complete"
t=2018-09-12T15:52:16+0200<br>
lvl=warn msg="Failed to get current raft nodes: failed to fetch raft
server address: failed to begin transaction: database is locked"
t=2018-09-12T15:56:55+0200<br>
lvl=warn msg="Failed to get current raft nodes: failed to fetch raft
server address: failed to begin transaction: database is locked"
t=2018-09-12T15:57:04+0200<br>
lvl=warn msg="Failed to get current raft nodes: failed to fetch raft
server address: failed to begin transaction: database is locked"
t=2018-09-12T15:57:13+0200<br>
lvl=warn msg="Failed to get current raft nodes: failed to fetch raft
server address: failed to begin transaction: database is locked"
t=2018-09-12T15:57:22+0200<br>
lvl=warn msg="Failed to get current raft nodes: failed to fetch raft
server address: failed to begin transaction: database is locked"
t=2018-09-12T15:57:31+0200<br>
lvl=warn msg="Failed to get current raft nodes: failed to fetch raft
server address: failed to begin transaction: database is locked"
t=2018-09-12T15:57:40+0200<br>
lvl=warn msg="Failed to get current raft nodes: failed to fetch raft
server address: failed to begin transaction: database is locked"
t=2018-09-12T15:57:49+0200<br>
lvl=info msg="Raft: Starting snapshot up to 597760"
t=2018-09-12T15:57:52+0200<br>
lvl=warn msg="Raft: Unable to get address for server id 1, using
fallback address 0: failed to begin transaction: database is locked"
t=2018-09-12T15:57:57+0200<br>
lvl=info msg="Raft: Compacting logs from 597569 to 597632"
t=2018-09-12T15:57:57+0200<br>
lvl=info msg="Raft: Snapshot to 597760 complete"
t=2018-09-12T15:57:57+0200<br>
lvl=warn msg="Failed to get current raft nodes: failed to fetch raft
server address: failed to begin transaction: database is locked"
t=2018-09-12T15:57:58+0200<br>
lvl=warn msg="Failed to get current raft nodes: failed to fetch raft
server address: failed to begin transaction: database is locked"
t=2018-09-12T15:58:07+0200<br>
lvl=warn msg="Failed to get current raft nodes: failed to fetch raft
server address: failed to begin transaction: database is locked"
t=2018-09-12T15:58:16+0200<br>
<br>
<br>
<blockquote type="cite"
cite="mid:CAG1y0sfnvcVpoa91akeudmg3REdQ0SO1YT_eGz+LVU+DEue=vg@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>If you temporarily move ~/.config/lxc somehere else
(to "remove" all the remotes, among other things),
does lxc command work?</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
I'll remember that for next time. Right now the server is working
again.<br>
<br>
Thanks<br>
</body>
</html>