[lxc-users] lxd hang - "panic: runtime error: slice bounds out of range"

[Tomasz Chmielewski]

lxd process on one of my servers started to hang a few days ago.

In syslog, I can see the following being repeated (log below).

I see a similar issue here:

https://github.com/lxc/lxd/issues/2089

but it's closed.

Running:

ii  lxd                              2.0.8-0ubuntu1~ubuntu16.04.2        
amd64        Container hypervisor based on LXC - daemon
ii  lxd-client                       2.0.8-0ubuntu1~ubuntu16.04.2        
amd64        Container hypervisor based on LXC - client

On Ubuntu 16.04 with 4.9.0 kernel from ubuntu ppa.


It seems to recover if I kill this process:

root     26853  0.0  0.0 222164 12228 ?        Ssl  07:13   0:00 
/usr/bin/lxd waitready --timeout=600


Sometimes need to kill it a few times until it recovers.


Jan 31 06:26:06 lxd01 lxd[15762]: error: LXD still not running after 
600s timeout.
Jan 31 06:26:06 lxd01 lxd[4402]: t=2017-01-31T06:26:06+0000 lvl=warn 
msg="CGroup memory swap accounting is disabled, swap limits will be 
ignored."
Jan 31 06:26:06 lxd01 lxd[4402]: panic: runtime error: slice bounds out 
of range
Jan 31 06:26:06 lxd01 lxd[4402]: goroutine 27 [running]:
Jan 31 06:26:06 lxd01 lxd[4402]: panic(0xadef00, 0xc82000e050)
Jan 31 06:26:06 lxd01 lxd[4402]: 
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
Jan 31 06:26:06 lxd01 lxd[4402]: 
main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc8201b22c0, 0x3a, 
0x0, 0x0, 0x0, 0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 
+0x104f
Jan 31 06:26:06 lxd01 lxd[4402]: 
main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc8201b22c0, 0x3a, 
0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 
+0x72
Jan 31 06:26:06 lxd01 lxd[4402]: 
main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f872e0f00c0, 
0xc8200cc540, 0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119 
+0xb0
Jan 31 06:26:06 lxd01 lxd[4402]: 
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240, 
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321 
+0x5c
Jan 31 06:26:06 lxd01 lxd[4402]: 
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60, 
0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510 
+0x22b
Jan 31 06:26:06 lxd01 lxd[4402]: 
main.(*containerLXC).Delete(0xc8200cc540, 0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366 
+0x30e
Jan 31 06:26:06 lxd01 lxd[4402]: main.snapshotDelete.func1(0xc8200c8000, 
0x0, 0x0)
Jan 31 06:26:06 lxd01 lxd[4402]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248 
+0x3e
Jan 31 06:26:06 lxd01 lxd[4402]: 
main.(*operation).Run.func1(0xc8200c8000, 0xc8200107e0)
Jan 31 06:26:06 lxd01 lxd[4402]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110 
+0x3a
Jan 31 06:26:06 lxd01 lxd[4402]: created by main.(*operation).Run
Jan 31 06:26:06 lxd01 lxd[4402]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137 
+0x127
Jan 31 06:36:06 lxd01 lxd[4408]: error: LXD still not running after 600s 
timeout.
Jan 31 06:36:06 lxd01 lxd[21952]: t=2017-01-31T06:36:06+0000 lvl=warn 
msg="CGroup memory swap accounting is disabled, swap limits will be 
ignored."
Jan 31 06:36:06 lxd01 lxd[21952]: panic: runtime error: slice bounds out 
of range
Jan 31 06:36:06 lxd01 lxd[21952]: goroutine 16 [running]:
Jan 31 06:36:06 lxd01 lxd[21952]: panic(0xadef00, 0xc82000e050)
Jan 31 06:36:06 lxd01 lxd[21952]: 
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
Jan 31 06:36:06 lxd01 lxd[21952]: 
main.(*storageBtrfs).getSubVolumes(0xc820104240, 0xc8201a0140, 0x3a, 
0x0, 0x0, 0x0, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 
+0x104f
Jan 31 06:36:06 lxd01 lxd[21952]: 
main.(*storageBtrfs).subvolsDelete(0xc820104240, 0xc8201a0140, 0x3a, 
0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 
+0x72
Jan 31 06:36:06 lxd01 lxd[21952]: 
main.(*storageBtrfs).ContainerDelete(0xc820104240, 0x7f0acd705938, 
0xc8204c8180, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119 
+0xb0
Jan 31 06:36:06 lxd01 lxd[21952]: 
main.(*storageBtrfs).ContainerSnapshotDelete(0xc820104240, 
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321 
+0x5c
Jan 31 06:36:06 lxd01 lxd[21952]: 
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc820107a60, 
0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510 
+0x22b
Jan 31 06:36:06 lxd01 lxd[21952]: 
main.(*containerLXC).Delete(0xc8204c8180, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366 
+0x30e
Jan 31 06:36:06 lxd01 lxd[21952]: 
main.snapshotDelete.func1(0xc8200d42c0, 0x0, 0x0)
Jan 31 06:36:06 lxd01 lxd[21952]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248 
+0x3e
Jan 31 06:36:06 lxd01 lxd[21952]: 
main.(*operation).Run.func1(0xc8200d42c0, 0xc820011740)
Jan 31 06:36:06 lxd01 lxd[21952]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110 
+0x3a
Jan 31 06:36:06 lxd01 lxd[21952]: created by main.(*operation).Run
Jan 31 06:36:06 lxd01 lxd[21952]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137 
+0x127
Jan 31 06:46:06 lxd01 lxd[21955]: error: LXD still not running after 
600s timeout.




Tomasz Chmielewski
https://lxadm.com