[lxc-users] lxd hang - "panic: runtime error: slice bounds out of range"

Tomasz Chmielewski mangoo at wpkg.org
Wed Feb 1 06:19:44 UTC 2017


Unfortunately it's still crashing, around 1 day after removing the 
docker container:

Feb  1 06:16:20 lxd01 lxd[2593]: error: LXD still not running after 600s 
timeout.
Feb  1 06:16:20 lxd01 systemd[1]: lxd.service: Control process exited, 
code=exited status=1
Feb  1 06:16:20 lxd01 systemd[1]: Failed to start LXD - main daemon.
Feb  1 06:16:20 lxd01 systemd[1]: lxd.service: Unit entered failed 
state.
Feb  1 06:16:20 lxd01 systemd[1]: lxd.service: Failed with result 
'exit-code'.
Feb  1 06:16:20 lxd01 systemd[1]: lxd.service: Service hold-off time 
over, scheduling restart.
Feb  1 06:16:20 lxd01 systemd[1]: Stopped LXD - main daemon.
Feb  1 06:16:20 lxd01 systemd[1]: Starting LXD - main daemon...
Feb  1 06:16:20 lxd01 lxd[29235]: t=2017-02-01T06:16:20+0000 lvl=warn 
msg="CGroup memory swap accounting is disabled, swap limits will be 
ignored."
Feb  1 06:16:20 lxd01 lxd[29235]: panic: runtime error: slice bounds out 
of range
Feb  1 06:16:20 lxd01 lxd[29235]: goroutine 16 [running]:
Feb  1 06:16:20 lxd01 lxd[29235]: panic(0xadef00, 0xc82000e050)
Feb  1 06:16:20 lxd01 lxd[29235]: 
#011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
Feb  1 06:16:20 lxd01 lxd[29235]: 
main.(*storageBtrfs).getSubVolumes(0xc8200fa240, 0xc82000b880, 0x3a, 
0x0, 0x0, 0x0, 0x0, 0x0)
Feb  1 06:16:20 lxd01 lxd[29235]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812 
+0x104f
Feb  1 06:16:20 lxd01 lxd[29235]: 
main.(*storageBtrfs).subvolsDelete(0xc8200fa240, 0xc82000b880, 0x3a, 
0x0, 0x0)
Feb  1 06:16:20 lxd01 lxd[29235]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574 
+0x72
Feb  1 06:16:20 lxd01 lxd[29235]: 
main.(*storageBtrfs).ContainerDelete(0xc8200fa240, 0x7feeacffd0c0, 
0xc8204fa480, 0x0, 0x0)
Feb  1 06:16:20 lxd01 lxd[29235]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119 
+0xb0
Feb  1 06:16:20 lxd01 lxd[29235]: 
main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200fa240, 
0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0)
Feb  1 06:16:20 lxd01 lxd[29235]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321 
+0x5c
Feb  1 06:16:20 lxd01 lxd[29235]: 
main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60, 
0x7feeacffd0c0, 0xc8204fa480, 0x0, 0x0)
Feb  1 06:16:20 lxd01 lxd[29235]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510 
+0x22b
Feb  1 06:16:20 lxd01 lxd[29235]: 
main.(*containerLXC).Delete(0xc8204fa480, 0x0, 0x0)
Feb  1 06:16:20 lxd01 lxd[29235]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366 
+0x30e
Feb  1 06:16:20 lxd01 lxd[29235]: 
main.snapshotDelete.func1(0xc8205400b0, 0x0, 0x0)
Feb  1 06:16:20 lxd01 lxd[29235]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248 
+0x3e
Feb  1 06:16:20 lxd01 lxd[29235]: 
main.(*operation).Run.func1(0xc8205400b0, 0xc820258f60)
Feb  1 06:16:20 lxd01 lxd[29235]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110 
+0x3a
Feb  1 06:16:20 lxd01 lxd[29235]: created by main.(*operation).Run
Feb  1 06:16:20 lxd01 lxd[29235]: 
#011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137 
+0x127
Feb  1 06:16:20 lxd01 systemd[1]: lxd.service: Main process exited, 
code=exited, status=2/INVALIDARGUMENT


Any clues what's causing this and how to fix?


Tomasz

On 2017-01-31 16:29, Tomasz Chmielewski wrote:
> I think it may be related to
> https://www.stgraber.org/2016/04/13/lxd-2-0-docker-in-lxd-712/
> 
> I have a docker container, with several dockers inside, and with lxd 
> snapshots.
> 
> Doing this:
> 
> # lxc delete docker
> error: Get
> http://unix.socket/1.0/operations/7d30bf41-3af6-4b48-b42c-06fdd2bba48b/wait:
> EOF
> 
> Results in:
> 
> Jan 31 07:28:36 lxd01 lxd[8363]: panic: runtime error: slice bounds out 
> of range
> Jan 31 07:28:36 lxd01 lxd[8363]: goroutine 867 [running]:
> Jan 31 07:28:36 lxd01 lxd[8363]: panic(0xadef00, 0xc82000e050)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
> Jan 31 07:28:36 lxd01 lxd[8363]:
> main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc82000bb80, 0x3a,
> 0x0, 0x0, 0x0, 0x0, 0x0)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
> +0x104f
> Jan 31 07:28:36 lxd01 lxd[8363]:
> main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc82000bb80, 0x3a,
> 0x0, 0x0)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
> +0x72
> Jan 31 07:28:36 lxd01 lxd[8363]:
> main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f5f55ba1900,
> 0xc8204a0480, 0x0, 0x0)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
> +0xb0
> Jan 31 07:28:36 lxd01 lxd[8363]:
> main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
> 0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
> +0x5c
> Jan 31 07:28:36 lxd01 lxd[8363]:
> main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
> 0x7f5f55ba1900, 0xc8204a0480, 0x0, 0x0)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
> +0x22b
> Jan 31 07:28:36 lxd01 lxd[8363]:
> main.(*containerLXC).Delete(0xc8204a0480, 0x0, 0x0)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
> +0x30e
> Jan 31 07:28:36 lxd01 lxd[8363]:
> main.containerDeleteSnapshots(0xc820090a00, 0xc82017c017, 0x6, 0x0,
> 0x0)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/containers.go:208
> +0x4ac
> Jan 31 07:28:36 lxd01 lxd[8363]:
> main.(*containerLXC).Delete(0xc8204a00c0, 0x0, 0x0)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2371
> +0x696
> Jan 31 07:28:36 lxd01 lxd[8363]:
> main.containerDelete.func1(0xc8200c8840, 0x0, 0x0)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_delete.go:22
> +0x3e
> Jan 31 07:28:36 lxd01 lxd[8363]:
> main.(*operation).Run.func1(0xc8200c8840, 0xc82011f320)
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
> +0x3a
> Jan 31 07:28:36 lxd01 lxd[8363]: created by main.(*operation).Run
> Jan 31 07:28:36 lxd01 lxd[8363]:
> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
> +0x127
> Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Main process exited,
> code=exited, status=2/INVALIDARGUMENT
> Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Unit entered failed 
> state.
> Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Failed with result 
> 'exit-code'.
> Jan 31 07:28:36 lxd01 systemd[1]: lxd.service: Service hold-off time
> over, scheduling restart.
> Jan 31 07:28:36 lxd01 systemd[1]: Stopped LXD - main daemon.
> Jan 31 07:28:36 lxd01 systemd[1]: Starting LXD - main daemon...
> Jan 31 07:28:36 lxd01 lxd[11969]: t=2017-01-31T07:28:36+0000 lvl=warn
> msg="CGroup memory swap accounting is disabled, swap limits will be
> ignored."
> Jan 31 07:28:37 lxd01 systemd[1]: Started LXD - main daemon.
> 
> 
> 
> 
> Tomasz
> 
> On 2017-01-31 16:17, Tomasz Chmielewski wrote:
>> lxd process on one of my servers started to hang a few days ago.
>> 
>> In syslog, I can see the following being repeated (log below).
>> 
>> I see a similar issue here:
>> 
>> https://github.com/lxc/lxd/issues/2089
>> 
>> but it's closed.
>> 
>> Running:
>> 
>> ii  lxd                              2.0.8-0ubuntu1~ubuntu16.04.2
>>   amd64        Container hypervisor based on LXC - daemon
>> ii  lxd-client                       2.0.8-0ubuntu1~ubuntu16.04.2
>>   amd64        Container hypervisor based on LXC - client
>> 
>> On Ubuntu 16.04 with 4.9.0 kernel from ubuntu ppa.
>> 
>> 
>> It seems to recover if I kill this process:
>> 
>> root     26853  0.0  0.0 222164 12228 ?        Ssl  07:13   0:00
>> /usr/bin/lxd waitready --timeout=600
>> 
>> 
>> Sometimes need to kill it a few times until it recovers.
>> 
>> 
>> Jan 31 06:26:06 lxd01 lxd[15762]: error: LXD still not running after
>> 600s timeout.
>> Jan 31 06:26:06 lxd01 lxd[4402]: t=2017-01-31T06:26:06+0000 lvl=warn
>> msg="CGroup memory swap accounting is disabled, swap limits will be
>> ignored."
>> Jan 31 06:26:06 lxd01 lxd[4402]: panic: runtime error: slice bounds 
>> out of range
>> Jan 31 06:26:06 lxd01 lxd[4402]: goroutine 27 [running]:
>> Jan 31 06:26:06 lxd01 lxd[4402]: panic(0xadef00, 0xc82000e050)
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> main.(*storageBtrfs).getSubVolumes(0xc8200f8240, 0xc8201b22c0, 0x3a,
>> 0x0, 0x0, 0x0, 0x0, 0x0)
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
>> +0x104f
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> main.(*storageBtrfs).subvolsDelete(0xc8200f8240, 0xc8201b22c0, 0x3a,
>> 0x0, 0x0)
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
>> +0x72
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> main.(*storageBtrfs).ContainerDelete(0xc8200f8240, 0x7f872e0f00c0,
>> 0xc8200cc540, 0x0, 0x0)
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
>> +0xb0
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> main.(*storageBtrfs).ContainerSnapshotDelete(0xc8200f8240,
>> 0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
>> +0x5c
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> main.(*storageLogWrapper).ContainerSnapshotDelete(0xc8200fda60,
>> 0x7f872e0f00c0, 0xc8200cc540, 0x0, 0x0)
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
>> +0x22b
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> main.(*containerLXC).Delete(0xc8200cc540, 0x0, 0x0)
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
>> +0x30e
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> main.snapshotDelete.func1(0xc8200c8000, 0x0, 0x0)
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
>> +0x3e
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> main.(*operation).Run.func1(0xc8200c8000, 0xc8200107e0)
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
>> +0x3a
>> Jan 31 06:26:06 lxd01 lxd[4402]: created by main.(*operation).Run
>> Jan 31 06:26:06 lxd01 lxd[4402]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
>> +0x127
>> Jan 31 06:36:06 lxd01 lxd[4408]: error: LXD still not running after
>> 600s timeout.
>> Jan 31 06:36:06 lxd01 lxd[21952]: t=2017-01-31T06:36:06+0000 lvl=warn
>> msg="CGroup memory swap accounting is disabled, swap limits will be
>> ignored."
>> Jan 31 06:36:06 lxd01 lxd[21952]: panic: runtime error: slice bounds
>> out of range
>> Jan 31 06:36:06 lxd01 lxd[21952]: goroutine 16 [running]:
>> Jan 31 06:36:06 lxd01 lxd[21952]: panic(0xadef00, 0xc82000e050)
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> #011/usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> main.(*storageBtrfs).getSubVolumes(0xc820104240, 0xc8201a0140, 0x3a,
>> 0x0, 0x0, 0x0, 0x0, 0x0)
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:812
>> +0x104f
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> main.(*storageBtrfs).subvolsDelete(0xc820104240, 0xc8201a0140, 0x3a,
>> 0x0, 0x0)
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:574
>> +0x72
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> main.(*storageBtrfs).ContainerDelete(0xc820104240, 0x7f0acd705938,
>> 0xc8204c8180, 0x0, 0x0)
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:119
>> +0xb0
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> main.(*storageBtrfs).ContainerSnapshotDelete(0xc820104240,
>> 0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage_btrfs.go:321
>> +0x5c
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> main.(*storageLogWrapper).ContainerSnapshotDelete(0xc820107a60,
>> 0x7f0acd705938, 0xc8204c8180, 0x0, 0x0)
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/storage.go:510
>> +0x22b
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> main.(*containerLXC).Delete(0xc8204c8180, 0x0, 0x0)
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_lxc.go:2366
>> +0x30e
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> main.snapshotDelete.func1(0xc8200d42c0, 0x0, 0x0)
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/container_snapshot.go:248
>> +0x3e
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> main.(*operation).Run.func1(0xc8200d42c0, 0xc820011740)
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:110
>> +0x3a
>> Jan 31 06:36:06 lxd01 lxd[21952]: created by main.(*operation).Run
>> Jan 31 06:36:06 lxd01 lxd[21952]:
>> #011/build/lxd-tfF8X9/lxd-2.0.8/obj-x86_64-linux-gnu/src/github.com/lxc/lxd/lxd/operations.go:137
>> +0x127
>> Jan 31 06:46:06 lxd01 lxd[21955]: error: LXD still not running after
>> 600s timeout.
>> 
>> 
>> 
>> 
>> Tomasz Chmielewski
>> https://lxadm.com
>> _______________________________________________
>> lxc-users mailing list
>> lxc-users at lists.linuxcontainers.org
>> http://lists.linuxcontainers.org/listinfo/lxc-users
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users


More information about the lxc-users mailing list