[lxc-users] Out-of-memory error

[Sebert, Holger.ext]

Hi,

I am encountering an out-of-memory error when using LXD 4.0.3 as a snap
package on Ubuntu 16.04. See further below for the exact log output.

The scenario in which this error occurs is somewhat complex, but
basically it is a Makefile performing the following tasks:

1. Create a container (call it "container 1").
2. Build lots of software in that container.
3. When finished, create a second container (call it "container 2")
4. Built even more software in "container 2".

In step 3 LXD hangs with the message "Creating container 2" and after
a while crashes with an out-of-memory exception.

When I restart the job at step 3 directly, it works without problems.
I.e. the error only occurs when the machine does a whole lots of stuff
beforehand.

Do you have any idea how I could debug this problem to give more
useful information about this problem?

Or do you even have an idea what could be wrong?

Note: I used LXD version 3.9 before, but as a Debian package (no snap),
there were no such problems.

Best,
Holger

---x---

Detailed log:

Using lxd version 4.0.3
*** create new container container-1313748364 for /home/hosebert.ext/data/ based on base-1453cd2104798d4c13583a16979a8e70 ***
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xc5fac3, 0x16)
	/snap/go/6439/src/runtime/panic.go:1116 +0x72
runtime.sysMap(0xc804000000, 0x800000000, 0x13643f8)
	/snap/go/6439/src/runtime/mem_linux.go:169 +0xc6
runtime.(*mheap).sysAlloc(0x1349740, 0x800000000, 0x42e017, 0x1349748)
	/snap/go/6439/src/runtime/malloc.go:727 +0x1e5
runtime.(*mheap).grow(0x1349740, 0x400000, 0x0)
	/snap/go/6439/src/runtime/mheap.go:1344 +0x85
runtime.(*mheap).allocSpan(0x1349740, 0x400000, 0x430100, 0x1364408, 0xfffffffffffffade)
	/snap/go/6439/src/runtime/mheap.go:1160 +0x6b6
runtime.(*mheap).alloc.func1()
	/snap/go/6439/src/runtime/mheap.go:907 +0x65
runtime.(*mheap).alloc(0x1349740, 0x400000, 0x7f9c01ff0101, 0x442718)
	/snap/go/6439/src/runtime/mheap.go:901 +0x85
runtime.largeAlloc(0x7fffffe00, 0xc000040101, 0xc000000180)
	/snap/go/6439/src/runtime/malloc.go:1177 +0x92
runtime.mallocgc.func1()
	/snap/go/6439/src/runtime/malloc.go:1071 +0x46
runtime.systemstack(0x0)
	/snap/go/6439/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	/snap/go/6439/src/runtime/proc.go:1116

goroutine 1 [running]:
runtime.systemstack_switch()
	/snap/go/6439/src/runtime/asm_amd64.s:330 fp=0xc000187008 sp=0xc000187000 pc=0x46b620
runtime.mallocgc(0x7fffffe00, 0xb51420, 0x1, 0x0)
	/snap/go/6439/src/runtime/malloc.go:1070 +0x938 fp=0xc0001870a8 sp=0xc000187008 pc=0x410078
runtime.makeslice(0xb51420, 0x7fffffe00, 0x7fffffe00, 0x4dda11)
	/snap/go/6439/src/runtime/slice.go:98 +0x6c fp=0xc0001870d8 sp=0xc0001870a8 pc=0x44ed0c
bytes.makeSlice(0x7fffffe00, 0x0, 0x0, 0x0)
	/snap/go/6439/src/bytes/buffer.go:229 +0x73 fp=0xc000187118 sp=0xc0001870d8 pc=0x500453
bytes.(*Buffer).grow(0xc000187210, 0x200, 0x1e00)
	/snap/go/6439/src/bytes/buffer.go:142 +0x156 fp=0xc000187168 sp=0xc000187118 pc=0x4ffd76
bytes.(*Buffer).ReadFrom(0xc000187210, 0xd45420, 0xc000138000, 0x0, 0x5401, 0xc00022721c)
	/snap/go/6439/src/bytes/buffer.go:202 +0x48 fp=0xc0001871d8 sp=0xc000187168 pc=0x500228
io/ioutil.readAll(0xd45420, 0xc000138000, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
	/snap/go/6439/src/io/ioutil/ioutil.go:36 +0xe5 fp=0xc000187258 sp=0xc0001871d8 pc=0x5ace25
io/ioutil.ReadAll(...)
	/snap/go/6439/src/io/ioutil/ioutil.go:45
main.(*cmdInit).create(0xc00017c510, 0xc00013b740, 0xc000229000, 0x2, 0x4, 0x0, 0x2, 0xc52500, 0x0, 0x0, ...)
	/root/lxd-pkg-snap/parts/lxd/go/src/github.com/lxc/lxd/lxc/init.go:90 +0x299f fp=0xc000187b80 sp=0xc000187258 pc=0xa8fbdf
main.(*cmdInit).Run(0xc00017c510, 0xc000280000, 0xc000229000, 0x2, 0x4, 0x0, 0x0)
	/root/lxd-pkg-snap/parts/lxd/go/src/github.com/lxc/lxd/lxc/init.go:74 +0xd6 fp=0xc000187be8 sp=0xc000187b80 pc=0xa8d1b6
main.(*cmdInit).Run-fm(0xc000280000, 0xc000229000, 0x2, 0x4, 0x0, 0x0)
	/root/lxd-pkg-snap/parts/lxd/go/src/github.com/lxc/lxd/lxc/init.go:62 +0x52 fp=0xc000187c30 sp=0xc000187be8 pc=0xae44d2
github.com/spf13/cobra.(*Command).execute(0xc000280000, 0xc000228fc0, 0x4, 0x4, 0xc000280000, 0xc000228fc0)
	/root/lxd-pkg-snap/parts/lxd/go/src/github.com/spf13/cobra/command.go:850 +0x47c fp=0xc000187d00 sp=0xc000187c30 pc=0x5ca13c
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001678c0, 0xc00013a130, 0x5, 0x5)
	/root/lxd-pkg-snap/parts/lxd/go/src/github.com/spf13/cobra/command.go:958 +0x375 fp=0xc000187dd8 sp=0xc000187d00 pc=0x5cacb5
github.com/spf13/cobra.(*Command).Execute(...)
	/root/lxd-pkg-snap/parts/lxd/go/src/github.com/spf13/cobra/command.go:895
main.main()
	/root/lxd-pkg-snap/parts/lxd/go/src/github.com/lxc/lxd/lxc/main.go:238 +0x1d1f fp=0xc000187f88 sp=0xc000187dd8 pc=0xa9a39f
runtime.main()
	/snap/go/6439/src/runtime/proc.go:204 +0x209 fp=0xc000187fe0 sp=0xc000187f88 pc=0x43a789
runtime.goexit()
	/snap/go/6439/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000187fe8 sp=0xc000187fe0 pc=0x46d401