[Lxc-users] Do not understand why OOM killer is being triggered by a DD
Aaron Staley
aaron at picloud.com
Mon Jun 17 16:05:41 UTC 2013
I've also asked this question on serverfault:
http://serverfault.com/questions/516074/why-can-applications-writing-large-files-in-a-memory-limited-lxc-container-get-k
The
answer remains inconclusive, but I believe there may be issues with how
file I/o caching is handled under the limitations on memory imposed on the
the lxc cgroup (cgroup.memory.limit_in_bytes) A similar thing seems to be
happening in this post:
http://serverfault.com/questions/488014/limit-private-memory-usage-per-user
As a simple way to reproduce this issue, you can do the following:
create an empty lxc container (in my case I did lxc-create -n testcon -t
ubuntu -- -r precise)
Modify the configuration of the container to set lxc.cgroup.memory.
limit_in_bytes = 300M
After starting the container, run dd if=/dev/zero of=test2 bs=100k
count=5010
While this command will most likely succeed, you will see that there are
memory allocation failures in memory.failcnt.
However, I have found that when I run such a command on an instance that
actually has low amounts of memory (say the t1.micro with only 590 MB),
allocation failures are much rarer (<30%) and I cannot reproduce an actual
crash out. The fact that errors are less frequent when hardware memory is
smaller suggest that the application/kernel's I/O caching manager is aware
of the actual memory limitations on the underlying hardware, but is unaware
of the container limitation.
In my case, I seek to use lxc as a way to split a single machine into
multiple virtualized containers with proportionate resources.
Unfortunately, there appear to be fundamental problems with my method of
imposing memory limitations on the containers; too much memory is being
allocated for I/O cache and when multiple containers are competing for I/O
cache memory, processes start to be killed.
I'd be surprised if this hasn't affected more people. Does anyone have an
idea for a workaround? Am I missing some needed configuration setting or is
there something more?
Thanks,
Aaron
> Message: 1
> Date: Fri, 14 Jun 2013 19:40:24 -0700
> From: Aaron Staley <aaron at picloud.com>
> Subject: [Lxc-users] Do not understand why OOM killer is being
> triggered by a DD
> To: lxc-users at lists.sourceforge.net
> Message-ID:
> <
> CAMcjixYO1d8HF0_hnJLJJEMLUG3Qayps8gzCbtGcZj-UyfWyTA at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hello,
>
> I'm running into a troublesome scenario where the OOM killer is hard
> killing processes in my container when I write a file with size exceeding
> the memory limitation.
>
> Scenario:
>
> I have an LXC container where memory.limit_in_bytes is set to 300 MB.
>
> I attempt to dd a ~500 MB file as follows:
>
> dd if=/dev/zero of=test2 bs=100k count=5010
>
> Roughly 20% of the time, the Linux OOM manager is triggered by this command
> and a process is killed. Needless to say, this is highly unintended
> behavior; dd is meant to simulate an actual "useful" file write by a
> program running inside the container.
>
>
> Details:
> While file caches get large (260 MB), rss and file map seem to stay quite
> low. Here's an example of what memory.stat may look like during the write:
> cache 278667264
> rss 20971520
> mapped_file 24576
> pgpgin 138147
> pgpgout 64993
> swap 0
> pgfault 55054
> pgmajfault 2
> inactive_anon 10637312
> active_anon 10342400
> inactive_file 278339584
> active_file 319488
> unevictable 0
> hierarchical_memory_limit 300003328
> hierarchical_memsw_limit 300003328
> total_cache 278667264
> total_rss 20971520
> total_mapped_file 24576
> total_pgpgin 138147
> total_pgpgout 64993
> total_swap 0
> total_pgfault 55054
> total_pgmajfault 2
> total_inactive_anon 10637312
> total_active_anon 10342400
> total_inactive_file 278339584
> total_active_file 319488
> total_unevictable 0
>
>
>
> Here's a paste from dmesg where the OOM triggered a kill. I'm not too
> familiar with the distinctions between the types of memory; one thing that
> stands out is that while "Node 0 Normal" is very low, there is plenty of
> Node 0 DMA32 memory free. Can anyone explain why a file write is causing
> the OOM? How do I prevent this from happening?
>
> The log:
>
> [1801523.686755] Task in /lxc/c-7 killed as a result of limit of /lxc/c-7
> [1801523.686758] memory: usage 292972kB, limit 292972kB, failcnt 39580
> [1801523.686760] memory+swap: usage 292972kB, limit 292972kB, failcnt 0
> [1801523.686762] Mem-Info:
> [1801523.686764] Node 0 DMA per-cpu:
> [1801523.686767] CPU 0: hi: 0, btch: 1 usd: 0
> [1801523.686769] CPU 1: hi: 0, btch: 1 usd: 0
> [1801523.686771] CPU 2: hi: 0, btch: 1 usd: 0
> [1801523.686773] CPU 3: hi: 0, btch: 1 usd: 0
> [1801523.686775] CPU 4: hi: 0, btch: 1 usd: 0
> [1801523.686778] CPU 5: hi: 0, btch: 1 usd: 0
> [1801523.686780] CPU 6: hi: 0, btch: 1 usd: 0
> [1801523.686782] CPU 7: hi: 0, btch: 1 usd: 0
> [1801523.686783] Node 0 DMA32 per-cpu:
> [1801523.686786] CPU 0: hi: 186, btch: 31 usd: 158
> [1801523.686788] CPU 1: hi: 186, btch: 31 usd: 114
> [1801523.686790] CPU 2: hi: 186, btch: 31 usd: 133
> [1801523.686792] CPU 3: hi: 186, btch: 31 usd: 69
> [1801523.686794] CPU 4: hi: 186, btch: 31 usd: 70
> [1801523.686796] CPU 5: hi: 186, btch: 31 usd: 131
> [1801523.686798] CPU 6: hi: 186, btch: 31 usd: 169
> [1801523.686800] CPU 7: hi: 186, btch: 31 usd: 30
> [1801523.686802] Node 0 Normal per-cpu:
> [1801523.686804] CPU 0: hi: 186, btch: 31 usd: 162
> [1801523.686806] CPU 1: hi: 186, btch: 31 usd: 184
> [1801523.686809] CPU 2: hi: 186, btch: 31 usd: 99
> [1801523.686811] CPU 3: hi: 186, btch: 31 usd: 82
> [1801523.686813] CPU 4: hi: 186, btch: 31 usd: 90
> [1801523.686815] CPU 5: hi: 186, btch: 31 usd: 99
> [1801523.686817] CPU 6: hi: 186, btch: 31 usd: 157
> [1801523.686819] CPU 7: hi: 186, btch: 31 usd: 138
> [1801523.686824] active_anon:60439 inactive_anon:28841 isolated_anon:0
> [1801523.686825] active_file:110417 inactive_file:907078 isolated_file:64
> [1801523.686827] unevictable:0 dirty:164722 writeback:1652 unstable:0
> [1801523.686828] free:445909 slab_reclaimable:176594
> slab_unreclaimable:14754
> [1801523.686829] mapped:4753 shmem:66 pagetables:3600 bounce:0
> [1801523.686831] Node 0 DMA free:7904kB min:8kB low:8kB high:12kB
> active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
> unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7648kB
> mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
> slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB
> unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0
> all_unreclaimable? no
> [1801523.686841] lowmem_reserve[]: 0 4016 7048 7048
> [1801523.686845] Node 0 DMA32 free:1770072kB min:6116kB low:7644kB
> high:9172kB active_anon:22312kB inactive_anon:12128kB active_file:4988kB
> inactive_file:2190136kB unevictable:0kB isolated(anon):0kB
> isolated(file):256kB present:4112640kB mlocked:0kB dirty:535072kB
> writeback:6452kB mapped:4kB shmem:4kB slab_reclaimable:72888kB
> slab_unreclaimable:1100kB kernel_stack:120kB pagetables:832kB unstable:0kB
> bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
> [1801523.686855] lowmem_reserve[]: 0 0 3031 3031
> [1801523.686859] Node 0 Normal free:5660kB min:4616kB low:5768kB
> high:6924kB active_anon:219444kB inactive_anon:103236kB
> active_file:436680kB inactive_file:1438176kB unevictable:0kB
> isolated(anon):0kB isolated(file):0kB present:3104640kB mlocked:0kB
> dirty:123816kB writeback:156kB mapped:19008kB shmem:260kB
> slab_reclaimable:633488kB slab_unreclaimable:57916kB kernel_stack:2800kB
> pagetables:13568kB unstable:0kB bounce:0kB writeback_tmp:0kB
> pages_scanned:0 all_unreclaimable? no
> [1801523.686869] lowmem_reserve[]: 0 0 0 0
> [1801523.686873] Node 0 DMA: 2*4kB 3*8kB 0*16kB 2*32kB 4*64kB 3*128kB
> 2*256kB 1*512kB 2*1024kB 2*2048kB 0*4096kB = 7904kB
> [1801523.686883] Node 0 DMA32: 129*4kB 87*8kB 86*16kB 89*32kB 87*64kB
> 65*128kB 12*256kB 5*512kB 2*1024kB 13*2048kB 419*4096kB = 1769852kB
> [1801523.686893] Node 0 Normal: 477*4kB 23*8kB 1*16kB 5*32kB 0*64kB 3*128kB
> 3*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 5980kB
> [1801523.686903] 1017542 total pagecache pages
> [1801523.686905] 0 pages in swap cache
> [1801523.686907] Swap cache stats: add 0, delete 0, find 0/0
> [1801523.686908] Free swap = 1048572kB
> [1801523.686910] Total swap = 1048572kB
> [1801523.722319] 1837040 pages RAM
> [1801523.722322] 58337 pages reserved
> [1801523.722323] 972948 pages shared
> [1801523.722324] 406948 pages non-shared
> [1801523.722326] [ pid ] uid tgid total_vm rss cpu oom_adj
> oom_score_adj name
> [1801523.722396] [31266] 0 31266 6404 511 6 0
> 0 init
> [1801523.722445] [32489] 0 32489 12370 688 7 -17
> -1000 sshd
> [1801523.722460] [32511] 101 32511 10513 325 0 0
> 0 rsyslogd
> [1801523.722495] [32625] 0 32625 17706 838 2 0
> 0 sshd
> [1801523.722522] [32652] 103 32652 5900 176 0 0
> 0 dbus-daemon
> [1801523.722583] [ 526] 0 526 1553 168 5 0
> 0 getty
> [1801523.722587] [ 530] 0 530 1553 168 1 0
> 0 getty
> [1801523.722593] [ 537] 2007 537 17706 423 5 0
> 0 sshd
> [1801523.722629] [ 538] 2007 538 16974 5191 1 0
> 0 python
> [1801523.722650] [ 877] 2007 877 2106 157 7 0
> 0 dd
> [1801523.722657] Memory cgroup out of memory: Kill process 538 (python)
> score 71 or sacrifice child
> [1801523.722674] Killed process 538 (python) total-vm:67896kB,
> anon-rss:17464kB, file-rss:3300kB
>
> I'm running on Linux ip-10-8-139-98 3.2.0-29-virtual #46-Ubuntu SMP Fri Jul
> 27 17:23:50 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux on Amazon EC2.
>
>
> Thanks for any help you can provide,
> Aaron Staley
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20130617/817f96d4/attachment.html>
More information about the lxc-users
mailing list