<div dir="ltr"><div style>Hello,</div><div style><br></div><div style>I'm running into a troublesome scenario where the OOM killer is hard killing processes in my container when I write a file with size exceeding the memory limitation.</div>
<div style><br></div><div style>Scenario:</div><div style><br></div><div style>I have an LXC container where memory.limit_in_bytes is set to 300 MB.</div><div><br></div><div>I attempt to dd a ~500 MB file as follows:</div>
<div><br></div><div>dd if=/dev/zero of=test2 bs=100k count=5010<br></div><div><br></div><div style>Roughly 20% of the time, the Linux OOM manager is triggered by this command and a process is killed. Needless to say, this is highly unintended behavior; dd is meant to simulate an actual "useful" file write by a program running inside the container.</div>
<div style><br></div><div style><br></div><div style>Details:</div><div style>While file caches get large (260 MB), rss and file map seem to stay quite low. Here's an example of what memory.stat may look like during the write:</div>
<div style><div>cache 278667264</div><div>rss 20971520</div><div>mapped_file 24576</div><div>pgpgin 138147</div><div>pgpgout 64993</div><div>swap 0</div><div>pgfault 55054</div><div>pgmajfault 2</div><div>inactive_anon 10637312</div>
<div>active_anon 10342400</div><div>inactive_file 278339584</div><div>active_file 319488</div><div>unevictable 0</div><div>hierarchical_memory_limit 300003328</div><div>hierarchical_memsw_limit 300003328</div><div>total_cache 278667264</div>
<div>total_rss 20971520</div><div>total_mapped_file 24576</div><div>total_pgpgin 138147</div><div>total_pgpgout 64993</div><div>total_swap 0</div><div>total_pgfault 55054</div><div>total_pgmajfault 2</div><div>total_inactive_anon 10637312</div>
<div>total_active_anon 10342400</div><div>total_inactive_file 278339584</div><div>total_active_file 319488</div><div>total_unevictable 0</div><div><br></div></div><div style><br></div><div style><br></div><div style>Here's a paste from dmesg where the OOM triggered a kill. I'm not too familiar with the distinctions between the types of memory; one thing that stands out is that while "Node 0 Normal" is very low, there is plenty of Node 0 DMA32 memory free. Can anyone explain why a file write is causing the OOM? How do I prevent this from happening?</div>
<div style><br></div><div style>The log:</div><div style><br></div><div><div>[1801523.686755] Task in /lxc/c-7 killed as a result of limit of /lxc/c-7</div><div>[1801523.686758] memory: usage 292972kB, limit 292972kB, failcnt 39580</div>
<div>[1801523.686760] memory+swap: usage 292972kB, limit 292972kB, failcnt 0</div><div>[1801523.686762] Mem-Info:</div><div>[1801523.686764] Node 0 DMA per-cpu:</div><div>[1801523.686767] CPU 0: hi: 0, btch: 1 usd: 0</div>
<div>[1801523.686769] CPU 1: hi: 0, btch: 1 usd: 0</div><div>[1801523.686771] CPU 2: hi: 0, btch: 1 usd: 0</div><div>[1801523.686773] CPU 3: hi: 0, btch: 1 usd: 0</div><div>[1801523.686775] CPU 4: hi: 0, btch: 1 usd: 0</div>
<div>[1801523.686778] CPU 5: hi: 0, btch: 1 usd: 0</div><div>[1801523.686780] CPU 6: hi: 0, btch: 1 usd: 0</div><div>[1801523.686782] CPU 7: hi: 0, btch: 1 usd: 0</div><div>[1801523.686783] Node 0 DMA32 per-cpu:</div>
<div>[1801523.686786] CPU 0: hi: 186, btch: 31 usd: 158</div><div>[1801523.686788] CPU 1: hi: 186, btch: 31 usd: 114</div><div>[1801523.686790] CPU 2: hi: 186, btch: 31 usd: 133</div><div>[1801523.686792] CPU 3: hi: 186, btch: 31 usd: 69</div>
<div>[1801523.686794] CPU 4: hi: 186, btch: 31 usd: 70</div><div>[1801523.686796] CPU 5: hi: 186, btch: 31 usd: 131</div><div>[1801523.686798] CPU 6: hi: 186, btch: 31 usd: 169</div><div>[1801523.686800] CPU 7: hi: 186, btch: 31 usd: 30</div>
<div>[1801523.686802] Node 0 Normal per-cpu:</div><div>[1801523.686804] CPU 0: hi: 186, btch: 31 usd: 162</div><div>[1801523.686806] CPU 1: hi: 186, btch: 31 usd: 184</div><div>[1801523.686809] CPU 2: hi: 186, btch: 31 usd: 99</div>
<div>[1801523.686811] CPU 3: hi: 186, btch: 31 usd: 82</div><div>[1801523.686813] CPU 4: hi: 186, btch: 31 usd: 90</div><div>[1801523.686815] CPU 5: hi: 186, btch: 31 usd: 99</div><div>[1801523.686817] CPU 6: hi: 186, btch: 31 usd: 157</div>
<div>[1801523.686819] CPU 7: hi: 186, btch: 31 usd: 138</div><div>[1801523.686824] active_anon:60439 inactive_anon:28841 isolated_anon:0</div><div>[1801523.686825] active_file:110417 inactive_file:907078 isolated_file:64</div>
<div>[1801523.686827] unevictable:0 dirty:164722 writeback:1652 unstable:0</div><div>[1801523.686828] free:445909 slab_reclaimable:176594 slab_unreclaimable:14754</div><div>[1801523.686829] mapped:4753 shmem:66 pagetables:3600 bounce:0</div>
<div>[1801523.686831] Node 0 DMA free:7904kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:7648kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no</div>
<div>[1801523.686841] lowmem_reserve[]: 0 4016 7048 7048</div><div>[1801523.686845] Node 0 DMA32 free:1770072kB min:6116kB low:7644kB high:9172kB active_anon:22312kB inactive_anon:12128kB active_file:4988kB inactive_file:2190136kB unevictable:0kB isolated(anon):0kB isolated(file):256kB present:4112640kB mlocked:0kB dirty:535072kB writeback:6452kB mapped:4kB shmem:4kB slab_reclaimable:72888kB slab_unreclaimable:1100kB kernel_stack:120kB pagetables:832kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no</div>
<div>[1801523.686855] lowmem_reserve[]: 0 0 3031 3031</div><div>[1801523.686859] Node 0 Normal free:5660kB min:4616kB low:5768kB high:6924kB active_anon:219444kB inactive_anon:103236kB active_file:436680kB inactive_file:1438176kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3104640kB mlocked:0kB dirty:123816kB writeback:156kB mapped:19008kB shmem:260kB slab_reclaimable:633488kB slab_unreclaimable:57916kB kernel_stack:2800kB pagetables:13568kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no</div>
<div>[1801523.686869] lowmem_reserve[]: 0 0 0 0</div><div>[1801523.686873] Node 0 DMA: 2*4kB 3*8kB 0*16kB 2*32kB 4*64kB 3*128kB 2*256kB 1*512kB 2*1024kB 2*2048kB 0*4096kB = 7904kB</div><div>[1801523.686883] Node 0 DMA32: 129*4kB 87*8kB 86*16kB 89*32kB 87*64kB 65*128kB 12*256kB 5*512kB 2*1024kB 13*2048kB 419*4096kB = 1769852kB</div>
<div>[1801523.686893] Node 0 Normal: 477*4kB 23*8kB 1*16kB 5*32kB 0*64kB 3*128kB 3*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 5980kB</div><div>[1801523.686903] 1017542 total pagecache pages</div><div>[1801523.686905] 0 pages in swap cache</div>
<div>[1801523.686907] Swap cache stats: add 0, delete 0, find 0/0</div><div>[1801523.686908] Free swap = 1048572kB</div><div>[1801523.686910] Total swap = 1048572kB</div><div>[1801523.722319] 1837040 pages RAM</div><div>
[1801523.722322] 58337 pages reserved</div><div>[1801523.722323] 972948 pages shared</div><div>[1801523.722324] 406948 pages non-shared</div><div>[1801523.722326] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name</div>
<div>[1801523.722396] [31266] 0 31266 6404 511 6 0 0 init</div><div>[1801523.722445] [32489] 0 32489 12370 688 7 -17 -1000 sshd</div><div>[1801523.722460] [32511] 101 32511 10513 325 0 0 0 rsyslogd</div>
<div>[1801523.722495] [32625] 0 32625 17706 838 2 0 0 sshd</div><div>[1801523.722522] [32652] 103 32652 5900 176 0 0 0 dbus-daemon</div><div>[1801523.722583] [ 526] 0 526 1553 168 5 0 0 getty</div>
<div>[1801523.722587] [ 530] 0 530 1553 168 1 0 0 getty</div><div>[1801523.722593] [ 537] 2007 537 17706 423 5 0 0 sshd</div><div>[1801523.722629] [ 538] 2007 538 16974 5191 1 0 0 python</div>
<div>[1801523.722650] [ 877] 2007 877 2106 157 7 0 0 dd</div><div>[1801523.722657] Memory cgroup out of memory: Kill process 538 (python) score 71 or sacrifice child</div><div>[1801523.722674] Killed process 538 (python) total-vm:67896kB, anon-rss:17464kB, file-rss:3300kB</div>
<div><br></div><div style>I'm running on Linux ip-10-8-139-98 3.2.0-29-virtual #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux on Amazon EC2.</div><div><br></div><div><br></div><div style>Thanks for any help you can provide,</div>
<div style>Aaron Staley</div>
</div></div>