<div dir="ltr"><div class="gmail_default" style="font-size:small">I have a lot of memory management issues using pure LXC. In my case, my box has only one container. I use LXC to be able to move my app around, not to squeeze performance out of hardware. What happens is my database gets killed the OOM manager, although there are gigabytes of RAM used for cache. The memory manager kills applications instead of reclaiming memory from disc cache. How can this be avoided?</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">My config at the host is:</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><div class="gmail_default">vm.hugepages_treat_as_movable=0</div><div class="gmail_default">vm.hugetlb_shm_group=27</div><div class="gmail_default">vm.nr_hugepages=2500</div><div class="gmail_default">vm.nr_hugepages_mempolicy=2500</div><div class="gmail_default">vm.nr_overcommit_hugepages=0</div><div class="gmail_default">vm.overcommit_memory=0</div><div class="gmail_default">vm.swappiness=0<br></div><div class="gmail_default">vm.vfs_cache_pressure=150</div><div class="gmail_default">vm.dirty_ratio=10<br></div><div class="gmail_default"><div class="gmail_default">vm.dirty_background_ratio=5</div><div class="gmail_default"><br></div><div class="gmail_default">This shows the issue</div><div class="gmail_default"><div class="gmail_default"><font face="monospace, monospace">[9449866.130270] Node 0 hugepages_total=1250 hugepages_free=1250 hugepages_surp=0 hugepages_size=2048kB</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130271] Node 1 hugepages_total=1250 hugepages_free=1248 hugepages_surp=0 hugepages_size=2048kB</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130271] 46181 total pagecache pages</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130273] 33203 pages in swap cache</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130274] Swap cache stats: add 248571542, delete 248538339, find 69031185/100062903</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130274] Free swap = 0kB</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130275] Total swap = 8305660kB</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130276] 20971279 pages RAM</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130276] 0 pages HighMem/MovableOnly</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130276] 348570 pages reserved</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130277] 0 pages cma reserved</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130277] 0 pages hwpoisoned</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130278] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130286] [ 618] 0 618 87181 135 168 3 3 0 systemd-journal</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130288] [ 825] 0 825 11343 130 25 3 0 0 systemd-logind</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130289] [ 830] 0 830 1642 31 8 3 0 0 mcelog</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130290] [ 832] 996 832 26859 51 23 3 47 0 chronyd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130292] [ 834] 0 834 4905 100 12 3 0 0 irqbalance</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130293] [ 835] 0 835 6289 177 15 3 0 0 smartd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130295] [ 837] 81 837 28499 258 28 3 149 -900 dbus-daemon</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130296] [ 857] 0 857 1104 16 7 3 0 0 rngd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130298] [ 859] 0 859 192463 37114 224 4 40630 0 NetworkManager</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130300] [ 916] 0 916 25113 229 50 3 0 -1000 sshd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130302] [ 924] 0 924 6490 50 17 3 0 0 atd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130303] [ 929] 0 929 35327 199 20 3 284 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130305] [ 955] 0 955 22199 3185 43 3 312 0 dhclient</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130307] [ 1167] 0 1167 6125 88 17 3 2 0 lxc-autostart</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130309] [ 1176] 0 1176 10818 275 24 3 38 0 systemd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130310] [ 1188] 0 1188 13303 1980 29 3 36 0 systemd-journal</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130312] [ 1372] 99 1372 3881 2 12 3 45 0 dnsmasq</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130313] [ 1375] 81 1375 6108 77 17 3 39 -900 dbus-daemon</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130315] [ 1394] 0 1394 6175 46 15 3 168 0 systemd-logind</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130316] [ 1395] 0 1395 78542 1142 69 3 4 0 rsyslogd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130317] [ 1397] 0 1397 1614 32 8 3 0 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130319] [ 1398] 0 1398 1614 31 8 3 0 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130320] [ 1400] 0 1400 1614 31 8 3 0 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130321] [ 1401] 0 1401 1614 2 8 3 30 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130322] [ 1402] 0 1402 1614 2 8 3 29 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130324] [ 1403] 0 1403 1614 31 8 3 0 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130325] [ 1404] 0 1404 1614 32 8 3 0 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130327] [ 1405] 0 1405 1614 32 8 3 0 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130328] [ 1406] 0 1406 1614 2 8 3 29 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130329] [ 1408] 0 1408 1614 2 8 3 30 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130330] [ 1409] 0 1409 1614 30 7 3 0 0 agetty</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130332] [18224] 0 18224 26456 0 43 4 404 0 VGAuthService</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130333] [18225] 0 18225 61032 95 58 3 258 0 vmtoolsd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130335] [28660] 0 28660 26372 44 54 4 202 -1000 sshd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130337] [18992] 998 18992 132859 876 54 3 13 0 polkitd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130339] [23849] 0 23849 10744 370 23 3 0 -1000 systemd-udevd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130340] [ 3484] 0 3484 184082 265 243 4 129 0 rsyslogd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130342] [31175] 32 31175 14328 35 30 3 102 0 rpcbind</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130344] [31205] 0 31205 111747 0 65 3 343 0 abrtd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130345] [31248] 0 31248 187819 30 167 4 291 0 abrt-dump-journ</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130347] [31303] 0 31303 172125 32 120 4 258 0 abrt-dump-journ</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130348] [16252] 0 16252 5659 129 17 3 24 0 crond</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130350] [11626] 0 11626 33235 25 15 3 130 0 crond</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130351] [11717] 0 11717 13897 109 26 3 3 -1000 auditd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130353] [31764] 0 31764 51162 73 36 3 50 0 gssproxy</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130355] [31372] 0 31372 11441 557 21 3 0 -1000 systemd-udevd</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130357] [12835] 27 12835 23997106 18960777 43207 119 2018937 0 mysqld</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130359] [ 8109] 0 8109 17597 220 39 3 3 0 crond</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130361] [ 8185] 0 8185 28282 57 10 3 0 0 safe_asterisk1</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130363] [ 8186] 0 8186 901036 8104 344 7 3 0 asterisk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130364] [ 8215] 0 8215 28282 57 10 4 0 0 safe_asterisk2</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130366] [ 8216] 0 8216 917544 8133 345 6 19 0 asterisk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130367] [ 8265] 0 8265 28282 57 10 3 0 0 safe_asterisk3</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130369] [ 8266] 0 8266 934048 8203 347 7 1 0 asterisk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130370] [ 8351] 0 8351 28282 57 10 3 0 0 safe_asterisk4</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130372] [ 8353] 0 8353 950557 8235 349 6 15 0 asterisk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130373] [ 8400] 0 8400 28282 57 10 3 0 0 safe_asterisk5</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130375] [ 8401] 0 8401 901033 8122 345 7 1 0 asterisk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130376] [ 8460] 0 8460 28282 57 9 3 0 0 safe_asterisk6</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130377] [ 8461] 0 8461 1148653 8136 361 8 47 0 asterisk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130379] [ 8537] 0 8537 28282 57 9 3 0 0 safe_asterisk7</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130403] [ 8538] 0 8538 983574 8148 349 6 2 0 asterisk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130405] [ 8594] 0 8594 28282 58 10 3 0 0 safe_asterisk8</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130406] [ 8596] 0 8596 950555 8131 350 7 61 0 asterisk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130408] [ 8649] 0 8649 28282 57 9 4 0 0 safe_asterisk9</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130409] [ 8651] 0 8651 901033 8139 345 6 0 0 asterisk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130411] [ 8712] 0 8712 28282 58 10 3 0 0 safe_asterisk10</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130413] [ 8714] 0 8714 901034 8114 342 6 14 0 asterisk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130415] [14800] 0 14800 31930 117 18 3 0 0 screen</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130416] [14801] 0 14801 28284 55 12 3 0 0 audit.sh</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130419] [17583] 0 17583 28284 55 10 3 0 0 audit.sh</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130421] [17584] 0 17584 40172 279 35 3 0 0 mysql</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130422] [17585] 0 17585 28373 22 12 3 0 0 awk</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130424] Out of memory: Kill process 12835 (mysqld) score 926 or sacrifice child</font></div><div class="gmail_default"><font face="monospace, monospace">[9449866.130661] Killed process 12835 (mysqld) total-vm:95988424kB, anon-rss:75843108kB, file-rss:0kB, shmem-rss:0kB</font></div><div class="gmail_default"><font face="monospace, monospace">[9449872.448957] oom_reaper: reaped process 12835 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB</font></div></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace">Philip</font></div><div><font face="monospace, monospace"><br></font></div></div><div class="gmail_default"><br></div><div class="gmail_default"><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jul 15, 2017 at 5:11 AM, Marat Khalili <span dir="ltr"><<a href="mailto:mkh@rqc.ru" target="_blank">mkh@rqc.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>I'm using LXC, and I frequently observe some unused containers get swapped out, even though system has plenty of RAM and no RAM limits are set. The only bad effect I observe is couple of seconds delay when you first log into them after some time. I guess it is absolutely normal since kernel tries to maximize amount of memory available for disk caches.<br>
<br>
If you don't like this behavior, instead of trying to fine tune kernel parameters why not disable swap altogether? Many people run it this way, it's mostly a matter of taste these days. (But first check your software for leaks.)<span class=""><br>
<br>
> For example, our “server-4” machine shows 8G total RAM, 500MB free, 2.5G available, and 5G of buff/cache. Yet, swap is at 5.5GB and has been slowly growing over the past few days. It seems something is preventing the apps from using the RAM.<br>
<br></span>
Did you identify what processes all this virtual memory belongs to?<span class=""><br>
<br>
> To be honest, we have been battling lots of memory/swap issues using LXD. We started with no tuning, but the app stack quickly ran out of memory. <br>
<br></span>
LXC/LXD is hardly responsible for your app stack memory usage. Either you underestimated it or there's a memory leak somewhere.<span class=""><br>
<br>
> Given all the issues we have had with memory and swap using LXD, we are seriously considering moving back to the traditional VM approach until LXC/LXD is better “baked”.<br>
<br></span>
Did your VMs use less memory? I don't think so. Limits could be better enforced, but VMs don't magically give you infinite RAM. <br><span class="HOEnZb"><font color="#888888">
-- <br>
<br>
With Best Regards,<br>
Marat Khalili<br><br></font></span><div class="gmail_quote"><span class="">On July 14, 2017 9:58:57 PM GMT+03:00, Ron Kelley <<a href="mailto:rkelleyrtp@gmail.com" target="_blank">rkelleyrtp@gmail.com</a>> wrote:</span><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<pre class="m_-522358098055299643k9mail"><span class="">Wondering if anyone else has similar issues.<br><br>We have 5x LXD 2.12 servers running (U16.04 - kernel 4.4.0-57-generic - 8G RAM, 19G SWAP). Each server is running about 50 LXD containers - Wordpress w/Nginx and PHP7. The servers have been running for about 15 days now, and swap space continues to grow. In addition, the kswapd0 process starts consuming CPU until we flush the system cache via "/bin/echo 3 > /proc/sys/vm/drop_caches” command.<br><br>Our LXD profile looks like this:<br>-------------------------<br>config:<br> limits.cpu: "2"<br> limits.memory: 512MB<br> limits.memory.swap: "true"<br> limits.memory.swap.priority: "1"<br>-------------------------<br><br><br>We also have added these to /etc/sysctl.conf<br>-------------------------<br>vm.swappiness=10<br>vm.vfs_cache_pressure=50<br>-------------------------<br><br>A quick “top” output shows plenty of available Memory and buff/cache. But, for some reason, the system continues to swap out the app. For example, our “server-4” machine shows 8G total RAM, 500MB free, 2.5G available, and 5G of buff/cache. Yet, swap is at 5.5GB and has been slowly growing over the past few days. It seems something is preventing the apps from using the RAM.<br><br><br>To be honest, we have been battling lots of memory/swap issues using LXD. We started with no tuning, but the app stack quickly ran out of memory. After editing the profile to allow 512MB RAM per container (and restarting the container), the kswapd0 issue happens. Given all the issues we have had with memory and swap using LXD, we are seriously considering moving back to the traditional VM approach until LXC/LXD is better “baked”.<br><br><br>-Ron<br><hr><br></span><span class="">lxc-users mailing list<br><a href="mailto:lxc-users@lists.linuxcontainers.org" target="_blank">lxc-users@lists.<wbr>linuxcontainers.org</a><br><a href="http://lists.linuxcontainers.org/listinfo/lxc-users" target="_blank">http://lists.linuxcontainers.<wbr>org/listinfo/lxc-users</a></span></pre></blockquote></div></div><br>______________________________<wbr>_________________<br>
lxc-users mailing list<br>
<a href="mailto:lxc-users@lists.linuxcontainers.org">lxc-users@lists.<wbr>linuxcontainers.org</a><br>
<a href="http://lists.linuxcontainers.org/listinfo/lxc-users" rel="noreferrer" target="_blank">http://lists.linuxcontainers.<wbr>org/listinfo/lxc-users</a><br></blockquote></div><br></div>