<div dir="ltr">Hi,<div><br></div><div>there is a server that currently runs ~100 containers.</div><div><br></div><div>One of those containers is a subject of my interest.</div><div><br></div><div>Brief details about the container: it runs ubuntu xenial, and it's a tiny file server (samba based) with near to no traffic at all.</div><div><br></div><div>I have found that after you upload files to that server, the available memory size is decreased (while the "buff/cache" size stays at 0). And if you remove the just uploaded files - the memory consumption drops to the same value as it was before uploading.</div><div><br></div><div>Here is a output of the top (sorted by resident memory size, processes with more than 500kib rss):</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div></div><div><div> 48 root 20 0 52048 18880 14428 S 0.0 0.9 0:10.11 /lib/systemd/systemd-journald</div></div><div><div>18609 www-data 20 0 349208 15404 7516 S 0.0 0.7 0:13.72 /usr/sbin/smbd -D</div></div><div><div> 7176 www-data 20 0 345500 10720 6720 S 6.7 0.5 0:06.91 /usr/sbin/smbd -D</div></div><div><div>25124 root 20 0 340104 9624 6744 S 0.0 0.5 0:00.12 /usr/sbin/smbd -D</div></div><div><div>37541 root 20 0 344828 8012 4520 S 0.0 0.4 0:02.36 /usr/sbin/smbd -D</div></div><div><div>15593 root 20 0 344352 6368 3444 S 0.0 0.3 0:00.39 /usr/sbin/smbd -D</div></div><div><div> 2450 root 20 0 336636 4072 1520 S 0.0 0.2 0:06.09 /usr/sbin/smbd -D</div></div><div><div>25401 root 20 0 40560 3728 3112 R 0.3 0.2 0:00.49 top</div></div><div><div> 2447 root 20 0 336636 3528 976 S 0.0 0.2 0:04.30 /usr/sbin/smbd -D</div></div><div><div>25287 root 20 0 19972 3044 2872 S 0.0 0.1 0:00.01 bash</div></div><div><div> 2476 root 20 0 238728 2944 1336 S 0.0 0.1 0:28.52 /usr/sbin/nmbd -D</div></div><div><div>25271 ivan 20 0 21328 2784 2764 S 0.0 0.1 0:00.04 -bash</div></div><div><div>24250 root 20 0 858936 2616 0 S 0.0 0.1 0:01.98 /usr/sbin/collectd</div></div><div><div> 2448 root 20 0 426848 2504 20 S 0.3 0.1 0:01.65 /usr/sbin/smbd -D</div></div><div><div> 1 root 20 0 37884 2488 1676 S 0.0 0.1 0:17.13 /sbin/init</div></div><div><div>25285 root 20 0 51660 2404 2400 S 0.0 0.1 0:00.00 sudo su</div></div><div><div>25270 ivan 20 0 95368 2172 1960 S 0.0 0.1 0:00.24 sshd: ivan@pts/0</div></div><div><div>25286 root 20 0 51008 1908 1908 S 0.0 0.1 0:00.00 su</div></div><div><div> 8041 zabbix 20 0 95520 1680 1512 S 0.0 0.1 0:02.10 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]</div></div><div><div>25240 root 20 0 95368 1620 1572 S 0.0 0.1 0:00.02 sshd: ivan [priv]</div></div><div><div> 145 message+ 20 0 42892 1164 872 S 0.0 0.1 0:01.55 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation</div></div><div><div> 6453 www-data 20 0 125348 1152 656 S 0.0 0.1 0:32.26 nginx: worker process</div></div><div><div>20811 postfix 20 0 67640 1136 656 S 0.0 0.1 0:00.86 qmgr -l -t unix -u</div></div><div><div> 8038 zabbix 20 0 95520 1084 880 S 0.0 0.1 0:01.04 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]</div></div><div><div> 8039 zabbix 20 0 95520 972 768 S 0.0 0.0 0:01.05 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]</div></div><div><div> 142 root 20 0 27732 924 636 S 0.0 0.0 0:05.32 /usr/sbin/cron -f</div></div><div><div> 8040 zabbix 20 0 95520 872 668 S 0.0 0.0 0:01.07 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]</div></div><div><div> 8037 zabbix 20 0 93444 728 628 S 0.0 0.0 0:14.57 /usr/sbin/zabbix_agentd: collector [idle 1 sec]</div></div><div><div> 6462 www-data 20 0 125348 500 0 S 0.0 0.0 0:32.14 nginx: worker process</div></div></blockquote><div><div><br></div><div><br></div><div>As you can see - the cumulative RSS is could barely get to the 100mb.</div><div><br></div><div>While this is what `free` returns:</div><div><br></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div># free -m</div></div></div><div><div><div> total used free shared buff/cache available</div></div></div><div><div><div>Mem: 2048 1785 261 1750 0 261</div></div></div><div><div><div>Swap: 512 14 497</div></div></div></blockquote><div><div><br></div><div>So, it clearly states about 85% of ram is occupied.</div><div><br></div><div>`slabtop` (due to cgroup limitations?) does not work:</div><div><br></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div># slabtop</div></div><div><div><div>fopen /proc/slabinfo: Permission denied</div></div></div></blockquote><div><div><br></div><div>But if I clear the system caches on the host</div><div><br></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div>echo 3 > /proc/sys/vm/drop_caches </div></div></blockquote><div><div><br></div><div>the container memory consumption drops to the expected <100mb.</div><div><br></div><div>So the question, how to monitor the memory consumption from the container reliably? And why does `free` count caches as used memory inside container?</div><div><br></div><div><br></div><div><br></div>-- <br><div class="gmail_signature">With best regards, Ivan Kurnosov</div>
</div></div>