<div dir="ltr">Here I attach the 2 files with the output of those commands: one.txt is right after I copied several gigabytes of files to that container (using samba), two.txt is after I removed it.<div><br></div><div>It looks like the most relevant number that is changed is `total_cache`</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 27 September 2017 at 02:24, Stéphane Graber <span dir="ltr"><<a href="mailto:stgraber@ubuntu.com" target="_blank">stgraber@ubuntu.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
This sounds like a lxcfs issue.<br>
<br>
Can you file a bug at <a href="https://github.com/lxc/lxcfs" rel="noreferrer" target="_blank">https://github.com/lxc/lxcfs</a> or find one which<br>
matches your symptoms.<br>
<br>
We'll want at least:<br>
<br>
- /proc/meminfo from the container<br>
- /sys/fs/cgroup/memory/lxc/<wbr>CONTAINER/memory.usage_in_<wbr>bytes from the host<br>
- /sys/fs/cgroup/memory/lxc/<wbr>CONTAINER/memory.stat from the host<br>
<br>
That should let us track down where the memory usage comes from and what<br>
may be wrong with it.<br>
<div class="HOEnZb"><div class="h5"><br>
On Wed, Sep 20, 2017 at 01:18:01PM +1200, Ivan Kurnosov wrote:<br>
> Hi,<br>
><br>
> there is a server that currently runs ~100 containers.<br>
><br>
> One of those containers is a subject of my interest.<br>
><br>
> Brief details about the container: it runs ubuntu xenial, and it's a tiny<br>
> file server (samba based) with near to no traffic at all.<br>
><br>
> I have found that after you upload files to that server, the available<br>
> memory size is decreased (while the "buff/cache" size stays at 0). And if<br>
> you remove the just uploaded files - the memory consumption drops to the<br>
> same value as it was before uploading.<br>
><br>
> Here is a output of the top (sorted by resident memory size, processes with<br>
> more than 500kib rss):<br>
><br>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
> 48 root 20 0 52048 18880 14428 S 0.0 0.9 0:10.11<br>
> /lib/systemd/systemd-journald<br>
> 18609 www-data 20 0 349208 15404 7516 S 0.0 0.7 0:13.72<br>
> /usr/sbin/smbd -D<br>
> 7176 www-data 20 0 345500 10720 6720 S 6.7 0.5 0:06.91<br>
> /usr/sbin/smbd -D<br>
> 25124 root 20 0 340104 9624 6744 S 0.0 0.5 0:00.12<br>
> /usr/sbin/smbd -D<br>
> 37541 root 20 0 344828 8012 4520 S 0.0 0.4 0:02.36<br>
> /usr/sbin/smbd -D<br>
> 15593 root 20 0 344352 6368 3444 S 0.0 0.3 0:00.39<br>
> /usr/sbin/smbd -D<br>
> 2450 root 20 0 336636 4072 1520 S 0.0 0.2 0:06.09<br>
> /usr/sbin/smbd -D<br>
> 25401 root 20 0 40560 3728 3112 R 0.3 0.2 0:00.49 top<br>
> 2447 root 20 0 336636 3528 976 S 0.0 0.2 0:04.30<br>
> /usr/sbin/smbd -D<br>
> 25287 root 20 0 19972 3044 2872 S 0.0 0.1 0:00.01 bash<br>
> 2476 root 20 0 238728 2944 1336 S 0.0 0.1 0:28.52<br>
> /usr/sbin/nmbd -D<br>
> 25271 ivan 20 0 21328 2784 2764 S 0.0 0.1 0:00.04 -bash<br>
> 24250 root 20 0 858936 2616 0 S 0.0 0.1 0:01.98<br>
> /usr/sbin/collectd<br>
> 2448 root 20 0 426848 2504 20 S 0.3 0.1 0:01.65<br>
> /usr/sbin/smbd -D<br>
> 1 root 20 0 37884 2488 1676 S 0.0 0.1 0:17.13<br>
> /sbin/init<br>
> 25285 root 20 0 51660 2404 2400 S 0.0 0.1 0:00.00 sudo su<br>
> 25270 ivan 20 0 95368 2172 1960 S 0.0 0.1 0:00.24 sshd:<br>
> ivan@pts/0<br>
> 25286 root 20 0 51008 1908 1908 S 0.0 0.1 0:00.00 su<br>
> 8041 zabbix 20 0 95520 1680 1512 S 0.0 0.1 0:02.10<br>
> /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]<br>
> 25240 root 20 0 95368 1620 1572 S 0.0 0.1 0:00.02 sshd:<br>
> ivan [priv]<br>
> 145 message+ 20 0 42892 1164 872 S 0.0 0.1 0:01.55<br>
> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile<br>
> --systemd-activation<br>
> 6453 www-data 20 0 125348 1152 656 S 0.0 0.1 0:32.26 nginx:<br>
> worker process<br>
> 20811 postfix 20 0 67640 1136 656 S 0.0 0.1 0:00.86 qmgr -l<br>
> -t unix -u<br>
> 8038 zabbix 20 0 95520 1084 880 S 0.0 0.1 0:01.04<br>
> /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]<br>
> 8039 zabbix 20 0 95520 972 768 S 0.0 0.0 0:01.05<br>
> /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]<br>
> 142 root 20 0 27732 924 636 S 0.0 0.0 0:05.32<br>
> /usr/sbin/cron -f<br>
> 8040 zabbix 20 0 95520 872 668 S 0.0 0.0 0:01.07<br>
> /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]<br>
> 8037 zabbix 20 0 93444 728 628 S 0.0 0.0 0:14.57<br>
> /usr/sbin/zabbix_agentd: collector [idle 1 sec]<br>
> 6462 www-data 20 0 125348 500 0 S 0.0 0.0 0:32.14 nginx:<br>
> worker process<br>
><br>
><br>
><br>
> As you can see - the cumulative RSS is could barely get to the 100mb.<br>
><br>
> While this is what `free` returns:<br>
><br>
> # free -m<br>
> total used free shared buff/cache<br>
> available<br>
> Mem: 2048 1785 261 1750 0<br>
> 261<br>
> Swap: 512 14 497<br>
><br>
><br>
> So, it clearly states about 85% of ram is occupied.<br>
><br>
> `slabtop` (due to cgroup limitations?) does not work:<br>
><br>
> # slabtop<br>
> fopen /proc/slabinfo: Permission denied<br>
><br>
><br>
> But if I clear the system caches on the host<br>
><br>
> echo 3 > /proc/sys/vm/drop_caches<br>
><br>
><br>
> the container memory consumption drops to the expected <100mb.<br>
><br>
> So the question, how to monitor the memory consumption from the container<br>
> reliably? And why does `free` count caches as used memory inside container?<br>
><br>
> --<br>
> With best regards, Ivan Kurnosov<br>
<br>
</div></div><div class="HOEnZb"><div class="h5">> ______________________________<wbr>_________________<br>
> lxc-users mailing list<br>
> <a href="mailto:lxc-users@lists.linuxcontainers.org">lxc-users@lists.<wbr>linuxcontainers.org</a><br>
> <a href="http://lists.linuxcontainers.org/listinfo/lxc-users" rel="noreferrer" target="_blank">http://lists.linuxcontainers.<wbr>org/listinfo/lxc-users</a><br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Stéphane Graber<br>
Ubuntu developer<br>
<a href="http://www.ubuntu.com" rel="noreferrer" target="_blank">http://www.ubuntu.com</a><br>
</font></span><br>______________________________<wbr>_________________<br>
lxc-users mailing list<br>
<a href="mailto:lxc-users@lists.linuxcontainers.org">lxc-users@lists.<wbr>linuxcontainers.org</a><br>
<a href="http://lists.linuxcontainers.org/listinfo/lxc-users" rel="noreferrer" target="_blank">http://lists.linuxcontainers.<wbr>org/listinfo/lxc-users</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">With best regards, Ivan Kurnosov</div>
</div>