[lxc-users] How to properly find what consumes memory inside the container.

Tue Sep 26 13:24:56 UTC 2017

Hi,

This sounds like a lxcfs issue.

Can you file a bug at https://github.com/lxc/lxcfs or find one which
matches your symptoms.

We'll want at least:

 - /proc/meminfo from the container
 - /sys/fs/cgroup/memory/lxc/CONTAINER/memory.usage_in_bytes from the host
 - /sys/fs/cgroup/memory/lxc/CONTAINER/memory.stat from the host

That should let us track down where the memory usage comes from and what
may be wrong with it.

On Wed, Sep 20, 2017 at 01:18:01PM +1200, Ivan Kurnosov wrote:
> Hi,
> 
> there is a server that currently runs ~100 containers.
> 
> One of those containers is a subject of my interest.
> 
> Brief details about the container: it runs ubuntu xenial, and it's a tiny
> file server (samba based) with near to no traffic at all.
> 
> I have found that after you upload files to that server, the available
> memory size is decreased (while the "buff/cache" size stays at 0). And if
> you remove the just uploaded files - the memory consumption drops to the
> same value as it was before uploading.
> 
> Here is a output of the top (sorted by resident memory size, processes with
> more than 500kib rss):
> 
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
>    48 root      20   0   52048  18880  14428 S   0.0  0.9   0:10.11
> /lib/systemd/systemd-journald
> 18609 www-data  20   0  349208  15404   7516 S   0.0  0.7   0:13.72
> /usr/sbin/smbd -D
>  7176 www-data  20   0  345500  10720   6720 S   6.7  0.5   0:06.91
> /usr/sbin/smbd -D
> 25124 root      20   0  340104   9624   6744 S   0.0  0.5   0:00.12
> /usr/sbin/smbd -D
> 37541 root      20   0  344828   8012   4520 S   0.0  0.4   0:02.36
> /usr/sbin/smbd -D
> 15593 root      20   0  344352   6368   3444 S   0.0  0.3   0:00.39
> /usr/sbin/smbd -D
>  2450 root      20   0  336636   4072   1520 S   0.0  0.2   0:06.09
> /usr/sbin/smbd -D
> 25401 root      20   0   40560   3728   3112 R   0.3  0.2   0:00.49 top
>  2447 root      20   0  336636   3528    976 S   0.0  0.2   0:04.30
> /usr/sbin/smbd -D
> 25287 root      20   0   19972   3044   2872 S   0.0  0.1   0:00.01 bash
>  2476 root      20   0  238728   2944   1336 S   0.0  0.1   0:28.52
> /usr/sbin/nmbd -D
> 25271 ivan      20   0   21328   2784   2764 S   0.0  0.1   0:00.04 -bash
> 24250 root      20   0  858936   2616      0 S   0.0  0.1   0:01.98
> /usr/sbin/collectd
>  2448 root      20   0  426848   2504     20 S   0.3  0.1   0:01.65
> /usr/sbin/smbd -D
>     1 root      20   0   37884   2488   1676 S   0.0  0.1   0:17.13
> /sbin/init
> 25285 root      20   0   51660   2404   2400 S   0.0  0.1   0:00.00 sudo su
> 25270 ivan      20   0   95368   2172   1960 S   0.0  0.1   0:00.24 sshd:
> ivan at pts/0
> 25286 root      20   0   51008   1908   1908 S   0.0  0.1   0:00.00 su
>  8041 zabbix    20   0   95520   1680   1512 S   0.0  0.1   0:02.10
> /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]
> 25240 root      20   0   95368   1620   1572 S   0.0  0.1   0:00.02 sshd:
> ivan [priv]
>   145 message+  20   0   42892   1164    872 S   0.0  0.1   0:01.55
> /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile
> --systemd-activation
>  6453 www-data  20   0  125348   1152    656 S   0.0  0.1   0:32.26 nginx:
> worker process
> 20811 postfix   20   0   67640   1136    656 S   0.0  0.1   0:00.86 qmgr -l
> -t unix -u
>  8038 zabbix    20   0   95520   1084    880 S   0.0  0.1   0:01.04
> /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
>  8039 zabbix    20   0   95520    972    768 S   0.0  0.0   0:01.05
> /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
>   142 root      20   0   27732    924    636 S   0.0  0.0   0:05.32
> /usr/sbin/cron -f
>  8040 zabbix    20   0   95520    872    668 S   0.0  0.0   0:01.07
> /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
>  8037 zabbix    20   0   93444    728    628 S   0.0  0.0   0:14.57
> /usr/sbin/zabbix_agentd: collector [idle 1 sec]
>  6462 www-data  20   0  125348    500      0 S   0.0  0.0   0:32.14 nginx:
> worker process
> 
> 
> 
> As you can see - the cumulative RSS is could barely get to the 100mb.
> 
> While this is what `free` returns:
> 
> # free -m
>               total        used        free      shared  buff/cache
>  available
> Mem:           2048        1785         261        1750           0
>  261
> Swap:           512          14         497
> 
> 
> So, it clearly states about 85% of ram is occupied.
> 
> `slabtop` (due to cgroup limitations?) does not work:
> 
> # slabtop
> fopen /proc/slabinfo: Permission denied
> 
> 
> But if I clear the system caches on the host
> 
> echo 3 > /proc/sys/vm/drop_caches
> 
> 
> the container memory consumption drops to the expected <100mb.
> 
> So the question, how to monitor the memory consumption from the container
> reliably? And why does `free` count caches as used memory inside container?
> 
> -- 
> With best regards, Ivan Kurnosov

> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users

-- 
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20170926/f9feb0e3/attachment.sig>