[lxc-users] LXD 2.14 - Ubuntu 16.04 - kernel 4.4.0-57-generic - SWAP continuing to grow

[Fajar A. Nugraha]

On Sat, Jul 15, 2017 at 10:48 PM, Ron Kelley <rkelleyrtp at gmail.com> wrote:

> Thanks for the great replies.
>
> Marat/Fajar:  How many servers do you guys have running in production, and
> what are their characteristics (RAM, CPU, workloads, etc)?


My biggest production one was AWS r4.16xlarge (almost 500GB memory), though
nowadays I mostly use r3.8xlarge (around half the memory, but more
cost-efficient). It's running many things from small web servers to large
hadoop instances.

My smallest production system had 16GB RAM. After some struggles (including
testing if swap would help, which did not), in the end I increase its RAM
to 128GB. MUCH better.


> I am trying to see if our systems generally align to what you are
> running.  Running without swap seems rather drastic and removes the “safety
> net” in the case of a bad program.


Which is why I mentioned setting the limits beforehand.

For a memory-limited system, you should be able to follow
http://digitaloceanvps.blogspot.co.id/2014/04/best-configuration-for-512mb-1gb-ram.html
or similar. Though since you said you use nginx, instead of setting appache
you should be able to just set php-fpm to use on-demand process manager
with a small (2-4) max process.


> In the end, we must have all containers/processes running 24/7.
>

Which is EXACTLY why I disable swap. I do NOT want a misconfigured
container dragging others down. And when you configure the applications
correctly, each container should stay within its limited memory. Things
like 'sudden spike in user access' would slow it down (e.g. due to waiting
for php-fpm process becomes available) but it would not create a spike in
memory usage.


> tldr;
> ----
> After digging into this a bit, it seems “top”, “top”, and “free” report
> similar swap usage, however, other tools report much less swap usage.  I
> found the following threads on the ‘net which include simple scripts to
> look in /proc and examine swap usage per process:
>
> https://stackoverflow.com/questions/479953/how-to-find-
> out-which-processes-are-swapping-in-linux
> https://www.cyberciti.biz/faq/linux-which-process-is-using-swap
>
> As some people pointed out, top/htop don’t accurately report the swap
> usage as they combine a number of memory fields together.  And, indeed,
> running the script in each container (looking at /proc) show markedly
> different results when all the numbers are added up.  For example, the
> “free” command on one of our servers reports 3G of swap in use, but the
> script that scans the /proc directory only shows 1.3G of real swap in use.
> Very odd.
>
> All that said, the real issue is to find out if one of our
> containers/processes has a memory leak (per Marat’s suggestion below).
> Unfortunately, LXD does not provide an easy way to track per-container
> stats, thus we must “roll our own” tools.
>
>
/proc/meminfo (and some other files) in the container is fake. It's created
by lxcfs, using numbers from cgroup.
cgroup would generally provide accurate info (e.g. 'how much memory is used
by processes under this cgroup'). If you're rolling your own tools, read
cgroup files directly (e.g. /sys/fs/cgroup/memory/lxc/ ...).

In any case, if your tools show memory usage in a container higher than its
configured limit, then its perfectly normal that it starts to swap. Even
when the host itself still have lots of memory.

-- 
Fajar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20170716/b0567cbf/attachment-0001.html>