[lxc-users] LXD 2.14 - Ubuntu 16.04 - kernel 4.4.0-57-generic - SWAP continuing to grow

Sat Jul 15 15:48:21 UTC 2017

Thanks for the great replies.  

Marat/Fajar:  How many servers do you guys have running in production, and what are their characteristics (RAM, CPU, workloads, etc)?  I am trying to see if our systems generally align to what you are running.  Running without swap seems rather drastic and removes the “safety net” in the case of a bad program.  In the end, we must have all containers/processes running 24/7.

tldr;
----
After digging into this a bit, it seems “top”, “top”, and “free” report similar swap usage, however, other tools report much less swap usage.  I found the following threads on the ‘net which include simple scripts to look in /proc and examine swap usage per process:

https://stackoverflow.com/questions/479953/how-to-find-out-which-processes-are-swapping-in-linux
https://www.cyberciti.biz/faq/linux-which-process-is-using-swap 

As some people pointed out, top/htop don’t accurately report the swap usage as they combine a number of memory fields together.  And, indeed, running the script in each container (looking at /proc) show markedly different results when all the numbers are added up.  For example, the “free” command on one of our servers reports 3G of swap in use, but the script that scans the /proc directory only shows 1.3G of real swap in use.  Very odd.

All that said, the real issue is to find out if one of our containers/processes has a memory leak (per Marat’s suggestion below).  Unfortunately, LXD does not provide an easy way to track per-container stats, thus we must “roll our own” tools.

-Ron

> On Jul 15, 2017, at 5:11 AM, Marat Khalili <mkh at rqc.ru> wrote:
> 
> I'm using LXC, and I frequently observe some unused containers get swapped out, even though system has plenty of RAM and no RAM limits are set. The only bad effect I observe is couple of seconds delay when you first log into them after some time. I guess it is absolutely normal since kernel tries to maximize amount of memory available for disk caches.
> 
> If you don't like this behavior, instead of trying to fine tune kernel parameters why not disable swap altogether? Many people run it this way, it's mostly a matter of taste these days. (But first check your software for leaks.)
> 
> > For example, our “server-4” machine shows 8G total RAM, 500MB free, 2.5G available, and 5G of buff/cache. Yet, swap is at 5.5GB and has been slowly growing over the past few days. It seems something is preventing the apps from using the RAM.
> 
> Did you identify what processes all this virtual memory belongs to?
> 
> > To be honest, we have been battling lots of memory/swap issues using LXD. We started with no tuning, but the app stack quickly ran out of memory. 
> 
> LXC/LXD is hardly responsible for your app stack memory usage. Either you underestimated it or there's a memory leak somewhere.
> 
> > Given all the issues we have had with memory and swap using LXD, we are seriously considering moving back to the traditional VM approach until LXC/LXD is better “baked”.
> 
> Did your VMs use less memory? I don't think so. Limits could be better enforced, but VMs don't magically give you infinite RAM. 
> -- 
> 
> With Best Regards,
> Marat Khalili
> 
> On July 14, 2017 9:58:57 PM GMT+03:00, Ron Kelley <rkelleyrtp at gmail.com> wrote:
> Wondering if anyone else has similar issues.
> 
> We have 5x LXD 2.12 servers running (U16.04 - kernel 4.4.0-57-generic - 8G RAM, 19G SWAP).  Each server is running about 50 LXD containers - Wordpress w/Nginx and PHP7.  The servers have been running for about 15 days now, and swap space continues to grow.  In addition, the kswapd0 process starts consuming CPU until we flush the system cache via "/bin/echo 3 > /proc/sys/vm/drop_caches” command.
> 
> Our LXD profile looks like this:
> -------------------------
> config:
>   limits.cpu: "2"
>   limits.memory: 512MB
>   limits.memory.swap: "true"
>   limits.memory.swap.priority: "1"
> -------------------------
> 
> 
> We also have added these to /etc/sysctl.conf
> -------------------------
> vm.swappiness=10
> vm.vfs_cache_pressure=50
> -------------------------
> 
> A quick “top” output shows plenty of available Memory and buff/cache.  But, for some reason, the system continues to swap out the app.  For example, our “server-4” machine shows 8G total RAM, 500MB free, 2.5G available, and 5G of buff/cache.  Yet, swap is at 5.5GB and has been slowly growing over the past few days.  It seems something is preventing the apps from using the RAM.
> 
> 
> To be honest, we have been battling lots of memory/swap issues using LXD.  We started with no tuning, but the app stack quickly ran out of memory.  After editing the profile to allow 512MB RAM per container (and restarting the container), the kswapd0 issue happens.  Given all the issues we have had with memory and swap using LXD, we are seriously considering moving back to the traditional VM approach until LXC/LXD is better “baked”.
> 
> 
> -Ron
> 
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users