[lxc-users] Container scaling - LXD 2.0

[Ron Kelley]

Thanks Fajar,

Appreciate the pointers.  We have already setup MariaDB with the small-instance tuning as well as setup php-fpm using the on-demand option as well.  The big issue now is RAM.

A brief background:
---------------------
A few years back, one of our customers asked us to host a small website for them.  As word got out, we starting hosting a few more.  Fast forward a few years and we are now hosting > 1300 sites.  We are currently running monolithic VMs (2vCPUs 2G RAM) that host about 60-70 sites each, and we are looking to move away from these huge VMs to something more scalable and secure like LXC.  The downside to this approach is the extra RAM overhead since each container will run its own copy of nginx/php-fpm/mariadb (for ease of portability).  

After doing some research, it seems KSM is enabled in the Ubuntu 16 kernel but is disabled by default.  I will be running some tests over the next few days to see if KSM can provide any benefit.  As for the 5G RAM question; our proposed model is to run a large VM instance (5-8G RAM, 4-6vCPUs) to host the same (or more) sites via LXC containers.  We are looking to protect each site from another as well as provide more fine-tuned system resources per site (limit RAM/CPU per site).  This is our main driver behind LXC.


Thanks again for the info.

-Ron



On 5/9/2016 12:48 AM, Fajar A. Nugraha wrote:
> On Mon, May 9, 2016 at 7:18 AM, Ronald Kelley <rkelleyrtp at gmail.com> wrote:
>> Greetings all,
>>
>> I am trying to get some data points on how many containers we can run on a single host if all the containers run the same applications (eg: Wordpress w/nginx, php-fpm, mysql).  We have a number of LXD 2.0 servers running on Ubuntu 16.04 - each server has 5G RAM, 20G Swap, and 4 CPUs.
> 
> When you use lxd you can already "overprovision" (as in, the sum of
> "limits.memory" on all running containers can be MUCH greater than
> total memory you have). See
> https://insights.ubuntu.com/2015/06/11/how-many-containers-can-you-run-on-your-machine/
> for example.
> 
> I can say that swapping will -- most of the time -- kill performance.
> Big time. Often to the point that it'd be hard to even ssh into the
> server to "fix" things. Which is why most of my servers are now
> swapless. YMMV though.
> 
> Do some experiments, monitor your swap activity (e.g. use "vmstat" to
> monitor swap in and swap out), and determine whether swap actually
> helps you, or cause more harm than good.
> 
> Also, what's the story with the 5G RAM? Even my NUCs has 32GB RAM nowadays.
> 
>> I have read about Kernel Samepage Memory (KSM), and it seems to be included in the Ubuntu 16.40 kernel.  So, in theory, we can over provision our containers by using KSM.
>>
>>
>> Any pointers?
> 
> I'd actually suggest "try other methods first". For example:
> - you can easily save some memory from php-fpm by using "pm =
> ondemand" and a small number in "pm.max_children" (e.g. 2).
> - use shared mysql instace when possible. If not, use smaller memory
> settings, e.g. http://www.tocker.ca/2014/03/10/configuring-mysql-to-use-minimal-memory.html
> 
> This entry from openvz should be relevant if you still want to use KSM
> for generic applications running inside a container:
> https://openvz.org/KSM_(kernel_same-page_merging)#Enabling_memory_deduplication_in_applications
>