[lxc-users] Experience with large number of LXC/LXD containers

Tue Mar 28 15:20:48 UTC 2017

----- Mail original -----
> De: "David Favor" <david at davidfavor.com>
> À: "lxc-users" <lxc-users at lists.linuxcontainers.org>
> Envoyé: Lundi 27 Mars 2017 12:55:09
> Objet: Re: [lxc-users] Experience with large number of LXC/LXD containers

> Serge E. Hallyn wrote:
>> On Tue, Mar 14, 2017 at 02:29:01AM +0100, Benoit GEORGELIN – Association Web4all
>> wrote:
>>> ----- Mail original -----
>>>> De: “Simos Xenitellis” <simos.lists at googlemail.com> À: “lxc-users”
>>>> <lxc-users at lists.linuxcontainers.org> Envoyé: Lundi 13 Mars 2017 20:22:03
>>>> Objet: Re: [lxc-users] Experience with large number of LXC/LXD containers On
>>>> Sun, Mar 12, 2017 at 11:28 PM, Benoit GEORGELIN – Association Web4all
>>>> <benoit.georgelin at web4all.fr> wrote:
>>>>> Hi lxc-users , I would like to know if you have any experience with a large
>>>>> number of LXC/LXD containers ? In term of performance, stability and limitation
>>>>> . I'm wondering for exemple, if having 100 containers behave the same of having
>>>>> 1.000 or 10.000 with the same configuration to avoid to talk about container
>>>>> usage. I have been looking around for a couple of days to found any user/admin
>>>>> feedback experience but i'm not able to find large deployments Is there any
>>>>> ressources limits or any maximum number that can be deployed on the same node ?
>>>>> Beside physical performance of the node, is there any specific behavior that a
>>>>> large number of LXC/LXD containers can experience ? I'm not aware of any test
>>>>> or limits that can occurs beside number of process. But I'm sure from LXC/LXD
>>>>> side it might have some technical contraints ? Maybe on namespace availability
>>>>> , or any other technical layer used by LXC/LXD I will be interested to here
>>>>> from your experience or if you have any links/books/story about this large
>>>>> deployments
>>>> This would be interesting to hear if someone can talk publicly about their large
>>>> deployment. In any case, it should be possible to create, for example, 1000 web
>>>> servers and then try to access each one and check any issues regarding the
>>>> response time. Another test would be to install 1000 Wordpress installations
>>>> and check again for the response time and resource usage. Such scripts to
>>>> create this massive number of containers would also be helpful to replicate any
>>>> issues in order to solve them. Simos
> Been reading this + here's a bit of info.

> I've been running LXC since early deployment + now LXD.

> There are a few big performance killers related to WordPress. If you keep these
> issues in mind, you'll be good.

> 1) I run 100s of sites across many containers on many machines.
> My business is private, high speed hosting, so I eat from my efforts.
> No theory here.
> I target WordPress site speed at 3000+ reqs/second, measured locally
> using ab (ApacheBench). This is a crude tool + sufficient, as I issue
> 1,000,000 simultaneous 5 thread connections against a server for 30 seconds.
> ab -k -t 30 -n 10000000 -c 5 $URL
> This will crash most machines, unless they're tuned well.

> 2) Memory + CPU. The big killer of performance anywhere is swap thrash. If top
> shows swapping for more than a few seconds, likely your system is heading
> toward a crash.
> Fix: I tend to deploy OVH machines with 128G of memory, as this is enough
> memory to handle huge spikes of memory usage across many sites, during
> traffic spikes... then recover...
> For example, running 100s of sites across many LXD containers, I've had
> machines sustain 250,000+ reqs/hour every day for months.
> At these traffic levels, <1 core used sustained + 50%ish memory use.
> Sites still show 3000+ reqs/sec using ab test above.

> 3) Database: I run MariaDB rather than MySQL as it's smokin' fast.
> I also relocate /tmp to tmpfs, so temp file i/o runs at memory speed,
> rather than disk speed.
> This ensures all MariaDB temp select set files (for complex selects)
> generate + access at memory speed.
> Also PHP session /tmp files run at memory speed.
> This is important to me, as many of my clients run large membership
> sites. Many are >40K members. This sites performance would circle
> the drain if /tmp was on disk.

> 4) Disk Thrash: Becomes the killer as traffic increases.

> 5) Apache Logging: For several clients I'm currently retuning my Apache logging
> to skip logging of successful serves of - images, css, js, fonts. I'll still
> long non-200s, as these need to be debugged.
> This can make a huge difference if memory pressure/use forces disk writes to
> actually go to disk, rather than kernel filesystem i/o buffers.
> Once memory pressure forces physical disk writes, disk i/o starves Apache from
> quickly serving uncached content. Very ugly.
> Right now I'm doing extensive filesystem testing, to reduce disk thrash during
> traffic spikes + related memory pressure.

> 6) Net Connection: If you're running 1000s of containers, best also check
> adapter
> saturation. I use 10Gig adapters + even at extreme traffic levels, they barely
> reach 10% saturation.
> This means 10Gig adapters are a must for me, as 10% is 1Gig, so using 1Gig
> adapters, site speed would begin to throttle, based on adapter saturation,
> which would be a bear to debug.

> 7) Apache: I've taken setting up Apache to kill off processes, after anywhere
> from 10K to 100K requests served. This ensures the kernel can garbage collect
> (resource reclamation) which also helps escape swapping.
> If you have 100,000s+ Apache processes running, with no kill off, then
> eventually
> they can potentially eat up a massive amount of memory, which takes a long time
> to reclaim, depending on other MPM config settings.

> So… General rule of thumb. Tune your entire WAMPL stack to run out of memory:
> WAMPL - WordPress running on Apache + PHP + MariaDB + Linux

> If your sites run at memory speed, makes no real difference how many containers
> you run. Possibly context switching might come into play if many of the sites
> running were high traffic sites.

> If problems occur, just look at your Apache logs across all containers. Move the
> site with highest traffic to another physical machine.

> Or, if top shows swapping, add more memory.


Hi David, 
interesting feedback, it's good to know about the details you gave (memory/swap) 
Happy hosting ;)