[lxc-users] Experience with large number of LXC/LXD containers

[David Favor]

Serge E. Hallyn wrote:
> On Tue, Mar 14, 2017 at 02:29:01AM +0100, Benoit GEORGELIN - Association Web4all wrote:
>> ----- Mail original -----
>>> De: "Simos Xenitellis" <simos.lists at googlemail.com>
>>> À: "lxc-users" <lxc-users at lists.linuxcontainers.org>
>>> Envoyé: Lundi 13 Mars 2017 20:22:03
>>> Objet: Re: [lxc-users] Experience with large number of LXC/LXD containers
>>> On Sun, Mar 12, 2017 at 11:28 PM, Benoit GEORGELIN - Association
>>> Web4all <benoit.georgelin at web4all.fr> wrote:
>>>> Hi lxc-users ,
>>>> I would like to know if you have any experience with a large number of
>>>> LXC/LXD containers ?
>>>> In term of performance, stability and limitation .
>>>> I'm wondering for exemple, if having 100 containers behave the same of
>>>> having 1.000 or 10.000 with the same configuration to avoid to talk about
>>>> container usage.
>>>> I have been looking around for a couple of days to found any user/admin
>>>> feedback experience but i'm not able to find large deployments
>>>> Is there any ressources limits or any maximum number that can be deployed on
>>>> the same node ?
>>>> Beside physical performance of the node, is there any specific behavior that
>>>> a large number of LXC/LXD containers can experience ? I'm not aware of any
>>>> test or limits that can occurs beside number of process. But I'm sure from
>>>> LXC/LXD side it might have some technical contraints ?
>>>> Maybe on namespace availability , or any other technical layer used by
>>>> LXC/LXD
>>>> I will be interested to here from your experience or if you have any
>>>> links/books/story about this large deployments
>>
>>> This would be interesting to hear if someone can talk publicly about
>>> their large deployment.
>>> In any case, it should be possible to create, for example, 1000 web servers
>>> and then try to access each one and check any issues regarding the
>>> response time.
>>> Another test would be to install 1000 Wordpress installations and
>>> check again for the response time
>>> and resource usage.
>>> Such scripts to create this massive number of containers would also be
>>> helpful to replicate
>>> any issues in order to solve them.
>>> Simos

Been reading this + here's a bit of info.

I've been running LXC since early deployment + now LXD.

There are a few big performance killers related to WordPress. If you keep
these issues in mind, you'll be good.

1) I run 100s of sites across many containers on many machines.

    My business is private, high speed hosting, so I eat from my efforts.
    No theory here.

    I target WordPress site speed at 3000+ reqs/second, measured locally
    using ab (ApacheBench). This is a crude tool + sufficient, as I issue
    1,000,000 simultaneous 5 thread connections against a server for 30 seconds.

       ab -k -t 30 -n 10000000 -c 5 $URL

    This will crash most machines, unless they're tuned well.

2) Memory + CPU. The big killer of performance anywhere is swap thrash. If top
    shows swapping for more than a few seconds, likely your system is heading
    toward a crash.

    Fix: I tend to deploy OVH machines with 128G of memory, as this is enough
    memory to handle huge spikes of memory usage across many sites, during
    traffic spikes... then recover...

    For example, running 100s of sites across many LXD containers, I've had
    machines sustain 250,000+ reqs/hour every day for months.

    At these traffic levels, <1 core used sustained + 50%ish memory use.

    Sites still show 3000+ reqs/sec using ab test above.

3) Database: I run MariaDB rather than MySQL as it's smokin' fast.

    I also relocate /tmp to tmpfs, so temp file i/o runs at memory speed,
    rather than disk speed.

    This ensures all MariaDB temp select set files (for complex selects)
    generate + access at memory speed.

    Also PHP session /tmp files run at memory speed.

    This is important to me, as many of my clients run large membership
    sites. Many are >40K members. This sites performance would circle
    the drain if /tmp was on disk.

4) Disk Thrash: Becomes the killer as traffic increases.

5) Apache Logging: For several clients I'm currently retuning my Apache logging
    to skip logging of successful serves of - images, css, js, fonts. I'll still
    long non-200s, as these need to be debugged.

    This can make a huge difference if memory pressure/use forces disk writes to
    actually go to disk, rather than kernel filesystem i/o buffers.

    Once memory pressure forces physical disk writes, disk i/o starves Apache from
    quickly serving uncached content. Very ugly.

    Right now I'm doing extensive filesystem testing, to reduce disk thrash during
    traffic spikes + related memory pressure.

6) Net Connection: If you're running 1000s of containers, best also check adapter
    saturation. I use 10Gig adapters + even at extreme traffic levels, they barely
    reach 10% saturation.

    This means 10Gig adapters are a must for me, as 10% is 1Gig, so using 1Gig
    adapters, site speed would begin to throttle, based on adapter saturation,
    which would be a bear to debug.

7) Apache: I've taken setting up Apache to kill off processes, after anywhere
    from 10K to 100K requests served. This ensures the kernel can garbage collect
    (resource reclamation) which also helps escape swapping.

    If you have 100,000s+ Apache processes running, with no kill off, then eventually
    they can potentially eat up a massive amount of memory, which takes a long time
    to reclaim, depending on other MPM config settings.

So... General rule of thumb. Tune your entire WAMPL stack to run out of memory:

    WAMPL - WordPress running on Apache + PHP + MariaDB + Linux

If your sites run at memory speed, makes no real difference how many containers
you run. Possibly context switching might come into play if many of the sites
running were high traffic sites.

If problems occur, just look at your Apache logs across all containers. Move
the site with highest traffic to another physical machine.

Or, if top shows swapping, add more memory.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20170327/f358a58b/attachment.html>