[Lxc-users] Help troubleshooting declining performance / high %SI CPU when running 9 Ubuntu 10.04 LXCs

[Ivan Fetch]

Hello Serge,


On Sep 15, 2011, at 12:25 PM, Serge E. Hallyn wrote:

> In addition to needing info Iliyan mentioned - what kernel are you running?
> Is it a standard backport kernel?  Do you know if a kernel update coincides
> with the performance degredation showing up?
> 
The kernel is 2.6.34-2-server #9-Ubuntu SMP  x86_64

This kernel hasn't been updated lately, so no relation there.

- Ivan



> Quoting Iliyan Stoyanov (ilf at ilf.me):
>> Hi Ivan,
>> 
>> you should probably do a monitoring with iostat and vmstat also. On the
>> top of my head I can think of at least 3 or 4 reasons why this might be
>> happening. I have similar problems with a simple laptop machine without
>> LXC containers on it (and don't have such on a server with a bunch of
>> containers on it). In my experience with bad SI everything always come
>> back to be RAM related. Also check your filesystem performance. Most of
>> the FSes nowadays keep a ton of the journalling info in RAM. I know my
>> response is not exactly an answer to your specific question but I hope
>> it might give you some pointers for better monitoring of the situation.
>> 
>> BR,
>> 
>> --ilf
>> 
>> On Thu, 2011-09-15 at 09:12 -0600, Ivan Fetch wrote:
>> 
>>> Hello,
>>> 
>>> I've inherited a Sun 4540 (thumper) machine running 9 LXC containers. During the past few weeks we've been troubleshooting a decline in performance, which ends up in high %SI (software interrupt) CPU usage. I'm hoping someone here can help troubleshoot and narrow down what the real issue is - this one really has me stumped.
>>> 
>>> THis box has 48 disks, 5 RAID6 which are in a RAID0, using md. Two NICs are bonded together, and a bridge is used for the box's IP, and the LXC network interfaces.
>>> 
>>> Linux is Ubuntu 10.04, LXC 0.6.3 , containers are also 10.04. Containers run Apache, some custom image processing, gaussian, and FTP server...
>>> 
>>> The box performs well after a reboot, with all containers back online. After ~5 days, we notice that the box is sluggish, and backup jobs (Netbackup) get less than 1Mb/sec over the network. CPU eventually reaches 61% SI. OTher processes (I am looking at ps -ax -o pcpu ..... |sort -n) begin taking much higher percent CPU than they should need, I imagine because the high %SI is taking cycles; E.G. I'll briefly see ps or sort or a shell using 6% CPU. Top shows %sy between 5-20, %wa under 5.
>>> Memory (32Gb) is mostly used for cache, and there is no swapping.
>>> 
>>> I know next-to-nothing about tracking down the cause for high %SI CPU usage.
>>> 
>>> 
>>> Thanks for any help looking at this with a clear head,
>>> 
>>> - Ivan
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> .
>>> ------------------------------------------------------------------------------
>>> Doing More with Less: The Next Generation Virtual Desktop 
>>> What are the key obstacles that have prevented many mid-market businesses
>>> from deploying virtual desktops?   How do next-generation virtual desktops
>>> provide companies an easier-to-deploy, easier-to-manage and more affordable
>>> virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
>>> _______________________________________________
>>> Lxc-users mailing list
>>> Lxc-users at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/lxc-users
> 
>> ------------------------------------------------------------------------------
>> Doing More with Less: The Next Generation Virtual Desktop 
>> What are the key obstacles that have prevented many mid-market businesses
>> from deploying virtual desktops?   How do next-generation virtual desktops
>> provide companies an easier-to-deploy, easier-to-manage and more affordable
>> virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
> 
>> _______________________________________________
>> Lxc-users mailing list
>> Lxc-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/lxc-users
> 























.