[Lxc-users] Help troubleshooting declining performance / high %SI CPU when running 9 Ubuntu 10.04 LXCs

[Ivan Fetch]

Hello,

I've inherited a Sun 4540 (thumper) machine running 9 LXC containers. During the past few weeks we've been troubleshooting a decline in performance, which ends up in high %SI (software interrupt) CPU usage. I'm hoping someone here can help troubleshoot and narrow down what the real issue is - this one really has me stumped.

THis box has 48 disks, 5 RAID6 which are in a RAID0, using md. Two NICs are bonded together, and a bridge is used for the box's IP, and the LXC network interfaces.

Linux is Ubuntu 10.04, LXC 0.6.3 , containers are also 10.04. Containers run Apache, some custom image processing, gaussian, and FTP server...

The box performs well after a reboot, with all containers back online. After ~5 days, we notice that the box is sluggish, and backup jobs (Netbackup) get less than 1Mb/sec over the network. CPU eventually reaches 61% SI. OTher processes (I am looking at ps -ax -o pcpu ..... |sort -n) begin taking much higher percent CPU than they should need, I imagine because the high %SI is taking cycles; E.G. I'll briefly see ps or sort or a shell using 6% CPU. Top shows %sy between 5-20, %wa under 5.
Memory (32Gb) is mostly used for cache, and there is no swapping.

I know next-to-nothing about tracking down the cause for high %SI CPU usage.


Thanks for any help looking at this with a clear head,

- Ivan






















.