[lxc-users] lxd process using lots of CPU

[Tomasz Chmielewski]

On a server with several ~idlish containers:


   PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
19104 root       20   0 2548M 44132 15236 S 140.  0.0 58h03:17 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
24966 root       20   0 2548M 44132 15236 S 18.2  0.0  2h45:36 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
19162 root       20   0 2548M 44132 15236 S 17.5  0.0  3h31:49 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
19120 root       20   0 2548M 44132 15236 S 16.2  0.0  3h16:11 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
19244 root       20   0 2548M 44132 15236 S 11.0  0.0  1h48:56 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
19123 root       20   0 2548M 44132 15236 S 11.0  0.0  3h34:42 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
19243 root       20   0 2548M 44132 15236 S 10.4  0.0  1h06:13 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
14962 root       20   0 2548M 44132 15236 R 10.4  0.0  3h17:27 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
19356 root       20   0 2548M 44132 15236 S  9.7  0.0  2h16:44 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
19161 root       20   0 2548M 44132 15236 R  9.7  0.0  1h26:40 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
19126 root       20   0 2548M 44132 15236 R  9.1  0.0 22:11.20 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
19115 root       20   0 2548M 44132 15236 R  8.4  0.0  2h55:21 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
   693 root       20   0 2548M 44132 15236 R  8.4  0.0  2h28:02 
/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log


That's actually one lxd process with many threads; view from htop.

Expected?

ii  liblxc1                            2.0.7-0ubuntu1~16.04.1            
   amd64        Linux Containers userspace tools (library)
ii  lxc-common                         2.0.7-0ubuntu1~16.04.1            
   amd64        Linux Containers userspace tools (common tools)
ii  lxcfs                              2.0.6-0ubuntu1~16.04.1            
   amd64        FUSE based filesystem for LXC
ii  lxd                                2.0.9-0ubuntu1~16.04.2            
   amd64        Container hypervisor based on LXC - daemon
ii  lxd-client                         2.0.9-0ubuntu1~16.04.2            
   amd64        Container hypervisor based on LXC - client



strace of the process mainly shows:

[pid 19124] poll([{fd=28, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19120] poll([{fd=13, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19124] <... poll resumed> )        = 1 ([{fd=28, 
revents=POLLNVAL}])
[pid 19120] <... poll resumed> )        = 1 ([{fd=13, 
revents=POLLNVAL}])
[pid 19124] poll([{fd=28, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19120] poll([{fd=13, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19124] <... poll resumed> )        = 1 ([{fd=28, 
revents=POLLNVAL}])
[pid 19120] <... poll resumed> )        = 1 ([{fd=13, 
revents=POLLNVAL}])
[pid 19124] poll([{fd=28, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19120] poll([{fd=13, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19124] <... poll resumed> )        = 1 ([{fd=28, 
revents=POLLNVAL}])
[pid 19120] <... poll resumed> )        = 1 ([{fd=13, 
revents=POLLNVAL}])
[pid 19124] poll([{fd=28, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19120] poll([{fd=13, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19124] <... poll resumed> )        = 1 ([{fd=28, 
revents=POLLNVAL}])
[pid 19120] <... poll resumed> )        = 1 ([{fd=13, 
revents=POLLNVAL}])
[pid 19124] poll([{fd=28, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19120] poll([{fd=13, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19124] <... poll resumed> )        = 1 ([{fd=28, 
revents=POLLNVAL}])
[pid 19120] <... poll resumed> )        = 1 ([{fd=13, 
revents=POLLNVAL}])
[pid 19124] poll([{fd=28, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>
[pid 19120] poll([{fd=13, 
events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1, -1 <unfinished ...>


Tomasz Chmielewski
https://lxadm.com