<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Follow-up. Seems to be a bug with the kernel (4.4.0-59). Heads-up to everyone…<div class=""><br class=""></div><div class=""><a href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842" class="">https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842</a></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div><div class="">On Jan 17, 2017, at 7:15 AM, Ron Kelley <<a href="mailto:rkelleyrtp@gmail.com" class="">rkelleyrtp@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Greetings all,<br class=""><br class="">Running Ubuntu 16.04 with 5G RAM, 20G SWAP, and LXD (LXC v.2.0.6 and LXD 2.0.8). We recently did a system update on our LXD servers and started getting a whole bunch of OOM messages from the containers. Something like this:<br class=""><br class=""><br class="">----------------------------------------------------------------------<br class="">Jan 17 06:20:54 LXD_Server_01 kernel: [259185.075154] mysqld invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0 Jan 17 06:20:54 LXD_Server_01 kernel: [259185.075158] mysqld cpuset=DB-Server3 mems_allowed=0 Jan 17 06:20:54 LXD_Server_01 kernel: [259185.075166] CPU: 0 PID: 27649 Comm: mysqld Not tainted 4.4.0-59-generic #80-Ubuntu Jan 17 06:20:54 LXD_Server_01 kernel: [259185.075167] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014<br class="">----------------------------------------------------------------------<br class=""><br class=""><br class=""><br class="">The container (www-somesitename-com) is using a custom profile like this:<br class="">----------------------------------------------------------------------<br class="">name: Dual_Network_MySQL<br class="">config:<br class=""> limits.cpu: "2"<br class=""> limits.memory: 512MB<br class=""> limits.memory.swap: "true"<br class=""> raw.lxc: lxc.cgroup.memory.memsw.limit_in_bytes = 1300M<br class="">description: ""<br class="">devices:<br class=""> eth0:<br class=""> name: eth0<br class=""> nictype: macvlan<br class=""> parent: eth1.2005<br class=""> type: nic<br class=""> eth1:<br class=""> name: eth1<br class=""> nictype: macvlan<br class=""> parent: eth1.2006<br class=""> type: nic<br class="">----------------------------------------------------------------------<br class=""><br class=""><br class=""><br class="">The above profile should give the container 1.8GB of RAM (512RAM + 1.3G SWAP). If I look at the container stats, I don’t see where RAM+SWAP were exceeded:<br class="">----------------------------------------------------------------------<br class="">Name: DB-Server3<br class="">Remote: unix:/var/lib/lxd/unix.socket<br class="">Architecture: x86_64<br class="">Created: 2016/10/17 06:47 UTC<br class="">Status: Running<br class="">Type: persistent<br class="">Profiles: Dual_Network_MySQL<br class="">Pid: 2215<br class="">Ips:<br class=""> eth0:<span class="Apple-tab-span" style="white-space:pre"> </span>inet<span class="Apple-tab-span" style="white-space:pre"> </span>1.2.3.4<br class=""> eth0:<span class="Apple-tab-span" style="white-space:pre"> </span>inet6<span class="Apple-tab-span" style="white-space:pre"> </span>XXXXX<br class=""> eth1:<span class="Apple-tab-span" style="white-space:pre"> </span>inet<span class="Apple-tab-span" style="white-space:pre"> </span>1.2.3.4<br class=""> eth1:<span class="Apple-tab-span" style="white-space:pre"> </span>inet6<span class="Apple-tab-span" style="white-space:pre"> </span>YYYY<br class=""> lo:<span class="Apple-tab-span" style="white-space:pre"> </span>inet<span class="Apple-tab-span" style="white-space:pre"> </span>127.0.0.1<br class=""> lo:<span class="Apple-tab-span" style="white-space:pre"> </span>inet6<span class="Apple-tab-span" style="white-space:pre"> </span>::1<br class="">Resources:<br class=""> Processes: 19<br class=""> Memory usage:<br class=""> Memory (current): 112.85MB<br class=""> Memory (peak): 271.26MB<br class=""> Swap (current): 12.23MB<br class=""> Swap (peak): 5.39MB<br class=""> Network usage:<br class=""> eth0:<br class=""> Bytes received: 4.17GB<br class=""> Bytes sent: 69.48GB<br class=""> Packets received: 25587831<br class=""> Packets sent: 31668639<br class=""> eth1:<br class=""> Bytes received: 1.53GB<br class=""> Bytes sent: 36.13GB<br class=""> Packets received: 9743914<br class=""> Packets sent: 14022159<br class=""> lo:<br class=""> Bytes received: 0 bytes<br class=""> Bytes sent: 0 bytes<br class=""> Packets received: 0<br class=""> Packets sent: 0<br class="">----------------------------------------------------------------------<br class=""><br class="">This happens on a variety of LXD servers (we have 5 running right now) and a variety of containers. Running “free -m” on the container server shows plenty of RAM and SWAP available. The only thing common is the OS running in the container (Ubuntu 16.04). It seems our CentOS7 containers don’t have this issue.<br class=""><br class="">Any clues/pointers?<br class=""><br class="">Thanks.<br class=""><br class=""><br class=""><br class=""><br class=""></div></div></div><br class=""></div></body></html>