[lxc-users] Centos-7 x86_64 container - systemd-journald process eat CPU

[Michael H. Warfield]

On Tue, 2014-07-29 at 20:37 +0400, mxs kolo wrote:
> HI all

> When container with centos-7 x86_64 started,  process systemd-journald
> eat cpu in next loop:

That's a journald problem that they (systemd) had claimed (in their
bugzilla report) was fixed.  Had something to do with object id's and
ordinals getting out of order.  I guess they lied.  It's one of the
curses of using opaque binary file formats for storing data like this.
Somehow the object id's get out of sequence and journald looses its mind
and does a sit and spin on the cpu.  It's been seen in non-LXC cases
(something to do with user journals) but seems to be worse under LXC.  I
had been disabling journald in Fedora containers but had recently
reenabled them.  That appears to have been a mistake.

AFA CentOS 7 in general goes - I have not yet had the time to
incorporate the systemd logic from the Fedora template into the CentOS
template.  I just got back from being out of pocket for the last three
weeks (and still completely not over being 6 hours jet lagged) and I'm
just starting to look at it.

Regards,
Mike

> epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=8, u64=8}}}, 1, -1) = 1
> writev(2, [{"/dev/kmsg buffer overrun, some messages lost.", 45},
> {"\n", 1}], 2) = 46
> read(8, "", 8192)                       = 0
> epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=8, u64=8}}}, 1, -1) = 1
> writev(2, [{"/dev/kmsg buffer overrun, some messages lost.", 45},
> {"\n", 1}], 2) = 46
> read(8, "", 8192)                       = 0
> epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=8, u64=8}}}, 1, -1) = 1
> writev(2, [{"/dev/kmsg buffer overrun, some messages lost.", 45},
> {"\n", 1}], 2) = 46
> read(8, "", 8192)                       = 0
> epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=8, u64=8}}}, 1, -1) = 1
> writev(2, [{"/dev/kmsg buffer overrun, some messages lost.", 45},
> {"\n", 1}], 2) = 46
> read(8, "", 8192)                       = 0
> epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=8, u64=8}}}, 1, -1) = 1
> writev(2, [{"/dev/kmsg buffer overrun, some messages lost.", 45},
> {"\n", 1}], 2) = 46
> 
> 
> [root at ce7-2 /]# ps ax
>   PID TTY      STAT   TIME COMMAND
>     1 ?        Ss     0:00 /sbin/init
>    10 ?        Rs     0:06 /usr/lib/systemd/systemd-journald
>    27 ?        Ss     0:00 /bin/dbus-daemon --system
> --address=systemd: --nofork --nopidfile --systemd-activation
>    28 ?        Ss     0:00 /usr/lib/systemd/systemd-logind
>    34 ?        Ss     0:00 /usr/sbin/crond -n
>    35 lxc/console Ss+   0:00 /sbin/agetty --noclear -s console 115200 38400 9600
>   155 ?        S      0:00 /bin/bash
>   224 ?        Ss     0:00 /usr/sbin/sshd -D
>   232 ?        R+     0:00 ps ax
> 
> After restart, it work fine:
> [root at ce7-2 /]# ps ax
>   PID TTY      STAT   TIME COMMAND
>     1 ?        Ss     0:00 /sbin/init
>    27 ?        Ss     0:00 /bin/dbus-daemon --system
> --address=systemd: --nofork --nopidfile --systemd-activation
>    28 ?        Ss     0:00 /usr/lib/systemd/systemd-logind
>    34 ?        Ss     0:00 /usr/sbin/crond -n
>    35 lxc/console Ss+   0:00 /sbin/agetty --noclear -s console 115200 38400 9600
>   155 ?        S      0:00 /bin/bash
>   224 ?        Ss     0:00 /usr/sbin/sshd -D
>   245 ?        Ss     0:00 /usr/lib/systemd/systemd-journald
>   248 ?        R+     0:00 ps ax
> 
> 
> I disable kmsg in container config (lxc.kmsg = 0) and delete symlink
> /var/lib/lxc/ce7-2/rootfs.dev/kmsg
> and problem go away, after start systemd-journald not cycled.
> 
> Probably centos template must detect $release -gt 6 and set lxc.kmsg=0
> 
> b.r.
>  Maxim Kozin
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users

-- 
Michael H. Warfield (AI4NB) | (770) 978-7061 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 465 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20140729/a8f054cf/attachment.sig>