[lxc-devel] [lxc-users] Centos-7 x86_64 container - systemd-journald process eat CPU

Mon Aug 25 17:13:46 UTC 2014

Important update on this topic.

Cross posting to -devel as well.

Seems there is a different issue other than the older object id bug that
they fixed.  This problem relates to a symlink in /dev for kmsg pointing
to console.  This, apparently, creates a logging loop and causes
systemd-journald to have a hissy fit on the CPU.

https://bugs.freedesktop.org/show_bug.cgi?id=80002

They closed that as "not a bug".

Anyone running into the this problem should...

1) Set "lxc.kmsg = 0" in the container config and

2) rm -f kmsg in the container /dev/ itself (NOT on the HOST!).
	2a) That would be (of course) ${rootfs}/dev/kmsg in the host.

Step 2 is necessary even with autodev because the devtmpfs logic that is
currently present will preserve the container /dev/ over container
shutdown and start to preserve persistence (it is not persistent across
host reboots, however).

3) Shut the container down and restart.

Templates configuring containers for systemd should both set autodev = 1
and kmsg = 0.  I'll be updating the CentOS, Fedora, and OpenSUSE
templates.

Regards,
Mike

On Tue, 2014-07-29 at 13:22 -0400, Michael H. Warfield wrote:
> On Tue, 2014-07-29 at 20:37 +0400, mxs kolo wrote:
> > HI all
> 
> > When container with centos-7 x86_64 started,  process systemd-journald
> > eat cpu in next loop:
> 
> That's a journald problem that they (systemd) had claimed (in their
> bugzilla report) was fixed.  Had something to do with object id's and
> ordinals getting out of order.  I guess they lied.  It's one of the
> curses of using opaque binary file formats for storing data like this.
> Somehow the object id's get out of sequence and journald looses its mind
> and does a sit and spin on the cpu.  It's been seen in non-LXC cases
> (something to do with user journals) but seems to be worse under LXC.  I
> had been disabling journald in Fedora containers but had recently
> reenabled them.  That appears to have been a mistake.
> 
> AFA CentOS 7 in general goes - I have not yet had the time to
> incorporate the systemd logic from the Fedora template into the CentOS
> template.  I just got back from being out of pocket for the last three
> weeks (and still completely not over being 6 hours jet lagged) and I'm
> just starting to look at it.
> 
> Regards,
> Mike
> 
> > epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=8, u64=8}}}, 1, -1) = 1
> > writev(2, [{"/dev/kmsg buffer overrun, some messages lost.", 45},
> > {"\n", 1}], 2) = 46
> > read(8, "", 8192)                       = 0
> > epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=8, u64=8}}}, 1, -1) = 1
> > writev(2, [{"/dev/kmsg buffer overrun, some messages lost.", 45},
> > {"\n", 1}], 2) = 46
> > read(8, "", 8192)                       = 0
> > epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=8, u64=8}}}, 1, -1) = 1
> > writev(2, [{"/dev/kmsg buffer overrun, some messages lost.", 45},
> > {"\n", 1}], 2) = 46
> > read(8, "", 8192)                       = 0
> > epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=8, u64=8}}}, 1, -1) = 1
> > writev(2, [{"/dev/kmsg buffer overrun, some messages lost.", 45},
> > {"\n", 1}], 2) = 46
> > read(8, "", 8192)                       = 0
> > epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=8, u64=8}}}, 1, -1) = 1
> > writev(2, [{"/dev/kmsg buffer overrun, some messages lost.", 45},
> > {"\n", 1}], 2) = 46
> > 
> > 
> > [root at ce7-2 /]# ps ax
> >   PID TTY      STAT   TIME COMMAND
> >     1 ?        Ss     0:00 /sbin/init
> >    10 ?        Rs     0:06 /usr/lib/systemd/systemd-journald
> >    27 ?        Ss     0:00 /bin/dbus-daemon --system
> > --address=systemd: --nofork --nopidfile --systemd-activation
> >    28 ?        Ss     0:00 /usr/lib/systemd/systemd-logind
> >    34 ?        Ss     0:00 /usr/sbin/crond -n
> >    35 lxc/console Ss+   0:00 /sbin/agetty --noclear -s console 115200 38400 9600
> >   155 ?        S      0:00 /bin/bash
> >   224 ?        Ss     0:00 /usr/sbin/sshd -D
> >   232 ?        R+     0:00 ps ax
> > 
> > After restart, it work fine:
> > [root at ce7-2 /]# ps ax
> >   PID TTY      STAT   TIME COMMAND
> >     1 ?        Ss     0:00 /sbin/init
> >    27 ?        Ss     0:00 /bin/dbus-daemon --system
> > --address=systemd: --nofork --nopidfile --systemd-activation
> >    28 ?        Ss     0:00 /usr/lib/systemd/systemd-logind
> >    34 ?        Ss     0:00 /usr/sbin/crond -n
> >    35 lxc/console Ss+   0:00 /sbin/agetty --noclear -s console 115200 38400 9600
> >   155 ?        S      0:00 /bin/bash
> >   224 ?        Ss     0:00 /usr/sbin/sshd -D
> >   245 ?        Ss     0:00 /usr/lib/systemd/systemd-journald
> >   248 ?        R+     0:00 ps ax
> > 
> > 
> > I disable kmsg in container config (lxc.kmsg = 0) and delete symlink
> > /var/lib/lxc/ce7-2/rootfs.dev/kmsg
> > and problem go away, after start systemd-journald not cycled.
> > 
> > Probably centos template must detect $release -gt 6 and set lxc.kmsg=0
> > 
> > b.r.
> >  Maxim Kozin
> > _______________________________________________
> > lxc-users mailing list
> > lxc-users at lists.linuxcontainers.org
> > http://lists.linuxcontainers.org/listinfo/lxc-users
> 

-- 
Michael H. Warfield (AI4NB) | (770) 978-7061 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 465 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20140825/b60f1589/attachment.sig>