[lxc-devel] systemd-226 and lxcfs
Christian Brauner
christianvanbrauner at gmail.com
Mon Oct 12 12:51:21 UTC 2015
Thanks for taking the time to investigate this. I noticed this as well
(https://github.com/lxc/lxc/issues/663). The new init.scope slice has been
introduced in systemd 226 in order to deal with unified cgroups which have been
introduced into the Kernel in version 4.2.1 or 4.2.2. The details why can be
found in the corresponding systemd commit:
https://github.com/systemd/systemd/pull/1116
I think it is unlikely upstream systemd can do anything about it. I think this
will probably need to be addressed in lxcfs. (Btw, Docker containers are seeing
the same problem with the new init.scope.)
On Mon, Oct 12, 2015 at 11:20:51AM +0200, Wolfgang Bumiller wrote:
> With systemd version 226 systemd apparently runs in a different cgroup
> (init.scope) which causes problems:
> https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1497420
> https://bugzilla.proxmox.com/show_bug.cgi?id=755
>
> Apparently it keeps trying to restart systemd-journald, which is
> accompanied by very informative log messages:
>
> Oct 12 08:49:36 myhost lxcfs[16970]: get_next_cgroup_dir: I was fed bad input
> (repeated a bunch of times)
>
> I put the foreground lxc-start output at the bottom.
> Since this is such a very informative error message I changed it a bit
> to print the two strings it's comparing:
> taskcg = lxc/103/init.scope
> querycg = lxc/103/system.slice/systemd-journald.service
>
> Now, this 'init.scope' seems to be new in systemd. I'm not quite sure
> if this is all there's to it, but since that was the first thing I saw
> it's what I went with, and I'm having some questions regarding my
> understanding of the lxcfs code regarding caller_is_in_ancestor() and
> get_next_cgroup_dir().
>
> For one, it seems overly restrictive - more than a usual system would
> be if I'm not mistaken?
> And on the other hand the strlen() comparison seems a bit off as does
> the creation of the 'next' string.
> I took the liberty of running the code in a standalone .c file with a
> few test-input lines.
> In particular, caller_is_in_ancestor contains the following comment:
> * if the answer is false and nextcg is not NULL, then *nextcg will point
> * to a nih_alloc'd string containing the next cgroup directory under cg
> ~ ~~~~~~~~
>
> But here's what I get:
>
> $ ./a.out
> cg = /a/b
> (1) Test: '1:foo:/a': true
> (2) Test: '1:foo:/a/b': true
> (3) Test: '1:foo:/a/b/c': false, got next: 'c'
> get_next_cgroup_dir: I was fed bad input : '/x' and '/a/b'
> (4) Test: '1:foo:/x': false
> get_next_cgroup_dir: I was fed bad input : '/x/b' and '/a/b'
> (5) Test: '1:foo:/x/b': false
> (6) Test: '1:foo:/x/b/z': false, got next: 'z'
> (7) Test: '1:foo:/foobar': false, got next: 'ar'
>
> The 7th test shows a rather obvious issue there.
> The returned 'next' string when the return value is false is taken
> from the wrong string. 1..5 seem fine, 6 returns 'z', which is not
> the directory under the 'cg' variable mentioned in the comment, but
> from the caller's path. 7 returns 'ar', which is
> &"/foobar"[strlen("/a/b")] which is basically how the string is
> created.
>
>
> So now I'm wondering if the current issue is an lxc-only issue, a
> systemd-issue, or both, as I'm pretty sure that I don't give a damn
> about whether systemd-journald manages to run when my system is
> broken. In a broken system I'd rather get a freaking bash emergency
> console than a system that refuses to do anything just because it has
> no freaking *logger*... grrr...
> (Well, I can attach to the namespaces anyway, so in containers that
> part is a non-issue, but really... come on...)
>
>
> --- lxc-start foreground output ---
>
> # lxc-start -lDEBUG -o ~/lxc.log -F -n 103
> systemd 226 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN)
> Detected virtualization lxc.
> Detected architecture x86-64.
>
> Welcome to Arch Linux!
>
> Set hostname to <carc>.
> Failed to install release agent, ignoring: No such file or directory
> display-manager.service: Cannot add dependency job, ignoring: Unit display-manager.service failed to load: No such file or directory.
> [ OK ] Started Forward Password Requests to Wall Directory Watch.
> Failed to reset devices.list on /lxc/103: Permission denied
> Failed to reset devices.list on /lxc/103/user.slice: Permission denied
> [ OK ] Created slice User and Session Slice.
> [ OK ] Reached target Encrypted Volumes.
> [ OK ] Listening on Journal Socket.
> [ OK ] Reached target Remote File Systems.
> [ OK ] Listening on Journal Audit Socket.
> [ OK ] Listening on /dev/initctl Compatibility Named Pipe.
> Failed to reset devices.list on /lxc/103/system.slice: Permission denied
> [ OK ] Created slice System Slice.
> Failed to reset devices.list on /lxc/103/system.slice/dev-mqueue.mount: Permission denied
> Mounting POSIX Message Queue File System...
> Failed to reset devices.list on /lxc/103/system.slice/system-container\x2dgetty.slice: Permission denied
> [ OK ] Created slice system-container\x2dgetty.slice.
> Failed to reset devices.list on /lxc/103/system.slice/dev-hugepages.mount: Permission denied
> Mounting Huge Pages File System...
> Failed to reset devices.list on /lxc/103/system.slice/systemd-remount-fs.service: Permission denied
> Starting Remount Root and Kernel File Systems...
> Failed to reset devices.list on /lxc/103/system.slice/tmp.mount: Permission denied
> Mounting Temporary Directory...
> Failed to reset devices.list on /lxc/103/system.slice/system-getty.slice: Permission denied
> [ OK ] Created slice system-getty.slice.
> [ OK ] Reached target Slices.
> [ OK ] Started Dispatch Password Requests to Console Directory Watch.
> [ OK ] Reached target Paths.
> [ OK ] Listening on Device-mapper event daemon FIFOs.
> [ OK ] Listening on Journal Socket (/dev/log).
> Failed to reset devices.list on /lxc/103/system.slice/systemd-journald.service: Permission denied
> Starting Journal Service...
> [ OK ] Reached target Swap.
> Failed to reset devices.list on /lxc/103/system.slice/proc-uptime.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/proc-meminfo.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/proc-stat.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/dev-lxc-tty1.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/proc-diskstats.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/dev-lxc-console.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/proc-sysrq\x2dtrigger.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/sys-devices-virtual-net.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/sys-fs-fuse-connections.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/proc-sys-net.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/something.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/dev-lxc-tty2.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/proc-cpuinfo.mount: Permission denied
> Failed to reset devices.list on /lxc/103/system.slice/-.mount: Permission denied
> Failed to reset devices.list on /lxc/103/init.scope: Permission denied
> dev-mqueue.mount: Mount process exited, code=exited status=219
> [FAILED] Failed to mount POSIX Message Queue File System.
> See 'systemctl status dev-mqueue.mount' for details.
> dev-mqueue.mount: Unit entered failed state.
> dev-hugepages.mount: Mount process exited, code=exited status=219
> [FAILED] Failed to mount Huge Pages File System.
> See 'systemctl status dev-hugepages.mount' for details.
> dev-hugepages.mount: Unit entered failed state.
> systemd-remount-fs.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Remount Root and Kernel File Systems.
> See 'systemctl status systemd-remount-fs.service' for details.
> systemd-remount-fs.service: Unit entered failed state.
> systemd-remount-fs.service: Failed with result 'exit-code'.
> tmp.mount: Mount process exited, code=exited status=219
> [FAILED] Failed to mount Temporary Directory.
> See 'systemctl status tmp.mount' for details.
> [DEPEND] Dependency failed for Basic System.
> [DEPEND] Dependency failed for D-Bus System Message Bus.
> dbus.service: Job dbus.service/start failed with result 'dependency'.
> [DEPEND] Dependency failed for Getty on tty1.
> getty at tty1.service: Job getty at tty1.service/start failed with result 'dependency'.
> [DEPEND] Dependency failed for Login Service.
> systemd-logind.service: Job systemd-logind.service/start failed with result 'dependency'.
> [DEPEND] Dependency failed for Console Getty.
> console-getty.service: Job console-getty.service/start failed with result 'dependency'.
> [DEPEND] Dependency failed for Multi-User System.
> [DEPEND] Dependency failed for Graphical Interface.
> graphical.target: Job graphical.target/start failed with result 'dependency'.
> multi-user.target: Job multi-user.target/start failed with result 'dependency'.
> [DEPEND] Dependency failed for Permit User Sessions.
> systemd-user-sessions.service: Job systemd-user-sessions.service/start failed with result 'dependency'.
> [DEPEND] Dependency failed for Container Getty on /dev/pts/0.
> container-getty at 0.service: Job container-getty at 0.service/start failed with result 'dependency'.
> [DEPEND] Dependency failed for Container Getty on /dev/pts/1.
> container-getty at 1.service: Job container-getty at 1.service/start failed with result 'dependency'.
> basic.target: Job basic.target/start failed with result 'dependency'.
> tmp.mount: Unit entered failed state.
> systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Journal Service.
> See 'systemctl status systemd-journald.service' for details.
> [DEPEND] Dependency failed for Flush Journal to Persistent Storage.
> systemd-journal-flush.service: Job systemd-journal-flush.service/start failed with result 'dependency'.
> systemd-journald.service: Unit entered failed state.
> systemd-journald.service: Failed with result 'exit-code'.
> Starting Journal Service...
> [ OK ] Reached target Login Prompts.
> Starting Create System Users...
> Starting Load/Save Random Seed...
> Starting Rebuild Hardware Database...
> Starting Rebuild Dynamic Linker Cache...
> systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Journal Service.
> See 'systemctl status systemd-journald.service' for details.
> systemd-journald.service: Unit entered failed state.
> systemd-journald.service: Failed with result 'exit-code'.
> systemd-sysusers.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Create System Users.
> See 'systemctl status systemd-sysusers.service' for details.
> systemd-sysusers.service: Unit entered failed state.
> systemd-sysusers.service: Failed with result 'exit-code'.
> systemd-random-seed.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Load/Save Random Seed.
> See 'systemctl status systemd-random-seed.service' for details.
> systemd-random-seed.service: Unit entered failed state.
> systemd-random-seed.service: Failed with result 'exit-code'.
> systemd-hwdb-update.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Rebuild Hardware Database.
> See 'systemctl status systemd-hwdb-update.service' for details.
> systemd-hwdb-update.service: Unit entered failed state.
> systemd-hwdb-update.service: Failed with result 'exit-code'.
> ldconfig.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Rebuild Dynamic Linker Cache.
> See 'systemctl status ldconfig.service' for details.
> ldconfig.service: Unit entered failed state.
> ldconfig.service: Failed with result 'exit-code'.
> Starting Journal Service...
> [ OK ] Reached target Local File Systems (Pre).
> [ OK ] Reached target Local File Systems.
> Starting Rebuild Journal Catalog...
> Starting Create Volatile Files and Directories...
> systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Journal Service.
> See 'systemctl status systemd-journald.service' for details.
> systemd-journald.service: Unit entered failed state.
> systemd-journald.service: Failed with result 'exit-code'.
> Starting Journal Service...
> [ ***] (1 of 3) A start job is running for Journal Service (9s / 1min 30s)systemd-journal-catalog-update.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Rebuild Journal Catalog.
> See 'systemctl status systemd-journal-catalog-update.service' for details.
> systemd-journal-catalog-update.service: Unit entered failed state.
> systemd-journal-catalog-update.service: Failed with result 'exit-code'.
> systemd-tmpfiles-setup.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Create Volatile Files and Directories.
> See 'systemctl status systemd-tmpfiles-setup.service' for details.
> systemd-tmpfiles-setup.service: Unit entered failed state.
> systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
> Starting Update UTMP about System Boot/Shutdown...
> Starting Update is Completed...
> systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Journal Service.
> See 'systemctl status systemd-journald.service' for details.
> systemd-journald.service: Unit entered failed state.
> systemd-journald.service: Failed with result 'exit-code'.
> Starting Journal Service...
> [ ***] (1 of 3) A start job is running for Update UTMP about System Boot/Shutdown (19s / no limit)systemd-update-utmp.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Update UTMP about System Boot/Shutdown.
> See 'systemctl status systemd-update-utmp.service' for details.
> systemd-update-utmp.service: Unit entered failed state.
> systemd-update-utmp.service: Failed with result 'exit-code'.
> systemd-update-done.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Update is Completed.
> See 'systemctl status systemd-update-done.service' for details.
> systemd-update-done.service: Unit entered failed state.
> systemd-update-done.service: Failed with result 'exit-code'.
> systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Journal Service.
> See 'systemctl status systemd-journald.service' for details.
> systemd-journald.service: Unit entered failed state.
> systemd-journald.service: Failed with result 'exit-code'.
> Starting Journal Service...
> [** ] A start job is running for Journal Service (9s / 1min 30s)systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Journal Service.
> See 'systemctl status systemd-journald.service' for details.
> systemd-journald.service: Unit entered failed state.
> systemd-journald.service: Failed with result 'exit-code'.
> Starting Journal Service...
> [ *** ] A start job is running for Journal Service (9s / 1min 30s)systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
> [FAILED] Failed to start Journal Service.
> See 'systemctl status systemd-journald.service' for details.
> systemd-journald.service: Unit entered failed state.
> systemd-journald.service: Failed with result 'exit-code'.
> Starting Journal Service...
> [ *** ] A start job is running for Journal Service (7s / 1min 30s)Received SIGRTMIN+14.
>
> _______________________________________________
> lxc-devel mailing list
> lxc-devel at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20151012/7e6d6c44/attachment.sig>
More information about the lxc-devel
mailing list