[lxc-devel] systemd-226 and lxcfs
Wolfgang Bumiller
w.bumiller at proxmox.com
Mon Oct 12 09:20:51 UTC 2015
With systemd version 226 systemd apparently runs in a different cgroup
(init.scope) which causes problems:
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1497420
https://bugzilla.proxmox.com/show_bug.cgi?id=755
Apparently it keeps trying to restart systemd-journald, which is
accompanied by very informative log messages:
Oct 12 08:49:36 myhost lxcfs[16970]: get_next_cgroup_dir: I was fed bad input
(repeated a bunch of times)
I put the foreground lxc-start output at the bottom.
Since this is such a very informative error message I changed it a bit
to print the two strings it's comparing:
taskcg = lxc/103/init.scope
querycg = lxc/103/system.slice/systemd-journald.service
Now, this 'init.scope' seems to be new in systemd. I'm not quite sure
if this is all there's to it, but since that was the first thing I saw
it's what I went with, and I'm having some questions regarding my
understanding of the lxcfs code regarding caller_is_in_ancestor() and
get_next_cgroup_dir().
For one, it seems overly restrictive - more than a usual system would
be if I'm not mistaken?
And on the other hand the strlen() comparison seems a bit off as does
the creation of the 'next' string.
I took the liberty of running the code in a standalone .c file with a
few test-input lines.
In particular, caller_is_in_ancestor contains the following comment:
* if the answer is false and nextcg is not NULL, then *nextcg will point
* to a nih_alloc'd string containing the next cgroup directory under cg
~ ~~~~~~~~
But here's what I get:
$ ./a.out
cg = /a/b
(1) Test: '1:foo:/a': true
(2) Test: '1:foo:/a/b': true
(3) Test: '1:foo:/a/b/c': false, got next: 'c'
get_next_cgroup_dir: I was fed bad input : '/x' and '/a/b'
(4) Test: '1:foo:/x': false
get_next_cgroup_dir: I was fed bad input : '/x/b' and '/a/b'
(5) Test: '1:foo:/x/b': false
(6) Test: '1:foo:/x/b/z': false, got next: 'z'
(7) Test: '1:foo:/foobar': false, got next: 'ar'
The 7th test shows a rather obvious issue there.
The returned 'next' string when the return value is false is taken
from the wrong string. 1..5 seem fine, 6 returns 'z', which is not
the directory under the 'cg' variable mentioned in the comment, but
from the caller's path. 7 returns 'ar', which is
&"/foobar"[strlen("/a/b")] which is basically how the string is
created.
So now I'm wondering if the current issue is an lxc-only issue, a
systemd-issue, or both, as I'm pretty sure that I don't give a damn
about whether systemd-journald manages to run when my system is
broken. In a broken system I'd rather get a freaking bash emergency
console than a system that refuses to do anything just because it has
no freaking *logger*... grrr...
(Well, I can attach to the namespaces anyway, so in containers that
part is a non-issue, but really... come on...)
--- lxc-start foreground output ---
# lxc-start -lDEBUG -o ~/lxc.log -F -n 103
systemd 226 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN)
Detected virtualization lxc.
Detected architecture x86-64.
Welcome to Arch Linux!
Set hostname to <carc>.
Failed to install release agent, ignoring: No such file or directory
display-manager.service: Cannot add dependency job, ignoring: Unit display-manager.service failed to load: No such file or directory.
[ OK ] Started Forward Password Requests to Wall Directory Watch.
Failed to reset devices.list on /lxc/103: Permission denied
Failed to reset devices.list on /lxc/103/user.slice: Permission denied
[ OK ] Created slice User and Session Slice.
[ OK ] Reached target Encrypted Volumes.
[ OK ] Listening on Journal Socket.
[ OK ] Reached target Remote File Systems.
[ OK ] Listening on Journal Audit Socket.
[ OK ] Listening on /dev/initctl Compatibility Named Pipe.
Failed to reset devices.list on /lxc/103/system.slice: Permission denied
[ OK ] Created slice System Slice.
Failed to reset devices.list on /lxc/103/system.slice/dev-mqueue.mount: Permission denied
Mounting POSIX Message Queue File System...
Failed to reset devices.list on /lxc/103/system.slice/system-container\x2dgetty.slice: Permission denied
[ OK ] Created slice system-container\x2dgetty.slice.
Failed to reset devices.list on /lxc/103/system.slice/dev-hugepages.mount: Permission denied
Mounting Huge Pages File System...
Failed to reset devices.list on /lxc/103/system.slice/systemd-remount-fs.service: Permission denied
Starting Remount Root and Kernel File Systems...
Failed to reset devices.list on /lxc/103/system.slice/tmp.mount: Permission denied
Mounting Temporary Directory...
Failed to reset devices.list on /lxc/103/system.slice/system-getty.slice: Permission denied
[ OK ] Created slice system-getty.slice.
[ OK ] Reached target Slices.
[ OK ] Started Dispatch Password Requests to Console Directory Watch.
[ OK ] Reached target Paths.
[ OK ] Listening on Device-mapper event daemon FIFOs.
[ OK ] Listening on Journal Socket (/dev/log).
Failed to reset devices.list on /lxc/103/system.slice/systemd-journald.service: Permission denied
Starting Journal Service...
[ OK ] Reached target Swap.
Failed to reset devices.list on /lxc/103/system.slice/proc-uptime.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/proc-meminfo.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/proc-stat.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/dev-lxc-tty1.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/proc-diskstats.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/dev-lxc-console.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/proc-sysrq\x2dtrigger.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/sys-devices-virtual-net.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/sys-fs-fuse-connections.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/proc-sys-net.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/something.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/dev-lxc-tty2.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/proc-cpuinfo.mount: Permission denied
Failed to reset devices.list on /lxc/103/system.slice/-.mount: Permission denied
Failed to reset devices.list on /lxc/103/init.scope: Permission denied
dev-mqueue.mount: Mount process exited, code=exited status=219
[FAILED] Failed to mount POSIX Message Queue File System.
See 'systemctl status dev-mqueue.mount' for details.
dev-mqueue.mount: Unit entered failed state.
dev-hugepages.mount: Mount process exited, code=exited status=219
[FAILED] Failed to mount Huge Pages File System.
See 'systemctl status dev-hugepages.mount' for details.
dev-hugepages.mount: Unit entered failed state.
systemd-remount-fs.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Remount Root and Kernel File Systems.
See 'systemctl status systemd-remount-fs.service' for details.
systemd-remount-fs.service: Unit entered failed state.
systemd-remount-fs.service: Failed with result 'exit-code'.
tmp.mount: Mount process exited, code=exited status=219
[FAILED] Failed to mount Temporary Directory.
See 'systemctl status tmp.mount' for details.
[DEPEND] Dependency failed for Basic System.
[DEPEND] Dependency failed for D-Bus System Message Bus.
dbus.service: Job dbus.service/start failed with result 'dependency'.
[DEPEND] Dependency failed for Getty on tty1.
getty at tty1.service: Job getty at tty1.service/start failed with result 'dependency'.
[DEPEND] Dependency failed for Login Service.
systemd-logind.service: Job systemd-logind.service/start failed with result 'dependency'.
[DEPEND] Dependency failed for Console Getty.
console-getty.service: Job console-getty.service/start failed with result 'dependency'.
[DEPEND] Dependency failed for Multi-User System.
[DEPEND] Dependency failed for Graphical Interface.
graphical.target: Job graphical.target/start failed with result 'dependency'.
multi-user.target: Job multi-user.target/start failed with result 'dependency'.
[DEPEND] Dependency failed for Permit User Sessions.
systemd-user-sessions.service: Job systemd-user-sessions.service/start failed with result 'dependency'.
[DEPEND] Dependency failed for Container Getty on /dev/pts/0.
container-getty at 0.service: Job container-getty at 0.service/start failed with result 'dependency'.
[DEPEND] Dependency failed for Container Getty on /dev/pts/1.
container-getty at 1.service: Job container-getty at 1.service/start failed with result 'dependency'.
basic.target: Job basic.target/start failed with result 'dependency'.
tmp.mount: Unit entered failed state.
systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
systemd-journal-flush.service: Job systemd-journal-flush.service/start failed with result 'dependency'.
systemd-journald.service: Unit entered failed state.
systemd-journald.service: Failed with result 'exit-code'.
Starting Journal Service...
[ OK ] Reached target Login Prompts.
Starting Create System Users...
Starting Load/Save Random Seed...
Starting Rebuild Hardware Database...
Starting Rebuild Dynamic Linker Cache...
systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
systemd-journald.service: Unit entered failed state.
systemd-journald.service: Failed with result 'exit-code'.
systemd-sysusers.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Create System Users.
See 'systemctl status systemd-sysusers.service' for details.
systemd-sysusers.service: Unit entered failed state.
systemd-sysusers.service: Failed with result 'exit-code'.
systemd-random-seed.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Load/Save Random Seed.
See 'systemctl status systemd-random-seed.service' for details.
systemd-random-seed.service: Unit entered failed state.
systemd-random-seed.service: Failed with result 'exit-code'.
systemd-hwdb-update.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Rebuild Hardware Database.
See 'systemctl status systemd-hwdb-update.service' for details.
systemd-hwdb-update.service: Unit entered failed state.
systemd-hwdb-update.service: Failed with result 'exit-code'.
ldconfig.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Rebuild Dynamic Linker Cache.
See 'systemctl status ldconfig.service' for details.
ldconfig.service: Unit entered failed state.
ldconfig.service: Failed with result 'exit-code'.
Starting Journal Service...
[ OK ] Reached target Local File Systems (Pre).
[ OK ] Reached target Local File Systems.
Starting Rebuild Journal Catalog...
Starting Create Volatile Files and Directories...
systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
systemd-journald.service: Unit entered failed state.
systemd-journald.service: Failed with result 'exit-code'.
Starting Journal Service...
[ ***] (1 of 3) A start job is running for Journal Service (9s / 1min 30s)systemd-journal-catalog-update.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Rebuild Journal Catalog.
See 'systemctl status systemd-journal-catalog-update.service' for details.
systemd-journal-catalog-update.service: Unit entered failed state.
systemd-journal-catalog-update.service: Failed with result 'exit-code'.
systemd-tmpfiles-setup.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
systemd-tmpfiles-setup.service: Unit entered failed state.
systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
Starting Update UTMP about System Boot/Shutdown...
Starting Update is Completed...
systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
systemd-journald.service: Unit entered failed state.
systemd-journald.service: Failed with result 'exit-code'.
Starting Journal Service...
[ ***] (1 of 3) A start job is running for Update UTMP about System Boot/Shutdown (19s / no limit)systemd-update-utmp.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Update UTMP about System Boot/Shutdown.
See 'systemctl status systemd-update-utmp.service' for details.
systemd-update-utmp.service: Unit entered failed state.
systemd-update-utmp.service: Failed with result 'exit-code'.
systemd-update-done.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Update is Completed.
See 'systemctl status systemd-update-done.service' for details.
systemd-update-done.service: Unit entered failed state.
systemd-update-done.service: Failed with result 'exit-code'.
systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
systemd-journald.service: Unit entered failed state.
systemd-journald.service: Failed with result 'exit-code'.
Starting Journal Service...
[** ] A start job is running for Journal Service (9s / 1min 30s)systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
systemd-journald.service: Unit entered failed state.
systemd-journald.service: Failed with result 'exit-code'.
Starting Journal Service...
[ *** ] A start job is running for Journal Service (9s / 1min 30s)systemd-journald.service: Main process exited, code=exited, status=219/CGROUP
[FAILED] Failed to start Journal Service.
See 'systemctl status systemd-journald.service' for details.
systemd-journald.service: Unit entered failed state.
systemd-journald.service: Failed with result 'exit-code'.
Starting Journal Service...
[ *** ] A start job is running for Journal Service (7s / 1min 30s)Received SIGRTMIN+14.
More information about the lxc-devel
mailing list