[lxc-users] lxc-destroy not working when run by cron
Claudio Kuenzler
ck at claudiokuenzler.com
Fri Sep 25 07:00:46 UTC 2015
Running the same command with strace actually shows the command fails.
The relevant part:
----------------------- % snip % -----------------------
stat("/var/lib/lxc/mycontainer/config", {st_mode=S_IFREG|0644, st_size=808,
...}) = 0
pipe([3, 4]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7f8da4af2b90) = 23100
close(4) = 0
read(3, "\377\377\377\377", 4) = 4
close(3) = 0
wait4(23100, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 23100
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=23100,
si_status=1, si_utime=0, si_stime=0} ---
socket(PF_LOCAL, SOCK_STREAM, 0) = 3
connect(3, {sa_family=AF_LOCAL,
sun_path=@"/var/lib/lxc/mycontainer/command"}, 40) = -1 ECONNREFUSED
(Connection refused)
connect(3, {sa_family=AF_LOCAL,
sun_path=@"/var/lib/lxc/mycontainer/command"}, 110) = -1 ECONNREFUSED
(Connection refused)
close(3) = 0
stat("/var/lib/lxc/mycontainer/config", {st_mode=S_IFREG|0644, st_size=808,
...}) = 0
open("/run/lxc/lock//var/lib/lxc/mycontainer", O_RDWR|O_CREAT, 0600) = 3
fcntl(3, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = 0
pipe([4, 5]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7f8da4af2b90) = 23101
close(5) = 0
read(4, "\377\377\377\377", 4) = 4
close(4) = 0
wait4(23101, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 23101
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=23101,
si_status=1, si_utime=0, si_stime=0} ---
socket(PF_LOCAL, SOCK_STREAM, 0) = 4
connect(4, {sa_family=AF_LOCAL,
sun_path=@"/var/lib/lxc/mycontainer/command"}, 40) = -1 ECONNREFUSED
(Connection refused)
connect(4, {sa_family=AF_LOCAL,
sun_path=@"/var/lib/lxc/mycontainer/command"}, 110) = -1 ECONNREFUSED
(Connection refused)
close(4) = 0
open("/var/lib/lxc/mycontainer/lxc_snapshots", O_RDONLY) = -1 ENOENT (No
such file or directory)
geteuid() = 0
pipe2([4, 5], O_CLOEXEC) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7f8da4af2b90) = 23102
close(5) = 0
fcntl(4, F_GETFL) = 0 (flags O_RDONLY)
fstat(4, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f8da4aff000
lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
read(4, "", 4096) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=23102,
si_status=127, si_utime=0, si_stime=0} ---
close(4) = 0
munmap(0x7f8da4aff000, 4096) = 0
wait4(23102, [{WIFEXITED(s) && WEXITSTATUS(s) == 127}], 0, NULL) = 23102
stat("/dev/vglxc/mycontainer", {st_mode=S_IFBLK|0660, st_rdev=makedev(252,
0), ...}) = 0
open("/sys/dev/block/252:0/dm/uuid", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f8da4aff000
read(4,
"LVM-AGQfFleAZmtwXaRlf3qusphA68nyKV56qCiz847f5Jd9gR5Em5pzaJdSfo24s4br\n",
4096) = 69
close(4) = 0
munmap(0x7f8da4aff000, 4096) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
child_tidptr=0x7f8da4af2b90) = 23103
wait4(23103, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 23103
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=23103,
si_status=1, si_utime=0, si_stime=0} ---
write(2, "lxc_container: ", 15) = 15
write(2, "lxccontainer.c: lxcapi_destroy: 2073 ", 37) = 37
write(2, "Error destroying rootfs for mycontainer", 44) = 44
write(2, "\n", 1) = 1
fcntl(3, F_SETLK, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
close(3) = 0
write(2, "Destroying mycontainer failed\n", 35) = 35
exit_group(1) = ?
+++ exited with 1 +++
----------------------- % snip % -----------------------
last child pid 23103:
----------------------- % snip % -----------------------
set_robust_list(0x7f8da4af2ba0, 24) = 0
execve("/usr/bin/lvremove", ["lvremove", "-f", "/dev/vglxc/mycontainer"],
[/* 7 vars */]) = -1 ENOENT (No such file or directory)
execve("/bin/lvremove", ["lvremove", "-f", "/dev/vglxc/mycontainer"], [/* 7
vars */]) = -1 ENOENT (No such file or directory)
exit_group(1) = ?
+++ exited with 1 +++
----------------------- % snip % -----------------------
So the lvremove command is not found.
When I compare it with a manual run, the last child pid, which removes the
LV, goes through more paths to find the lvremove command and then continues:
----------------------- % snip % -----------------------
set_robust_list(0x7fde2db25ba0, 24) = 0
execve("/usr/local/sbin/lvremove", ["lvremove", "-f",
"/dev/vglxc/mycontainer"], [/* 34 vars */]) = -1 ENOENT (No such file or
directory)
execve("/usr/local/bin/lvremove", ["lvremove", "-f",
"/dev/vglxc/mycontainer"], [/* 34 vars */]) = -1 ENOENT (No such file or
directory)
execve("/usr/sbin/lvremove", ["lvremove", "-f", "/dev/vglxc/mycontainer"],
[/* 34 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/bin/lvremove", ["lvremove", "-f", "/dev/vglxc/mycontainer"],
[/* 34 vars */]) = -1 ENOENT (No such file or directory)
execve("/sbin/lvremove", ["lvremove", "-f", "/dev/vglxc/mycontainer"], [/*
34 vars */]) = 0
brk(0) = 0x1a48000
[...]
----------------------- % snip % -----------------------
Looking at thismakes me think of checking the $PATH... And indeed:
# echo $PATH as root on cli:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# cronjob:
58 08 * * * echo $PATH > /var/log/path.log
# cat /var/log/path.log
/usr/bin:/bin
OK, makes sense now.
On Fri, Sep 25, 2015 at 8:14 AM, Claudio Kuenzler <ck at claudiokuenzler.com>
wrote:
> Hello,
>
> Been pulling my hair out (what's left of it) because a script basically
> used for auto-re-creation of containers is working when I launch it
> manually, but the very same script does not work when run by cron.
>
> I narrowed it down to the "lxc-destroy" command which simply does not do
> anything when run by cron. You can reproduce it with a simple cronjob like
> this:
>
> 03 08 * * * lxc-stop -n mycontainer && lxc-destroy -n mycontainer
>
> What happens then? The container is correctly stopped - but remains
> stopped. The container is still showing up in lxc-ls (in state STOPPED).
>
> I suspect there is a problem deleting the logical volume (lxc.rootfs) of
> the container when lxc-destroy is launched, because a quick "lvs" after the
> cronjob ran through still shows the LV of the container.
>
> Ubuntu 14.04.1 LTS, 3.16.0-48-generic, lxc 1.0.7-0ubuntu0.2
>
> Any ideas?
>
> cheers,
> ck
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20150925/ca21e2ab/attachment.html>
More information about the lxc-users
mailing list