[Lxc-users] lxc-wait doesn't notice container shutdown

[Christoph Willing]

On 08/07/2013, at 11:02 PM, Michael H. Warfield <mhw at WittsEnd.com> wrote:

> On Mon, 2013-07-08 at 18:27 +1000, Christoph Willing wrote: 
>> On 05/07/2013, at 9:53 PM, Serge Hallyn <serge.hallyn at ubuntu.com> wrote:
>> 
>>> Quoting Christoph Willing (cwilling at users.sourceforge.net):
>>>> Since upgrading from lxc-0.7.5 to 0.9.0 I have a problem with lxc-wait.
>>>> 
>>>> Previously, scripts containing an lxc-wait for the STOPPED state would continue as expected when the nominated container shut itself down i.e. the script received the STOPPED state and lxc-wait exits. However with 0.9.0, lxc-wait doesn't seem to receive the STOPPED state when the container shuts itself down  - the scripts just keep waiting. I can run lxc-stop manually, whereupon the waiting script then sees that the container gets the message and continues as before.
>>>> 
>>>> On the other hand, the same scripts see the RUNNING state of a newly started container and continue execution as before.
>>>> 
>>>> So although lxc-wait is working (receives states sent explicitly via lxc-start/stop), it no longer receives any indication from the container that is is shutting down.
>>>> 
>>>> Is this new behaviour expected in 0.9.0?
>>> 
>>> No it sounds unexpected.  Would you be able to code the above into a
>>> little test script to reproduce?  (something like
>>> 
>>> 	sudo lxc-create -t ubuntu -n x1
>>> 	sudo lxc-start -n x1 -d
>>> 	sudo nohup lxc-wait -s STOPPED -n x1 > /tmp/outout 2>&1 &
>>> 	pid=$!
>>> 	sudo lxc-sttach -n x1 -- poweroff
>>> 	tail /tmp/outout
>>> 	ps -p $pid && echo "lxc-wait still running - FAIL"
>>> 	ps -p $pid || echo "lxc-wait exited - PASS"
>>> 
>>> )
>>> 
>>> Also please tell us which distro+release you're on and the exact package
>>> or upstream git version (there have been very recent changes...)  Is
>>> lxc-wait a script or a program in yours?  (which lxc-wait;  file `which lxc-wait`)
>> 
>> I found & fixed the problem but, for interest, the affected distro is Slackware "current" (the next release), using package lxc-0.9.0-x86_64-1 from which lxc-wait is a program:
>> 
>> chris at current:~$ which lxc-wait;  file `which lxc-wait`
>> /usr/bin/lxc-wait
>> /usr/bin/lxc-wait: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), stripped
>> 
>> 
>> Since there is no distributed lxc-slackware template, I've made my own
>> which I install locally. I've been using the same basic template for
>> about the last 3 Slackware releases and have had no problems. Part of
>> the the template's execution involves patching some of Slackware's
>> default rc.d scripts for use in the container. For some reason, the
>> patching removes the reboot/poweroff calls in rc.6 (the shutdown
>> script). I don't recall now why that was done - whether it was
>> mistaken or intentional - yet everything had all worked as expected
>> until now. Anyway, assuming it was a mistake and so changing my
>> template to leave the reboot/poweroff calls intact has fixed the
>> problem - lxc-wait receives the STOPPED state again.
> 
> Historically, removing the calls to "reboot/poweroff" were done, at the
> very least, to circumvent the "read-only" remounts of mounted file
> systems in the host.  We've been plagued by that problem on and off
> again for some time and never really nailed it.  I think the latest
> fixes for dealing with systemd based systems is ultimately incompatible
> with some of the host workarounds (bind mounts) which some of use were
> using to avoid that problem as well.  There may have been other reasons
> but that was, most emphatically, one of them.

Thanks for jogging my memory Mike.

I do now remember the problem of read-only remounts and it explains why I would have removed the reboot/shutdown calls. Now that I need them back in, I'd better check my containers & scripts for a return of those bad behaviours - although I didn't notice any of them while troubleshooting my absence of STOPPED state problem, so fingers crossed ..

chris