[lxc-users] Trying out migration, getting "error: checkpoint failed"

[Tycho Andersen]

On Tue, May 05, 2015 at 04:26:50PM +0100, Giles Thomas wrote:
> Hi Tycho,
> 
> On 05/05/15 14:43, Tycho Andersen wrote:
> >On Tue, May 05, 2015 at 02:18:23PM +0100, Giles Thomas wrote:
> >>On 04/05/15 14:15, Tycho Andersen wrote:
> >>>What versions of criu and liblxc do you have installed? Can you look in
> >>>/var/lib/lxd/lxc/<container>/lxc.log (or /var/log/lxd/<container>/lxc.log;
> >>>there was a bug fixed recently that moved it to the latter location).
> >>Heh, I hadn't realised that criu wasn't automatically installed as a
> >>dependency for lxd (might be worth adding to the blog post?), so I didn't
> >>have it installed at all.
> >Ah yeah. It seems I forgot to mention anything about criu at all :)
> >
> >>However, installing it doesn't fix the problem.  I currently have criu
> >>1.4-1~ubuntu14.04.1~ppa1 and liblxc1
> >>1.1.2+master~20150428-0938-0ubuntu1~trusty.   The former was installed just
> >>with an apt-get install criu, and the latter via following your blog post
> >>pretty much robotically...
> >>
> >>/var/log/lxd/migratee/lxc.log is present, but empty.
> >Does /var/lib/lxd/lxc/migratee/lxc.log exist?
> 
> Aha!  It doesn't, but there is a file /var/lib/lxd/lxc/migratee/log, which I
> missed last time around.   I just tried the move again, with a tail -f on
> that file running in the background, and got some output:
> 
>     root at XXXXXXX:/var/log/lxd/migratee# lxc move lxd:migratee lxd2:migratee
>                 lxc 1430839338.008 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.010 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.010 DEBUG    lxc_commands -
> commands.c:lxc_cmd_get_state:574 - 'migratee' is in 'RUNNING' state
>                 lxc 1430839338.010 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.012 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.014 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.014 DEBUG    lxc_commands -
> commands.c:lxc_cmd_get_state:574 - 'migratee' is in 'RUNNING' state
>                 lxc 1430839338.016 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.019 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.020 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.020 DEBUG    lxc_commands -
> commands.c:lxc_cmd_get_state:574 - 'migratee' is in 'RUNNING' state
>                 lxc 1430839338.022 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.025 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.027 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.027 DEBUG    lxc_commands -
> commands.c:lxc_cmd_get_state:574 - 'migratee' is in 'RUNNING' state
>                 lxc 1430839338.028 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.887 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.889 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839338.889 DEBUG    lxc_commands -
> commands.c:lxc_cmd_get_state:574 - 'migratee' is in 'RUNNING' state
>                 lxc 1430839340.029 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839340.031 DEBUG    lxc_commands -
> commands.c:lxc_cmd_handler:888 - peer has disconnected
>                 lxc 1430839340.031 DEBUG    lxc_commands -
> commands.c:lxc_cmd_get_state:574 - 'migratee' is in 'RUNNING' state
>     error: checkpoint failed
>     root at XXXXXXX:/var/log/lxd/migratee#
> 
> Anything useful there?  Or anything that points at any other logs I should
> be checking?

Can you check the lxd stderr by chance (probably lives in /var/log
somewhere depending on what init system you're using)? I suspect that
liblxc is rejecting dumping the container in its internal predump
checks, but the above log doesn't say way unfortunately.

Sorry for all the confusion, the logging stuff here is still a bit of
a mess, although a bit better on the current lxd master.

Tycho

> >You might try building criu from their git, I'm not sure the version
> >in our PPA is new enough to actually work with liblxc 1.1.2.
> 
> Sure, I can definitely give that a go -- I'll wait for half an hour or so,
> though, just in case you see the log above and immediately realise what
> stupid thing I've done wrong ;-)
> 
> 
> All the best,
> 
> Giles
> 
> -- 
> Giles Thomas <giles at pythonanywhere.com>
> 
> PythonAnywhere: Develop and host Python from your browser
> <https://www.pythonanywhere.com/>
> 
> A product from PythonAnywhere LLP
> 17a Clerkenwell Road, London EC1M 5RD, UK
> VAT No.: GB 893 5643 79
> Registered in England and Wales as company number OC378414.
> Registered address: 28 Ely Place, 3rd Floor, London EC1N 6TD, UK
> 
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users