[lxc-users] LXD Live Migration

Thu Nov 5 16:58:04 UTC 2015

Hi Jamie,

Thanks for trying it out.

On Thu, Nov 05, 2015 at 11:39:43AM +0000, Jamie Brown wrote:
> Hello again,
> 
> Oddly, I've now re-installed the old server and configured it identically to before (except now using RAID) and tried migrating a container back and I am getting a different failure;
> 
> # lxc move host2:test host1:test
> 
> error: Error transferring container data: restore failed:
> (00.007414)      1: Error (mount.c:2030): Can't mount at ./dev/.lxd-mounts: No such file or directory
> (00.026443) Error (cr-restore.c:1939): Restoring FAILED.
> 
> The container appears in the remote container list whilst moving, but then after failure it is deleted and it is in the STOPPED state on the source host.

Right, the restore failed, so the container had already been stopped
from the dump, so it was stopped on the target. What we should really
do is leave it in a frozen state after the dump, and once the restore
succeeds then we can kill it. Hopefully that's something I can
implement this cycle.

As for the actual error, sounds like the target LXD didn't have
shmounts but the source one did. Are they using different backing
stores? What version of LXD are they?

> 
> Here's the output from the log, not sure how much is relevant to the migration attempt.
> 
> # lxc info --show-log test
> ...
> lxc 1446723150.396 DEBUG    lxc_start - start.c:__lxc_start:1210 - unknown exit status for init: 9
>             lxc 1446723150.396 DEBUG    lxc_start - start.c:__lxc_start:1215 - Pushing physical nics back to host namespace
>             lxc 1446723150.396 DEBUG    lxc_start - start.c:__lxc_start:1218 - Tearing down virtual network devices used by container
>             lxc 1446723150.396 WARN     lxc_conf - conf.c:lxc_delete_network:2939 - failed to remove interface '(null)'
>             lxc 1446723150.396 INFO     lxc_error - error.c:lxc_error_set_and_log:55 - child <10499> ended on signal (9)
>             lxc 1446723150.396 WARN     lxc_conf - conf.c:lxc_delete_network:2939 - failed to remove interface '(null)'
>             lxc 1446723295.520 WARN     lxc_cgmanager - cgmanager.c:cgm_get:993 - do_cgm_get exited with error
>             lxc 1446723295.522 WARN     lxc_cgmanager - cgmanager.c:cgm_get:993 - do_cgm_get exited with error
> 
> 
> If I try to migrate a container in the reverse direction, I get a similar error;
> 
> # lxc move host1:test1 host2:test1
> error: Error transferring container data: restore failed:
> (00.001093) Error (cgroup.c:1204): cg: 	Can't mount controller dir .criu.cgyard.aOuQtF/net_cls: No such file or directory

This is probably because the kernel on host1 is newer than the
kernel on host2 and has net_cls cgroup support where as host2's
doesn't.

Tycho

> 
> 
> 
> Any ideas?
> 
> -- Jamie
> 
> 
> 
> On 05/11/2015, 08:05, "lxc-users on behalf of Jamie Brown" <lxc-users-bounces at lists.linuxcontainers.org on behalf of Jamie.Brown at mpec.co.uk> wrote:
> 
> >Thanks Tycho, installing CRIU solved the problem;
> >
> ># apt-get install criu
> >
> >Should this package not be included as a dependency for LXD, or at least provide a meaningful warning if the package isn’t available? It seems odd to advertise out-the-box live migration in LXD, but then have to install another package to provide it.
> >
> >Is this in the documentation anywhere?
> >
> >Thanks again.
> >
> >-- Jamie
> >
> >
> >
> >
> >On 04/11/2015, 16:47, "lxc-users on behalf of Tycho Andersen" <lxc-users-bounces at lists.linuxcontainers.org on behalf of tycho.andersen at canonical.com> wrote:
> >
> >>On Wed, Nov 04, 2015 at 01:48:44PM +0000, Jamie Brown wrote:
> >>> Greetings all.
> >>> 
> >>> I’ve been using LXD in a development environment for a few weeks and so far very impressed, 
> >>> I can see a really bright future for this technology!
> >>> 
> >>> However, today I thought I’d try out the live migration, based on the following guide;
> >>> https://insights.ubuntu.com/2015/05/06/live-migration-in-lxd/
> >>> 
> >>> I believe I have followed the steps correctly, however when I run the move command, I 
> >>> receive the following output;
> >>> 
> >>> # lxc move host1:test host2:test
> >>> error: Error transferring container data: checkpoint failed:
> >>> Problem accessing CRIU log: open /tmp/lxd_migration_899480871/dump.log: no such file or directory
> >>> 
> >>> The file it is referring to above doesn't exist. However, there are other lxd_migration_* 
> >>> directories with different numbers appended. Each time I attempt the migration a new directory 
> >>> is created (e.g. lxd_migration_192965652), but there is no dump.log in there.
> >>> 
> >>> The migration doesn't create a log file as per the guide above in;
> >>> /var/log/lxd/test/migration_{dump|restore}_.log
> >>> 
> >>> Steps I've taken;
> >>> 
> >>> - Copied all profiles from host1 to host2
> >>> - Added the migratable profile to the container
> >>> - Removed lxcfs package (on both hosts)
> >>> - Added the remote HTTPS hosts for both the local and remote hosts
> >>> 
> >>> Both hosts are running Ubuntu 14.04.3 LTS (x64) with LXD version 0.21.
> >>> 
> >>> The only difference I can tell between my hosts and the guide is that the 'migratable'
> >>> profile (which came out-the-box with my LXD installation) doesn't contain the autostart
> >>> entries as in the guide above;
> >>> 
> >>> # lxc profile show migratable
> >>> name: migratable
> >>> config:
> >>>   raw.lxc: |-
> >>>     lxc.console = none
> >>>     lxc.cgroup.devices.deny = c 5:1 rwm
> >>>     lxc.seccomp =
> >>>   security.privileged: "true"
> >>> devices: {}
> >>> 
> >>> 
> >>> Any help would be much appreciated!
> >>
> >>Have you installed CRIU? lxc info --show-log test probably has more
> >>info about what failed, but my guess is that it can't find CRIU if you
> >>haven't installed it.
> >>
> >>Tycho
> >>
> >>> Thank you,
> >>> 
> >>> Jamie
> >>> 
> >>> _______________________________________________
> >>> lxc-users mailing list
> >>> lxc-users at lists.linuxcontainers.org
> >>> http://lists.linuxcontainers.org/listinfo/lxc-users
> >>_______________________________________________
> >>lxc-users mailing list
> >>lxc-users at lists.linuxcontainers.org
> >>http://lists.linuxcontainers.org/listinfo/lxc-users
> >_______________________________________________
> >lxc-users mailing list
> >lxc-users at lists.linuxcontainers.org
> >http://lists.linuxcontainers.org/listinfo/lxc-users
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users