[lxc-users] LXC, criu and cgroups...

[Dirk Geschke]

Hi Tycho,

> Is it doing i/o? Looks to me like it's (trying) to send the rootfs.
> Adding the f argument to ps (i.e. something like `ps auxf`) will show
> it in tree form, so you can figure out which rsync server corresponds
> to which lxd easily.

the processes followed the PIDs, and no, there was no traffic.

I killed the rsync processs, restarted it all and now I got further.
The error messages are still there, but traffic was send to port 8443
across the network. But finally I got a second error.

On the source host:

   2015/04/07 22:49:29 got error getting next reader websocket: close 1005 , &{{%!s(*net.netFD=&{{10 0 0} 18 1 1 false unix 0xc2080e39e0 0xc2080e3a00 {140705565387096}})}}
   2015/04/07 22:50:24 got error getting next reader websocket: close 1005 , &{{%!s(*net.netFD=&{{10 0 0} 18 1 1 false unix 0xc2080e2720 0xc2080e2740 {140705565387288}})}}
   2015/04/07 22:50:25 operation %!s(func() shared.OperationResult=0x4d0120) finished: { restore failed}

and on the target host:

   2015/04/07 22:49:29 got error getting next reader websocket: close 1005 , &{%!s(*os.File=&{0xc20809f860}) {{%!s(int32=0) %!s(uint32=0)} %!s(uint32=1)} <nil>}
   2015/04/07 22:50:24 got error getting next reader websocket: close 1005 , &{%!s(*os.File=&{0xc20809e840}) {{%!s(int32=0) %!s(uint32=0)} %!s(uint32=1)} <nil>}
   2015/04/07 22:50:26 operation %!s(func() shared.OperationResult=0x4ccba0) finished: { restore failed}

But now the move command does fail with an error:

   lxcuser at karl:~$ lxc move otto:ubuntix local:ubuntix
   error: restore failed

And now I have a migration-restore log, it ends up with:

   (00.004890) Warn  (cr-restore.c:1016): Set CLONE_PARENT | CLONE_NEWPID but it might cause restore problem,because not all kernels support such clone flags combinations!
   (00.004899) Forking task with 1 pid (flags 0x7c028000)
   (00.004919) Saved netns fd for links restore
   (00.005335) Wait until namespaces are created
   (00.006008) UNS: Daemon started
   (00.006873) Running setup-namespaces scripts
   (00.006900)     [/usr/local/share/lxc/lxc-restore-net]
   (00.027303)      1: Restoring namespaces 1 flags 0x7c028000
   (00.027375)      1: Error (image.c:255): Unable to open netdev-8.img: Permission denied
   (00.040629) UNS: calling 0x456140 (-1, 1)
   (00.040671) UNS: daemon calls 0x456140 (-1, 1)
   (00.040681) UNS: `- daemon exits w/ 0
   (00.040938) UNS: daemon stopped
   (00.040946) Error (cr-restore.c:1879): Restoring FAILED.

So it looks here like a problem wiht lxc-restore-net. But why is
it not able to read netdev-8.img? The file is cleaned up, but 
some old, failing syncs show that everyone can read it?

  -rw-r--r-- 1 root root  53 Apr  7 21:51 lxd_migration_231938523/netdev-8.img

But we are stepping ahead...

Any ideas how to debug this? Hmm, lxc-restore-net seems to be 
a shell script...

Best regards

Dirk
-- 
+----------------------------------------------------------------------+
| Dr. Dirk Geschke       / Plankensteinweg 61    / 85435 Erding        |
| Telefon: 08122-559448  / Mobil: 0176-96906350 / Fax: 08122-9818106   |
| dirk at geschke-online.de / dirk at lug-erding.de  / kontakt at lug-erding.de |
+----------------------------------------------------------------------+