[lxc-users] How to configure lxd to comply with commands
Stéphane Graber
stgraber at ubuntu.com
Wed Aug 17 22:16:23 UTC 2016
You're making progress, now the two LXD daemons are definitely talking
to each other. You're running into CRIU issues :)
If you did not intend to do live migration, then stop the source
container and try moving it again.
If you did intend to use live migration, then there are quite a few
issues still being worked on:
https://launchpad.net/ubuntu/+source/criu/+bugs
Yours appears to be number 3 on that list.
You can workaround it by unmounting /sys/kernel/debug/tracing/ and
/sys/kernel/debug/ in your source container.
On Wed, Aug 17, 2016 at 03:11:38PM -0700, jjs - mainphrame wrote:
> Aw, shucks, it was looking like it might work - but no joy.
>
> root at olympia:~# lxc config set core.https_address 192.168.111.193:8443
> root at olympia:~# lxc move kangal lxd1:
> error: Error transferring container data: migration restore failed
> (00.192913) 1: Error (mount.c:2406): mnt: Can't mount at
> ./sys/kernel/debug: Invalid argument
> (00.206829) Error (cr-restore.c:1352): 23251 killed by signal 9
> (00.255466) Error (cr-restore.c:2182): Restoring FAILED.
>
> Any ideas from the error message? The only thing I see in the logs is this,
> only sending side:
> Aug 17 14:49:15 olympia kernel: [317636.764104] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317636.897001] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317636.965993] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317637.025985] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317637.099008] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317637.223028] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317637.276026] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317637.407065] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317637.466059] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317637.559652] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317637.671028] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:16 olympia kernel: [317637.760296] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:17 olympia kernel: [317637.840036] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:17 olympia kernel: [317637.944061] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:17 olympia kernel: [317638.017056] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:17 olympia kernel: [317638.074090] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:20 olympia kernel: [317640.800234] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:20 olympia kernel: [317641.335276] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:25 olympia kernel: [317646.079503] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:25 olympia kernel: [317646.273487] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:25 olympia kernel: [317646.444487] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:25 olympia kernel: [317646.694511] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:26 olympia kernel: [317647.107553] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:27 olympia kernel: [317648.322648] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:27 olympia kernel: [317648.549680] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:27 olympia kernel: [317648.606610] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:28 olympia kernel: [317649.377678] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:28 olympia kernel: [317649.445660] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:28 olympia kernel: [317649.509650] cgroup: new mount options
> do not match the existing superblock, will be ignored
> Aug 17 14:49:29 olympia kernel: [317649.921865] Netfilter messages via
> NETLINK v0.30.
> Aug 17 14:49:29 olympia kernel: [317649.946436] ctnetlink v0.93:
> registering with nfnetlink.
> Aug 17 14:49:29 olympia kernel: [317650.033557] br0: port 3(vethVHQ0XE)
> entered disabled state
> Aug 17 14:49:29 olympia kernel: [317650.039297] device vethVHQ0XE left
> promiscuous mode
> Aug 17 14:49:29 olympia kernel: [317650.039302] br0: port 3(vethVHQ0XE)
> entered disabled state
> Aug 17 14:49:30 olympia kernel: [317651.765330] audit_printk_skb: 69
> callbacks suppressed
> Aug 17 14:49:30 olympia kernel: [317651.765333] audit: type=1400
> audit(1471470570.966:77): apparmor="STATUS" operation="profile_remove"
> info="profile does not exist" error=-2 profile="unconfined"
> name="lxd-kangal_</var/lib/lxd>" pid=25080 comm="apparmor_parser"
>
>
> And this, on the receiving side:
> Aug 17 14:49:56 ronnie kernel: [107460.037655] audit: type=1400
> audit(1471470596.676:89): apparmor="STATUS" operation="profile_load"
> profile="unconfined" name="lxd-kangal_</var/lib/lxd>" pid=23239
> comm="apparmor_parser"
> Aug 17 14:49:56 ronnie NetworkManager[2696]: <warn> [1471470596.8817]
> device (vethIA06PF): failed to find device 11 'vethIA06PF' with udev
> Aug 17 14:49:56 ronnie NetworkManager[2696]: <info> [1471470596.8828]
> manager: (vethIA06PF): new Veth device
> (/org/freedesktop/NetworkManager/Devices/10)
> Aug 17 14:49:56 ronnie NetworkManager[2696]: <info> [1471470596.8991]
> devices added (path: /sys/devices/virtual/net/vethIA06PF, iface: vethIA06PF)
> Aug 17 14:49:56 ronnie NetworkManager[2696]: <info> [1471470596.8991]
> device added (path: /sys/devices/virtual/net/vethIA06PF, iface:
> vethIA06PF): no ifupdown configuration found.
> Aug 17 14:49:57 ronnie NetworkManager[2696]: <info> [1471470597.0303]
> device (vethIA06PF): driver 'veth' does not support carrier detection.
> Aug 17 14:49:57 ronnie NetworkManager[2696]: <info> [1471470597.0351]
> devices removed (path: /sys/devices/virtual/net/vethIA06PF, iface:
> vethIA06PF)
>
>
>
> Jake
>
>
> On Wed, Aug 17, 2016 at 2:50 PM, jjs - mainphrame <jjs at mainphrame.com>
> wrote:
>
> > Thanks Stephane, that goes to the root of the problem. I must have been
> > thinking in the back of my mind that I'd already set that, but come to
> > think of it, that was back when this box was running 14.04. before a clean
> > install of 16.04
> >
> > Jake
> >
> > On Wed, Aug 17, 2016 at 1:45 PM, Stéphane Graber <stgraber at ubuntu.com>
> > wrote:
> >
> >> So one way around the problem would be with:
> >>
> >> lxc config set core.https_address 192.168.111.193:8443
> >>
> >> on your olympia host. This will force LXD to use that IP during
> >> container transfer and should fix your problem.
> >>
> >> On Wed, Aug 17, 2016 at 01:28:52PM -0700, jjs - mainphrame wrote:
> >> > Hi Stephane -
> >> >
> >> > lxd1, lxd2, and kangal are all on the same lan, and connectivity is
> >> good:
> >> >
> >> > root at olympia:~# ssh kangal w
> >> > root at kangal's password:
> >> > 13:26:53 up 3 days, 14:50, 0 users, load average: 0.29, 0.37, 0.37
> >> > USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
> >> >
> >> > root at ronnie:~# ssh kangal w
> >> > root at kangal's password:
> >> > 13:27:09 up 3 days, 14:51, 0 users, load average: 0.29, 0.36, 0.37
> >> > USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
> >> >
> >> > Regards,
> >> >
> >> > Jake
> >> >
> >> >
> >> > On Wed, Aug 17, 2016 at 1:24 PM, Stéphane Graber <stgraber at ubuntu.com>
> >> > wrote:
> >> >
> >> > > On Wed, Aug 17, 2016 at 01:14:43PM -0700, jjs - mainphrame wrote:
> >> > > > Greetings,
> >> > > >
> >> > > > I'm running lxd version 2.0.3-0ubuntu1~ubuntu16.04.2
> >> > > >
> >> > > > I'm trying to get lxd to correctly execute a move of a container
> >> from one
> >> > > > lxd host to another. I have two ubuntu 16.04 hosts, ronnie
> >> (designated as
> >> > > > lxd1) and olympia (designated as lxd2):
> >> > > >
> >> > > > root at olympia:~# lxc remote list
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | NAME | URL |
> >> PROTOCOL
> >> > > > | PUBLIC | STATIC |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | images | https://images.linuxcontainers.org | lxd
> >> > > > | YES | NO |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | local (default) | unix:// | lxd
> >> > > > | NO | YES |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | lxd1 | https://192.168.111.20:8443 | lxd
> >> > > > | NO | NO |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | ubuntu | https://cloud-images.ubuntu.com/releases |
> >> > > > simplestreams | YES | YES |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | ubuntu-daily | https://cloud-images.ubuntu.com/daily |
> >> > > > simplestreams | YES | YES |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > >
> >> > > > root at ronnie:~# lxc remote list
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | NAME | URL |
> >> PROTOCOL
> >> > > > | PUBLIC | STATIC |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | images | https://images.linuxcontainers.org | lxd
> >> > > > | YES | NO |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | local (default) | unix:// | lxd
> >> > > > | NO | YES |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | lxd2 | https://192.168.111.193:8443 | lxd
> >> > > > | NO | NO |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | ubuntu | https://cloud-images.ubuntu.com/releases |
> >> > > > simplestreams | YES | YES |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > > | ubuntu-daily | https://cloud-images.ubuntu.com/daily |
> >> > > > simplestreams | YES | YES |
> >> > > > +-----------------+-----------------------------------------
> >> > > -+---------------+--------+--------+
> >> > > >
> >> > > > You can see that the remotes are configured with their local lan
> >> > > addresses.
> >> > > > So far so good?
> >> > > >
> >> > > >
> >> > > >
> >> > > > Here are the 2 containers currently on lxd2:
> >> > > > root at olympia:~# lxc list
> >> > > > +--------+---------+-----------------------+------+---------
> >> > > ---+-----------+
> >> > > > | NAME | STATE | IPV4 | IPV6 | TYPE |
> >> > > SNAPSHOTS |
> >> > > > +--------+---------+-----------------------+------+---------
> >> > > ---+-----------+
> >> > > > | akita | RUNNING | 192.168.111.22 (eth0) | | PERSISTENT | 0
> >> > > |
> >> > > > +--------+---------+-----------------------+------+---------
> >> > > ---+-----------+
> >> > > > | kangal | RUNNING | 192.168.111.44 (eth0) | | PERSISTENT | 0
> >> > > |
> >> > > > +--------+---------+-----------------------+------+---------
> >> > > ---+-----------+
> >> > > >
> >> > > >
> >> > > >
> >> > > > Now, I try to move a container from lxd2 to lxd1:
> >> > > > root at olympia:~# lxc move kangal lxd1:
> >> > > > error: Error transferring container data: Unable to connect to:
> >> > > > 192.168.1.8:8443
> >> > >
> >> > > Is there any way for lxd1 to connect to kangal?
> >> > >
> >> > > The way LXD currently deals with cross-host communication is that the
> >> > > client has the source host issue a token which the client sends to the
> >> > > target along with instructions on how to connect to the source.
> >> > >
> >> > > The target then directly connects to the source to fetch the data
> >> > > (in this case, the container).
> >> > >
> >> > > This means that there must be a way for the target to connect to the
> >> > > source on the LXD port without being blocked by firewalls or going
> >> > > through NAT.
> >> > >
> >> > >
> >> > > Some more details can be found at
> >> > > https://www.stgraber.org/2016/04/12/lxd-2-0-remote-hosts-
> >> > > and-container-migration-612/
> >> > > in the "network requirements" and "how this all works" sections.
> >> > >
> >> > >
> >> > > The client is currently supposed to iterate through all the IPs that
> >> the
> >> > > source server advertises (see addresses field in "lxc info kangal"),
> >> the
> >> > > one that's in the error message is usually the last one of those.
> >> > >
> >> > >
> >> > > If core.https_address is set on the source host, then only that
> >> address
> >> > > will be attempted since it's the only one LXD will be listening on.
> >> > >
> >> > >
> >> > > As mentioned in the blog post, we do have a plan to improve the
> >> > > situation by having the client relay the data in cases where the two
> >> > > servers can't talk, but we haven't made much progress on implementing
> >> > > that so far.
> >> > >
> >> > > >
> >> > > > Why is it trying to connect to 192.168.1.8? That is a local wireless
> >> > > > address on lxd2, but it was never mentioned in any lxd
> >> configuration:
> >> > > >
> >> > > > root at olympia:~# ifconfig wlp3s0
> >> > > > wlp3s0 Link encap:Ethernet HWaddr 80:56:f2:05:ce:6c
> >> > > > inet addr:192.168.1.8 Bcast:192.168.1.255
> >> Mask:255.255.255.0
> >> > > > inet6 addr: fe80::8256:f2ff:fe05:ce6c/64 Scope:Link
> >> > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> >> > > > RX packets:401712 errors:0 dropped:0 overruns:0 frame:0
> >> > > > TX packets:219214 errors:0 dropped:0 overruns:0 carrier:0
> >> > > > collisions:0 txqueuelen:1000
> >> > > > RX bytes:95097222 (95.0 MB) TX bytes:34201360 (34.2 MB)
> >> > > >
> >> > > >
> >> > > > So my question is, how do we get lxd to ignore the local wireless
> >> IP, and
> >> > > > execute the lxc move command using the configured IPs?
> >> > > >
> >> > > > Regards,
> >> > > >
> >> > > > Jake
> >> > >
> >> > > > _______________________________________________
> >> > > > lxc-users mailing list
> >> > > > lxc-users at lists.linuxcontainers.org
> >> > > > http://lists.linuxcontainers.org/listinfo/lxc-users
> >> > >
> >> > >
> >> > > --
> >> > > Stéphane Graber
> >> > > Ubuntu developer
> >> > > http://www.ubuntu.com
> >> > >
> >> > > _______________________________________________
> >> > > lxc-users mailing list
> >> > > lxc-users at lists.linuxcontainers.org
> >> > > http://lists.linuxcontainers.org/listinfo/lxc-users
> >> > >
> >>
> >> > _______________________________________________
> >> > lxc-users mailing list
> >> > lxc-users at lists.linuxcontainers.org
> >> > http://lists.linuxcontainers.org/listinfo/lxc-users
> >>
> >>
> >> --
> >> Stéphane Graber
> >> Ubuntu developer
> >> http://www.ubuntu.com
> >>
> >> _______________________________________________
> >> lxc-users mailing list
> >> lxc-users at lists.linuxcontainers.org
> >> http://lists.linuxcontainers.org/listinfo/lxc-users
> >>
> >
> >
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users
--
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20160817/964fccf2/attachment.sig>
More information about the lxc-users
mailing list