[lxc-users] lxc progress and a few questions

Fri Apr 8 23:23:29 UTC 2016

Ubuntu 16.04, up to date -

After today's updates, including a kernel upgrade to 4.4.0-18, I tried
live migration again:

root at raskolnikov:~# lxc move third lxd2:

One hour later:

root at raskolnikov:~# lxc move third lxd2:

Still stuck, and the migration file in /var/log/lxd/third has not been created.

Tycho said on Mar 30 that the situation should be sorted soon, but
mentioned the git repo:
https://github.com/tych0/criu/tree/cgroup-root-mount

Should live migration work with criu from git?

Feel free to advise me on what information I can supply, not only for
the ct migration issues, but also for the new dhcp issue

Regards,

Jake

On Thu, Apr 7, 2016 at 11:01 PM, jjs - mainphrame <jjs at mainphrame.com> wrote:
> (Bump) -
>
> Any thoughts on what to try for the CT migration and dhcp issues?
> Running up to date ubuntu 16.04 beta -
>
> Regards,
>
> Jake
>
> On Wed, Apr 6, 2016 at 3:18 PM, jjs - mainphrame <jjs at mainphrame.com> wrote:
>> Greetings -
>>
>> I'be not yet been able to reproduce that one shining moment from Mar
>> 29 when live migration of privileged containers was working, under
>> kernel 4.4.0-15
>>
>> To recap. live container migration broke with 4.4.0-16, and is still
>> broken in 4.4.0-17 - but  now, instead of producing an error message,
>> an attempt to live migrate a container merely hangs forever. Is that
>> expected, or should I be seeing something more? BTW - the migration
>> dump log for that container hasn't been touched for a week. I'll be
>> glad to supply more info if this is not a known issue.
>>
>> Recent updates seem to have created a new problem. the CTs which
>> configure their own network settings work (aside from migration) but
>> none of the CTs which depend on dhcp are getting IPs. BTW I'm using a
>> bridge connected to my local network and dhcp, not the default lxc
>> dhcp server. I see the packets on the host bridge, but they don't
>> reach the dhcp server. I'd be curious to know if there have been any
>> dhcp issues since recent updates. If not, I'll need to troubleshoot
>> other causes, but it's odd that dhcp simply stops working for all CTs
>> on both lxd hosts after updates.
>>
>> Jake
>>
>>
>> On Wed, Mar 30, 2016 at 6:27 AM, Tycho Andersen
>> <tycho.andersen at canonical.com> wrote:
>>> On Tue, Mar 29, 2016 at 11:17:26PM -0700, jjs - mainphrame wrote:
>>>> Well, I've found some interesting things here today. I created a couple of
>>>> privileged xenial containers, and sure enough, I was able to live migrate
>>>> them back and forth between the 2 lxd hosts.
>>>>
>>>> So far, so good.
>>>>
>>>> Then I did an apt upgrade - among the changes was a kernel change from
>>>>  4.4.0-15 to 4.4.0-16 - and live migration stopped working.
>>>>
>>>> Here are the failure messages that resulted from attempting the very same
>>>> live migrations that worked before the upgrade and reboot into 4.4.0-16:
>>>>
>>>> root at raskolnikov:~# lxc move akira lxd2:
>>>> error: Error transferring container data: checkpoint failed:
>>>> (00.092234) Error (mount.c:740): mnt: 83:./sys/fs/cgroup/devices doesn't
>>>> have a proper root mount
>>>> (00.098187) Error (cr-dump.c:1600): Dumping FAILED.
>>>>
>>>>
>>>> root at ronnie:~# lxc move third lxd:
>>>> error: Error transferring container data: checkpoint failed:
>>>> (00.076107) Error (mount.c:740): mnt: 326:./sys/fs/cgroup/perf_event
>>>> doesn't have a proper root mount
>>>> (00.080388) Error (cr-dump.c:1600): Dumping FAILED.
>>>
>>> Yep, this is a known issue with -16. We need both a kernel patch and a
>>> patch to CRIU before it will start working again. I have a branch at:
>>>
>>> https://github.com/tych0/criu/tree/cgroup-root-mount
>>>
>>> which should work if you want to keep playing with it, but hopefully
>>> we'll have the situation sorted out in the next few days.
>>>
>>> Tycho
>>>
>>>> Jake
>>>>
>>>> PS - Thanks for the html mail heads-up - I've been using google mail
>>>> services for this domain. I'll have to look into the config options, and
>>>> see if I can do the needful.
>>>
>>>>
>>>> On Tue, Mar 29, 2016 at 12:45 PM, Andrey Repin <anrdaemon at yandex.ru> wrote:
>>>>
>>>> > Greetings, jjs - mainphrame!
>>>> >
>>>> > >> On Mon, Mar 28, 2016 at 08:47:24PM -0700, jjs - mainphrame wrote:
>>>> >  >>> I've looked at ct migration between 2 ubuntu 16.04 hosts today, and
>>>> > had
>>>> >  >>> some interesting problems;  I find that migration of stopped
>>>> > containers
>>>> >  >>> works fairly reliably; but live migration, well, it transfers a lot of
>>>> >  >>> data, then exits with a failure message. I can then move the same
>>>> >  >>> container, stopped, with no problem.
>>>> >  >>>
>>>> >  >>> The error is the same every time, a failure of "mkdtemp" -
>>>> > >>
>>>> > >>  It looks like your host /tmp isn't writable by the uid map that the
>>>> > >>  container is being restored as?
>>>> >
>>>> >
>>>> > > Which is odd, since /tmp has 1777 perms on both hosts, so I don't see how
>>>> > > it could be a permissions problem. Surely the default apparmor profile is
>>>> > > not the cause? You did give me a new idea though, and I'll set up a test
>>>> > > with privileged containers for comparison. Is there a switch to enable
>>>> > verbose logging?
>>>> >
>>>> > I've ran into the same issue once. Stumbled upon it for nearly a month,
>>>> > falsely
>>>> > blaming LXC.
>>>> > Recreating a container's rootfs from scratch resolved the issue.
>>>> > I know not of what caused it to begin with, must've been some kind of
>>>> > glitch.
>>>> >
>>>> > P.S.
>>>> > It would be great if you can configure your mail client to not use HTML
>>>> > format
>>>> > for lists.
>>>> >
>>>> >
>>>> > --
>>>> > With best regards,
>>>> > Andrey Repin
>>>> > Tuesday, March 29, 2016 22:43:04
>>>> >
>>>> > Sorry for my terrible english...
>>>> > _______________________________________________
>>>> > lxc-users mailing list
>>>> > lxc-users at lists.linuxcontainers.org
>>>> > http://lists.linuxcontainers.org/listinfo/lxc-users
>>>> >
>>>
>>>> _______________________________________________
>>>> lxc-users mailing list
>>>> lxc-users at lists.linuxcontainers.org
>>>> http://lists.linuxcontainers.org/listinfo/lxc-users
>>>
>>> _______________________________________________
>>> lxc-users mailing list
>>> lxc-users at lists.linuxcontainers.org
>>> http://lists.linuxcontainers.org/listinfo/lxc-users