[lxc-users] lxc progress and a few questions

Tue Apr 26 00:02:59 UTC 2016

Thanks Tycho,

I do mean to look into zfs, but I've got a lot on my plate currently,
so I can wait - in the meantime, I've got lots of other things to work
on.

Regards,

Jake

On Mon, Apr 25, 2016 at 4:46 PM, Tycho Andersen
<tycho.andersen at canonical.com> wrote:
> Hi,
>
> On Mon, Apr 18, 2016 at 12:47:31PM -0700, jjs - mainphrame wrote:
>> Greetings -
>>
>> As the 16.04 release date draws near, there have been ongoing daily
>> updates but on my (ext4 based) xenial systems, migration of any lxd
>> containers, stopped or running, hangs indefinitely.
>>
>> There is an strace output of an unsucessful attempt at
>> http://pastebin.com/raw/ge9zNUaX
>>
>> Short of the one suggestion to format /var/lib as zfs and see what
>> happens, is there anything else I can try?
>
> I've just filed https://github.com/lxc/lxd/issues/1944 and hope to
> look at it this week; I've been quite busy with other issues here for
> release.
>
> I'm not aware of anything you can do right now other than switching to
> ZFS (which is really worth doing, ZFS is sweet).
>
> Tycho
>
>> Regards,
>>
>> Jake
>>
>> On Mon, Apr 11, 2016 at 8:25 AM, Tycho Andersen
>> <tycho.andersen at canonical.com> wrote:
>> > hi,
>> >
>> > On Fri, Apr 08, 2016 at 04:40:43PM -0700, jjs - mainphrame wrote:
>> >> Ah, never mind - it doesn't appear to be solely a criu issue - even
>> >> migration of stopped containers hangs forever now.
>> >
>> > Yeah, I think there is some behavior either with rsync or our
>> > websocket libraries that's causing this hang. I've looked into it but
>> > haven't had any luck .Try the zfs backend, that should work better.
>> >
>> > Also, there is a bug with xenial kernels -16+ that prevent CRIU from
>> > working (you'll get an EBUSY trying to mount sysfs), so if you have
>> > -15 or below that should work.
>> >
>> > Sorry for the delay, been dealing with a family emergency.
>> >
>> > Tycho
>> >
>> >> Jake
>> >>
>> >> On Fri, Apr 8, 2016 at 4:23 PM, jjs - mainphrame <jjs at mainphrame.com> wrote:
>> >> > Ubuntu 16.04, up to date -
>> >> >
>> >> > After today's updates, including a kernel upgrade to 4.4.0-18, I tried
>> >> > live migration again:
>> >> >
>> >> > root at raskolnikov:~# lxc move third lxd2:
>> >> >
>> >> > One hour later:
>> >> >
>> >> > root at raskolnikov:~# lxc move third lxd2:
>> >> >
>> >> > Still stuck, and the migration file in /var/log/lxd/third has not been created.
>> >> >
>> >> > Tycho said on Mar 30 that the situation should be sorted soon, but
>> >> > mentioned the git repo:
>> >> > https://github.com/tych0/criu/tree/cgroup-root-mount
>> >> >
>> >> > Should live migration work with criu from git?
>> >> >
>> >> > Feel free to advise me on what information I can supply, not only for
>> >> > the ct migration issues, but also for the new dhcp issue
>> >> >
>> >> > Regards,
>> >> >
>> >> > Jake
>> >> >
>> >> >
>> >> > On Thu, Apr 7, 2016 at 11:01 PM, jjs - mainphrame <jjs at mainphrame.com> wrote:
>> >> >> (Bump) -
>> >> >>
>> >> >> Any thoughts on what to try for the CT migration and dhcp issues?
>> >> >> Running up to date ubuntu 16.04 beta -
>> >> >>
>> >> >> Regards,
>> >> >>
>> >> >> Jake
>> >> >>
>> >> >> On Wed, Apr 6, 2016 at 3:18 PM, jjs - mainphrame <jjs at mainphrame.com> wrote:
>> >> >>> Greetings -
>> >> >>>
>> >> >>> I'be not yet been able to reproduce that one shining moment from Mar
>> >> >>> 29 when live migration of privileged containers was working, under
>> >> >>> kernel 4.4.0-15
>> >> >>>
>> >> >>> To recap. live container migration broke with 4.4.0-16, and is still
>> >> >>> broken in 4.4.0-17 - but  now, instead of producing an error message,
>> >> >>> an attempt to live migrate a container merely hangs forever. Is that
>> >> >>> expected, or should I be seeing something more? BTW - the migration
>> >> >>> dump log for that container hasn't been touched for a week. I'll be
>> >> >>> glad to supply more info if this is not a known issue.
>> >> >>>
>> >> >>> Recent updates seem to have created a new problem. the CTs which
>> >> >>> configure their own network settings work (aside from migration) but
>> >> >>> none of the CTs which depend on dhcp are getting IPs. BTW I'm using a
>> >> >>> bridge connected to my local network and dhcp, not the default lxc
>> >> >>> dhcp server. I see the packets on the host bridge, but they don't
>> >> >>> reach the dhcp server. I'd be curious to know if there have been any
>> >> >>> dhcp issues since recent updates. If not, I'll need to troubleshoot
>> >> >>> other causes, but it's odd that dhcp simply stops working for all CTs
>> >> >>> on both lxd hosts after updates.
>> >> >>>
>> >> >>> Jake
>> >> >>>
>> >> >>>
>> >> >>> On Wed, Mar 30, 2016 at 6:27 AM, Tycho Andersen
>> >> >>> <tycho.andersen at canonical.com> wrote:
>> >> >>>> On Tue, Mar 29, 2016 at 11:17:26PM -0700, jjs - mainphrame wrote:
>> >> >>>>> Well, I've found some interesting things here today. I created a couple of
>> >> >>>>> privileged xenial containers, and sure enough, I was able to live migrate
>> >> >>>>> them back and forth between the 2 lxd hosts.
>> >> >>>>>
>> >> >>>>> So far, so good.
>> >> >>>>>
>> >> >>>>> Then I did an apt upgrade - among the changes was a kernel change from
>> >> >>>>>  4.4.0-15 to 4.4.0-16 - and live migration stopped working.
>> >> >>>>>
>> >> >>>>> Here are the failure messages that resulted from attempting the very same
>> >> >>>>> live migrations that worked before the upgrade and reboot into 4.4.0-16:
>> >> >>>>>
>> >> >>>>> root at raskolnikov:~# lxc move akira lxd2:
>> >> >>>>> error: Error transferring container data: checkpoint failed:
>> >> >>>>> (00.092234) Error (mount.c:740): mnt: 83:./sys/fs/cgroup/devices doesn't
>> >> >>>>> have a proper root mount
>> >> >>>>> (00.098187) Error (cr-dump.c:1600): Dumping FAILED.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> root at ronnie:~# lxc move third lxd:
>> >> >>>>> error: Error transferring container data: checkpoint failed:
>> >> >>>>> (00.076107) Error (mount.c:740): mnt: 326:./sys/fs/cgroup/perf_event
>> >> >>>>> doesn't have a proper root mount
>> >> >>>>> (00.080388) Error (cr-dump.c:1600): Dumping FAILED.
>> >> >>>>
>> >> >>>> Yep, this is a known issue with -16. We need both a kernel patch and a
>> >> >>>> patch to CRIU before it will start working again. I have a branch at:
>> >> >>>>
>> >> >>>> https://github.com/tych0/criu/tree/cgroup-root-mount
>> >> >>>>
>> >> >>>> which should work if you want to keep playing with it, but hopefully
>> >> >>>> we'll have the situation sorted out in the next few days.
>> >> >>>>
>> >> >>>> Tycho
>> >> >>>>
>> >> >>>>> Jake
>> >> >>>>>
>> >> >>>>> PS - Thanks for the html mail heads-up - I've been using google mail
>> >> >>>>> services for this domain. I'll have to look into the config options, and
>> >> >>>>> see if I can do the needful.
>> >> >>>>
>> >> >>>>>
>> >> >>>>> On Tue, Mar 29, 2016 at 12:45 PM, Andrey Repin <anrdaemon at yandex.ru> wrote:
>> >> >>>>>
>> >> >>>>> > Greetings, jjs - mainphrame!
>> >> >>>>> >
>> >> >>>>> > >> On Mon, Mar 28, 2016 at 08:47:24PM -0700, jjs - mainphrame wrote:
>> >> >>>>> >  >>> I've looked at ct migration between 2 ubuntu 16.04 hosts today, and
>> >> >>>>> > had
>> >> >>>>> >  >>> some interesting problems;  I find that migration of stopped
>> >> >>>>> > containers
>> >> >>>>> >  >>> works fairly reliably; but live migration, well, it transfers a lot of
>> >> >>>>> >  >>> data, then exits with a failure message. I can then move the same
>> >> >>>>> >  >>> container, stopped, with no problem.
>> >> >>>>> >  >>>
>> >> >>>>> >  >>> The error is the same every time, a failure of "mkdtemp" -
>> >> >>>>> > >>
>> >> >>>>> > >>  It looks like your host /tmp isn't writable by the uid map that the
>> >> >>>>> > >>  container is being restored as?
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > > Which is odd, since /tmp has 1777 perms on both hosts, so I don't see how
>> >> >>>>> > > it could be a permissions problem. Surely the default apparmor profile is
>> >> >>>>> > > not the cause? You did give me a new idea though, and I'll set up a test
>> >> >>>>> > > with privileged containers for comparison. Is there a switch to enable
>> >> >>>>> > verbose logging?
>> >> >>>>> >
>> >> >>>>> > I've ran into the same issue once. Stumbled upon it for nearly a month,
>> >> >>>>> > falsely
>> >> >>>>> > blaming LXC.
>> >> >>>>> > Recreating a container's rootfs from scratch resolved the issue.
>> >> >>>>> > I know not of what caused it to begin with, must've been some kind of
>> >> >>>>> > glitch.
>> >> >>>>> >
>> >> >>>>> > P.S.
>> >> >>>>> > It would be great if you can configure your mail client to not use HTML
>> >> >>>>> > format
>> >> >>>>> > for lists.
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > --
>> >> >>>>> > With best regards,
>> >> >>>>> > Andrey Repin
>> >> >>>>> > Tuesday, March 29, 2016 22:43:04
>> >> >>>>> >
>> >> >>>>> > Sorry for my terrible english...
>> >> >>>>> > _______________________________________________
>> >> >>>>> > lxc-users mailing list
>> >> >>>>> > lxc-users at lists.linuxcontainers.org
>> >> >>>>> > http://lists.linuxcontainers.org/listinfo/lxc-users
>> >> >>>>> >
>> >> >>>>
>> >> >>>>> _______________________________________________
>> >> >>>>> lxc-users mailing list
>> >> >>>>> lxc-users at lists.linuxcontainers.org
>> >> >>>>> http://lists.linuxcontainers.org/listinfo/lxc-users
>> >> >>>>
>> >> >>>> _______________________________________________
>> >> >>>> lxc-users mailing list
>> >> >>>> lxc-users at lists.linuxcontainers.org
>> >> >>>> http://lists.linuxcontainers.org/listinfo/lxc-users
>> >> _______________________________________________
>> >> lxc-users mailing list
>> >> lxc-users at lists.linuxcontainers.org
>> >> http://lists.linuxcontainers.org/listinfo/lxc-users
>> > _______________________________________________
>> > lxc-users mailing list
>> > lxc-users at lists.linuxcontainers.org
>> > http://lists.linuxcontainers.org/listinfo/lxc-users
>> _______________________________________________
>> lxc-users mailing list
>> lxc-users at lists.linuxcontainers.org
>> http://lists.linuxcontainers.org/listinfo/lxc-users
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users