[lxc-devel] [CRIU] LXC live migrate

[Pavel Emelyanov]

On 11/27/2013 06:19 AM, Qiang Huang wrote:
> On 2013/11/27 0:19, Marian Marinov wrote:
>>
>> On my test setup it works for processes like apache, dovecot and mysql.
>>
>> However it does not work with containers:
>>
>> root at s321:~# criu dump -D deb1 -t 19332 --file-locks
>> (00.004962) Error (namespaces.c:155): Can't dump nested pid namespace for 28352
>> (00.004985) Error (namespaces.c:321): Can't make pidns id
>> (00.005327) Error (cr-dump.c:1811): Dumping FAILED.
>> root at s321:~#
>> When I try to dump the init process(which I believe I should not do), here is what I see:
>>    http://pastebin.com/DFC0ADpp
>>
>> (00.291294) Error (tty.c:222): tty: Unexpected format on path /dev/tty1
>> (00.291315) Error (cr-dump.c:1491): Dump files (pid: 29702) failed with -1
>> (00.291892) Error (cr-dump.c:1811): Dumping FAILED.
>>
>> This is my setup:
>> 19332 ?        Ss     0:00 lxc-start -n deb1 -d
>> 28352 ?        Ss     0:00  \_ init [3]
>> 28393 ?        Ss     0:00      \_ /usr/sbin/apache2 -k start
>> 28419 ?        S      0:00      |   \_ /usr/sbin/apache2 -k start
>> 28422 ?        Sl     0:00      |   \_ /usr/sbin/apache2 -k start
>> 28423 ?        Sl     0:00      |   \_ /usr/sbin/apache2 -k start
>> 28489 ?        S      0:00      \_ /bin/sh /usr/bin/mysqld_safe
>> 28620 ?        Sl     0:00      |   \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql 
>> --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port
>> 28621 ?        S      0:00      |   \_ logger -t mysqld -p daemon.error
>> 28598 ?        Ss     0:00      \_ /usr/sbin/sshd
>> 29702 pts/0    Ss+    0:00      \_ /sbin/getty 38400 tty1 linux
>>
>> I rebooted the container without getty on tty1 and then I got this:
>>
>> (00.260757) Error (mount.c:255): 86:/dev/tty4 doesn't have a proper root mount

This is the reason. That's container's console which is a bind mounted tty from
the host. And since this is an external connection, CRIU doesn't dump one.

There are two ways to resolve this. The first is disable container's console. It's
fast, but ugly. The second way is supporting one, but it would require criu hacking.
We should detect, that this is an external tty, decide, that we're OK to disconnect
it after dump and on restore -- connect it back.

>> (00.261007) Error (namespaces.c:445): Namespaces dumping finished with error 65280
>> (00.261454) Error (cr-dump.c:1811): Dumping FAILED.
>>
>> This ithe relevant container config
>> ## Device config
>> lxc.cgroup.devices.deny = a
>> # /dev/null and zero
>> lxc.cgroup.devices.allow = c 1:3 rwm
>> lxc.cgroup.devices.allow = c 1:5 rwm
>> # consoles
>> lxc.cgroup.devices.allow = c 5:1 rwm
>> lxc.cgroup.devices.allow = c 5:0 rwm
>> lxc.cgroup.devices.allow = c 4:0 rwm
>> lxc.cgroup.devices.allow = c 4:1 rwm
>> # /dev/{,u}random
>> lxc.cgroup.devices.allow = c 1:9 rwm
>> lxc.cgroup.devices.allow = c 1:8 rwm
>> lxc.cgroup.devices.allow = c 136:* rwm
>> lxc.cgroup.devices.allow = c 5:2 rwm
>> # rtc
>> lxc.cgroup.devices.allow = c 254:0 rm
>>
>> # mounts point
>> lxc.mount.entry = devpts dev/pts devpts gid=5,mode=620 0 0
>> lxc.mount.auto = proc:mixed sys:ro
>>
>>
>> Am I doing something wrong?
> 
> According to the criu TODO list: http://criu.org/Todo
> cgroups in container is not supported yet, so I doubt it would work for
> normal containers.

AFAIK cgroups are used _inside_ containers only with recent guest templates.
In OpenVZ we use more old ones (and more stable) so haven't meet this yet.
And yes, cgroups are in plans for the nearest future :)

> I'm interested in this too, so let's cc CRIU list and find out what is wrong :)
> 
>>
>> Marian
>>

Thanks,
Pavel