[lxc-devel] [CRIU] LXC live migrate

[Pavel Emelyanov]

On 11/27/2013 01:18 PM, Marian Marinov wrote:
>>>> I rebooted the container without getty on tty1 and then I got this:
>>>>
>>>> (00.260757) Error (mount.c:255): 86:/dev/tty4 doesn't have a proper root mount
>>
>> This is the reason. That's container's console which is a bind mounted tty from
>> the host. And since this is an external connection, CRIU doesn't dump one.
>>
>> There are two ways to resolve this. The first is disable container's console. It's
>> fast, but ugly. The second way is supporting one, but it would require criu hacking.
>> We should detect, that this is an external tty, decide, that we're OK to disconnect
>> it after dump and on restore -- connect it back.
> 
> The ugly fix does not work either. Because even if you comment the lxc.tty option criu is complaining:
> 
> (00.243390) Error (mount.c:255): 82:/dev/console doesn't have a proper root mount

But this also means, that the /dev/console is bind mounted from the host,
so external console didn't disappear.

> (00.243626) Error (namespaces.c:445): Namespaces dumping finished with error 65280
> (00.244029) Error (cr-dump.c:1811): Dumping FAILED.
> 
>>
>>>> (00.261007) Error (namespaces.c:445): Namespaces dumping finished with error 65280
>>>> (00.261454) Error (cr-dump.c:1811): Dumping FAILED.
>>>>
>>>> Am I doing something wrong?
>>>
>>> According to the criu TODO list: http://criu.org/Todo
>>> cgroups in container is not supported yet, so I doubt it would work for
>>> normal containers.
> 
> Pavel, can you give me some pointers for this?

Well, here's what I have in mind. The vision below may be wrong, so we can discuss it.

The cgroups support can be two-fold. First, is how to dump and restore those cgroups
inside which you started the container. It looks better if these cgroups are handled
by the tool that launches the container, i.e. the lxc one. IOW dump and restore should
look like this

1. freeze the tasks
2. find where the cgroups of the container are,
   read the desired information from them and save,
   then dump the container
3. unfreeze the tasks

Although criu doesn't provide separate freeze and dump commands, you can provide the
"post-dump" action script that will be invoked before tasks are unfrozen and where
you can dump the cgroups state (http://criu.org/Action_scripts).

Restore would look like this

1. Create container
2. Configure cgroups
3. Restore tasks into it

It can be done literally like this, but you can as well make use of the "setup-namespaces"
action script of criu.

The second fold of the problem is with cgroups, that are created inside the container.
In that case we should start form mount.c file and add support for dumping the "cgroup"
filesystem. The fstypes[] array is about this. The ->dump and ->restore callbacks for
cgroup fs should read the desired information about cgroups and restore it back.

With this I can now see two issues.

First, the image for external cgroups should be the same as the one for internal ones.
And since criu uses protocol buffers (a.k.a. ptorobuf) for images, lxc will have to
somehow deal with it.

And the second one is -- filesystems are mounted before forking tasks, but since 
restoring cgroups implies putting tasks into them, we'll have to modify the criu
code more to properly put tasks into cgroups after they are forked.

> I would be interested in helping out with this part.

Oh, that's great! If you need any technical details about criu -- don't hesitate to ask
me or on the list (but it would be easier if you subscribe on it, not-on-list users'
messages require approval).

Thanks,
Pavel