[Lxc-users] [Dmtcp-forum] Running DMTCP inside a LXC container

Sun Oct 6 10:08:43 UTC 2013

Hi Kapil,

I have figured out that the file /proc/1/environ contains the line
"container=lxc".

That would appear to be a good way to figure out whether we are
running inside a container. The rights and ownership of this file are
read-only/root/root, though. There does not appear to be same line in
processes' environ files. We could maybe heuristically determine that
from the /proc/PID/cgroup of jailed processes, as the control group
field in this file seems to reliably start with "/lxc/". Example:

agravier at vm-0:~/dmtcp-2.0$ cat /proc/413/cgroup

9:hugetlb:/lxc/vm-524f2a6f88716f1d3800785a
8:perf_event:/lxc/vm-524f2a6f88716f1d3800785a
7:blkio:/lxc/vm-524f2a6f88716f1d3800785a
6:freezer:/lxc/vm-524f2a6f88716f1d3800785a
5:devices:/lxc/vm-524f2a6f88716f1d3800785a
4:memory:/lxc/vm-524f2a6f88716f1d3800785a
3:cpuacct:/lxc/vm-524f2a6f88716f1d3800785a
2:cpu:/lxc/vm-524f2a6f88716f1d3800785a
1:cpuset:/lxc/vm-524f2a6f88716f1d3800785a

To understand this file, here is an excerpt from proc(5):

       /proc/[pid]/cgroup (since Linux 2.6.24)
              This file describes control groups to which the process/task
              belongs.  For each cgroup hierarchy there is one entry
              containing colon-separated fields of the form:

                  5:cpuacct,cpu,cpuset:/daemons

              The colon-separated fields are, from left to right:

                  1. hierarchy ID number

                  2. set of subsystems bound to the hierarchy

                  3. control group in the hierarchy to which the process
                     belongs

              This file is present only if the CONFIG_CGROUPS kernel
              configuration option is enabled.

Regarding the path translation between control groups, I *think* that
there are two cases: files that are under the container name, and
files that are common to all containers. I think that because all I
see is either:

/var/lib/lxc/vm-524f2a6f88716f1d3800785a/overlay/<here comes the
filepath within the container>

and:

/var/lib/lxc/rootfs/<filepath within the container>

I'm pretty sure that vm-524f2a6f88716f1d3800785a is my current LXC
container's "name", it's the only one I can see.

So, path translation could be about removing /var/lib/lxc/<container
name>/overlay/ and /var/lib/lxc/rootfs/ prefixes.

It would be nice if someone from the LXC group could correct me if I'm
wrong or if there are better ways to do all these things.

Regards,
Alexandre

On Sat, Oct 5, 2013 at 10:51 PM, Kapil Arya <kapil at ccs.neu.edu> wrote:
> Hi Alexandre,
>
> Thanks for contacting us about this issue.
>
> We haven't tried DMTCP with LXC earlier so we can't be sure about the
> situation. However, I guess you are seeing these warnings because
> /proc/self/maps is listing those files with absolute paths. Can you verify
> this by looking at /proc/self/maps of a process running inside the
> container?
>
> In any case, was DMTCP able to generate checkpoint images? Also, what is a
> good way to figure out whether a process is running inside a container?
> Also, is there a way to translate between absolute and _fake_ paths?
>
> We might need to write a small DMTCP plugin for LXC to takes care of this.
>
> Kapil
>
>
> On Sat, Oct 5, 2013 at 4:31 PM, Alexandre Gravier <al.gravier at gmail.com>
> wrote:
>>
>> Hello,
>>
>> I'm trying to use DMTCP to save my user sessions on an online service
>> that offers LXC virtual machines. The service shutdowns the container
>> after 15 minutes of inactivity on my side, and I would like to be able
>> to save my tmux session and all related processes to disk before that
>> happens.
>>
>> So, I've compiled DMTCP 2.0, and 40 out of 45 tests pass*. I figured
>> that I may try to do something simple first, so I launch the
>> coordinator in a terminal, and `bin/dmtcp_launch cat` in another. The
>> coordinator notes that "NOTE at dmtcp_coordinator.cpp:1039 in
>> onConnect; REASON='worker connected'". It all seems fine, and the
>> clients list contains my cat.
>>
>> The problem comes when I actually make a checkpoint using 'c' in the
>> coordinator. The coordinator goes through what seems to be a
>> reasonable list of stages, from "starting checkpoint, suspending all
>> nodes" to "restarting all nodes", but in the terminal where the
>> dmtcp-monitored "cat" is running, there is an avalanche of errors that
>> all look like:
>>
>> [11795] mtcp_writeckpt.c:718 write_ckpt_to_file:
>>   ERROR: error statting
>>
>> /var/lib/lxc/vm-524f2a6f88716f1d3800785a/overlay/home/agravier/dmtcp-2.0/plugin/ipc/libdmtcp_ipc.so
>> : No such file or directory
>> [11795] mtcp_writeckpt.c:718 write_ckpt_to_file:
>>   ERROR: error statting
>> /var/lib/lxc/rootfs/lib/x86_64-linux-gnu/ld-2.17.so : No such file or
>> directory
>>
>> And there are dozens of those.
>>
>> Visibly, mtcp is looking for files using their absolute path *outside*
>> of the LXC container, which, of course, does not correspond to their
>> paths inside of it.
>>
>> I would like to understand how LXC chroots in a way that still allows
>> MTCP to get hold of the original filepaths. Why does MTCP get confused
>> when other applications don't? Or rather, why it doesn't get confused
>> by LXC :) ?
>>
>> Is there is anything that I might do about all that?
>>
>> Thanks and regards,
>> Alexandre
>>
>>
>> * Failing tests: shared-memory (rstr), pty2, bash (rstr), script, screen.
>>
>>
>> ------------------------------------------------------------------------------
>> October Webinars: Code for Performance
>> Free Intel webinars can help you accelerate application performance.
>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most
>> from
>> the latest Intel processors and coprocessors. See abstracts and register >
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=60134791&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Dmtcp-forum mailing list
>> Dmtcp-forum at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/dmtcp-forum
>
>