[lxc-users] Using valgrind with lxc

[Vallevand, Mark K]

Valgrind meet containers.
Containers meet valgrind.

I've found what lxc doesn't like when running valgrind.

The lxc_start() checks to see if there are extra file descriptors open and won't call __lxc_start().
vdr1: inherited fd 1024 on /home/vallevand/trunk_s4m/s4m-appliance/src/vdrd/vgVdrTest
vdr1: inherited fd 1025 on /tmp/valgrind_proc_24989_cmdline_4fbfb9a5 (deleted)VdrTest
vdr1: inherited fd 1026 on /dev/pts/1ind_proc_24989_cmdline_4fbfb9a5 (deleted)VdrTest
vdr1: inherited fd 1027 on pipe:[768863]_proc_24989_cmdline_4fbfb9a5 (deleted)VdrTest
vdr1: inherited fd 1028 on pipe:[768863]_proc_24989_cmdline_4fbfb9a5 (deleted)VdrTest

Vdr1 is the name of my container.  All those open files in the child process are related to valgrind.

If I call __lxc_start() rather than lxc_start(), I see this:
vdr1: sync wake failure : Broken pipe
vdr1: failed to spawn 'vdr1'
And, just before that there is some complaining from valgrind:
==25086== Syscall param clone(child_tidptr) contains uninitialised byte(s)
==25086==    at 0x56622E1: clone (clone.S:84)
==25086==    by 0x4E3BD38: __lxc_start (in /usr/lib/lxc/liblxc.so.0.7.5)
==25086==    by 0x4014C9: vgVdrStartClone (vgVdrTest.c:88)
==25086==    by 0x400F0A: main (vgVdrTest.c:337)
==25086==
==1== Syscall param wait4(status) points to unaddressable byte(s)
==1==    at 0x53607C4: wait (wait.c:32)
==1==    by 0x4E3A400: ??? (in /usr/lib/lxc/liblxc.so.0.7.5)
==1==    by 0x566231C: clone (clone.S:112)
==1==  Address 0xffffffffffffffd4 is not stack'd, malloc'd or (recently) free'd
==1==
==1== Invalid write of size 4
==1==    at 0x4E3A4FF: ??? (in /usr/lib/lxc/liblxc.so.0.7.5)
==1==    by 0x566231C: clone (clone.S:112)
==1==  Address 0xffffffffffffffc0 is not stack'd, malloc'd or (recently) free'd
==1==
==1==
==1== Process terminating with default action of signal 11 (SIGSEGV)
==1==  Access not within mapped region at address 0xFFFFFFFFFFFFFFC0
==1==    at 0x4E3A4FF: ??? (in /usr/lib/lxc/liblxc.so.0.7.5)
==1==    by 0x566231C: clone (clone.S:112)

Our program is designed to close all open file descriptors in the child process before calling lxc_start().  That code can try to close all file descriptors to make sure something doesn't sneak through.  However, closing the file descriptors associated with valgrind does not work.  I get errno=0 Bad File Descriptor.  Valgrind really has them held open.  I am running as root in all these tests.

I've also reproduced the problem using the 'lxc-' programs.  If you do something like 'lxc-create -n XXX' and then something like 'valgrind lxc-start -n XXX -- ls' you'll see it.  Well, the flavor of the error with open file descriptors.

My hopes aren't high, but any ideas are very welcome.

Regards.
Mark K Vallevand
"If there are no dogs in Heaven, then when I die I want to go where they went."
-Will Rogers

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
From: lxc-users [mailto:lxc-users-bounces at lists.linuxcontainers.org] On Behalf Of Vallevand, Mark K
Sent: Thursday, September 25, 2014 09:19 AM
To: lxc-users at lists.linuxcontainers.org
Subject: [lxc-users] Using valgrind with lxc

In our program, we do a fork() and in the child process the lxc library is called to start a program in a container using lxc_start().

We don't care about valgrind in the child process.  You can disable valgrind messages from child processes, but you cannot detach valgrind unless you exec() a new binary on top.  However, valgrind and lxc do not play nicely, at least with the versions in Ubuntu 12.04 LTS.   I'm getting an error back from lxc_start().  I'm having trouble getting logs to see why its failing, so I don't know exactly what's failing, yet.

But, I'm looking for any ideas for getting valgrind to work with programs that use lxc_start().
Any suggestions will be welcome.  And, thanks!


Regards.
Mark K Vallevand
"If there are no dogs in Heaven, then when I die I want to go where they went."
-Will Rogers

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20140929/9f1a3509/attachment.html>