[lxc-users] Using valgrind with lxc

[Serge Hallyn]

Hi,

For the sake of testing I'd go ahead and just 'return 0' at the
top of lxc_check_inherited.

We can talk about adding an option to do this, i.e.
lxc.close_all_fds = -1 maybe.  It's a very rare case where
that should be done, though.

-serge

Quoting Vallevand, Mark K (Mark.Vallevand at UNISYS.com):
> Valgrind meet containers.
> Containers meet valgrind.
> 
> I've found what lxc doesn't like when running valgrind.
> 
> The lxc_start() checks to see if there are extra file descriptors open and won't call __lxc_start().
> vdr1: inherited fd 1024 on /home/vallevand/trunk_s4m/s4m-appliance/src/vdrd/vgVdrTest
> vdr1: inherited fd 1025 on /tmp/valgrind_proc_24989_cmdline_4fbfb9a5 (deleted)VdrTest
> vdr1: inherited fd 1026 on /dev/pts/1ind_proc_24989_cmdline_4fbfb9a5 (deleted)VdrTest
> vdr1: inherited fd 1027 on pipe:[768863]_proc_24989_cmdline_4fbfb9a5 (deleted)VdrTest
> vdr1: inherited fd 1028 on pipe:[768863]_proc_24989_cmdline_4fbfb9a5 (deleted)VdrTest
> 
> Vdr1 is the name of my container.  All those open files in the child process are related to valgrind.
> 
> If I call __lxc_start() rather than lxc_start(), I see this:
> vdr1: sync wake failure : Broken pipe
> vdr1: failed to spawn 'vdr1'
> And, just before that there is some complaining from valgrind:
> ==25086== Syscall param clone(child_tidptr) contains uninitialised byte(s)
> ==25086==    at 0x56622E1: clone (clone.S:84)
> ==25086==    by 0x4E3BD38: __lxc_start (in /usr/lib/lxc/liblxc.so.0.7.5)
> ==25086==    by 0x4014C9: vgVdrStartClone (vgVdrTest.c:88)
> ==25086==    by 0x400F0A: main (vgVdrTest.c:337)
> ==25086==
> ==1== Syscall param wait4(status) points to unaddressable byte(s)
> ==1==    at 0x53607C4: wait (wait.c:32)
> ==1==    by 0x4E3A400: ??? (in /usr/lib/lxc/liblxc.so.0.7.5)
> ==1==    by 0x566231C: clone (clone.S:112)
> ==1==  Address 0xffffffffffffffd4 is not stack'd, malloc'd or (recently) free'd
> ==1==
> ==1== Invalid write of size 4
> ==1==    at 0x4E3A4FF: ??? (in /usr/lib/lxc/liblxc.so.0.7.5)
> ==1==    by 0x566231C: clone (clone.S:112)
> ==1==  Address 0xffffffffffffffc0 is not stack'd, malloc'd or (recently) free'd
> ==1==
> ==1==
> ==1== Process terminating with default action of signal 11 (SIGSEGV)
> ==1==  Access not within mapped region at address 0xFFFFFFFFFFFFFFC0
> ==1==    at 0x4E3A4FF: ??? (in /usr/lib/lxc/liblxc.so.0.7.5)
> ==1==    by 0x566231C: clone (clone.S:112)
> 
> Our program is designed to close all open file descriptors in the child process before calling lxc_start().  That code can try to close all file descriptors to make sure something doesn't sneak through.  However, closing the file descriptors associated with valgrind does not work.  I get errno=0 Bad File Descriptor.  Valgrind really has them held open.  I am running as root in all these tests.
> 
> I've also reproduced the problem using the 'lxc-' programs.  If you do something like 'lxc-create -n XXX' and then something like 'valgrind lxc-start -n XXX -- ls' you'll see it.  Well, the flavor of the error with open file descriptors.
> 
> My hopes aren't high, but any ideas are very welcome.
> 
> Regards.
> Mark K Vallevand
> "If there are no dogs in Heaven, then when I die I want to go where they went."
> -Will Rogers
> 
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
> From: lxc-users [mailto:lxc-users-bounces at lists.linuxcontainers.org] On Behalf Of Vallevand, Mark K
> Sent: Thursday, September 25, 2014 09:19 AM
> To: lxc-users at lists.linuxcontainers.org
> Subject: [lxc-users] Using valgrind with lxc
> 
> In our program, we do a fork() and in the child process the lxc library is called to start a program in a container using lxc_start().
> 
> We don't care about valgrind in the child process.  You can disable valgrind messages from child processes, but you cannot detach valgrind unless you exec() a new binary on top.  However, valgrind and lxc do not play nicely, at least with the versions in Ubuntu 12.04 LTS.   I'm getting an error back from lxc_start().  I'm having trouble getting logs to see why its failing, so I don't know exactly what's failing, yet.
> 
> But, I'm looking for any ideas for getting valgrind to work with programs that use lxc_start().
> Any suggestions will be welcome.  And, thanks!
> 
> 
> Regards.
> Mark K Vallevand
> "If there are no dogs in Heaven, then when I die I want to go where they went."
> -Will Rogers
> 
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.

> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users