[lxc-devel] [PATCH] tests: Introduce lxc-test-concurrent for testing basic actions concurrently

[Serge Hallyn]

Quoting Dwight Engen (dwight.engen at oracle.com):
> On Fri, 13 Sep 2013 17:29:53 -0400
> Dwight Engen <dwight.engen at oracle.com> wrote:
> 
> > On Fri, 13 Sep 2013 12:09:55 -0400
> > S.Çağlar Onur <caglar at 10ur.org> wrote:
> > 
> > > Hi Dwight,
> > > 
> > > Yes, I only observed a hang so far but not this assertion (in fact I
> > > don't remember ever seeing that). What I'm seeing is this;
> > 
> > Okay, something funny is going on, but I don't know what yet. That
> > assertion is coming from liblxc.so->libgnutls->libgcrypt and seems to
> > be complaining that we're unlocking something that is already
> > unlocked. So I compiled lxc without GNUTLS support (by commenting out
> > the check for it in configure.ac) and now I get past that and get
> > hangs similar to yours.
> > 
> > Interestingly, I modified your program to just do the create and
> > destroy and not the start nor stop and I still get the hangs during
> > the creation part.
> 
> Sorry to reply to myself: So now I modified your program to do the
> create single threaded and everything else threaded (see attached
> patch) and that doesn't hang for me.
> 
> It looked like to me that the hung threads were stuck on process_lock()
> so I added a backtrace there to debug it a bit more and it looks like
> the hang is in the flow:
> 
>  lxcapi_create()
>    c->save_config()
>      container_disk_lock()
>        lxclock()
>          process_lock()
> 
> Not sure why yet though. Serge, do you think this could have to do with
> the fork()ing from a thread in that flow?

Well, we've got save_config() doing an fopen without the process_lock()
for one thing which is bad.  That could explain it in a few ways - we
could have corruption due to both tasks changing fdtable at the same
time, or perhaps because open() is a cancelation point...

Does doing a process_lock() around the fopen and fclose in
lxcapi_save_config() fix it?

As for the forks, everything I've seen suggests those should be safe.
Unless we fork with some mutexes held, in which case the mutexes will
be copied held into the new task, and the new task might deadlock
against nothing.  But that shouldn't affect the parent thread which is
what you're seeing above.

-serge