[lxc-devel] Working with glibc (PID/TID caches).

[Serge Hallyn]

Quoting Eric W. Biederman (ebiederm at xmission.com):
> Serge Hallyn <serge.hallyn at ubuntu.com> writes:
> 
> > Quoting Eric W. Biederman (ebiederm at xmission.com):
> >> It would be nice if at least malloc and C++ new were safe (and
> >> documented as safe) after fork in a pthread environment.  That would go
> >> a long ways to allowing running interesting set up code without having
> >> to jump through hoops.
> >
> > I'm pretty sure malloc is in fact thread-safe.  For some time I was
> > paranoid about file table operations (like fopen) and mutexed them
> > all, but we eventually realized that those are also thread-safe.
> 
> The dangerous part I was referring to is what happens when someone
> holds a mutex and calls fork(3) or clone(3).
> 
> I had missed the existence of pthread_atfork and malloc in glibc
> very clearly has atfork handlers so malloc should be safe after
> fork(3).
> 
> However I justed tested it and clone(3) without CLONE_VM does not
> call the atfork handlers which is a nasty problem to work with.

Fascinating!

> ***** Can libc please be modified to call the atfork handlers from
>       clone(3) without CLONE_VM.
> 
>       Pretty Please with sugar on top.
> 
> > The only hoop we run through in lxc that I can think of is that we
> > use pthread_atfork() to drop any mutexes which some thread may have
> > been holding at fork time.  With that, we have Çaglar doing some
> > rather impressive concurrency runs using the lxc-go api, and afaik
> > no issues currently (except in libnih/libdbus, which we have to
> > paper over by mutexing it, sadly).
> 
> I had failed to notice the pthread_atfork handlers and those definitely
> solve (or at least make it possible to solve) the nasties that pthreads
> introduce into fork.
> 
> If we can get those same fixes for clone(3) that would be a real help.

Yes, the lack of that is very non-obvious and certainly could've bitten
us in lxc.

> Serge are you using unshare in liblxc right now?  Using clone(3) where
> it doesn't call the pthread_atfork handlers is starting to scare me a
> little.

In lxc, we almost always fork followed by unshare.  We only clone(2)
when we are starting the actual container (or the reboot-support test
before that), and we insist on being single-threaded before that (for
console reasons).  This is of course precisely because clone(2) is so
much more trouble to use than fork(), so look, it's a feature :)

-serge