[lxc-devel] Working with glibc (PID/TID caches).

[Serge Hallyn]

Quoting Eric W. Biederman (ebiederm at xmission.com):
> 
> While talking about pain points there is a very large one.  In short
> pthreads break fork().
> 
> At least according to the open group after fork (when more than one
> pthread is active) and by extension clone(CLONE_NEW...) it is only safe
> to call async-signal-safe functions.  The rationale being that some
> other thread might have possibly held a lock during fork.
> 
> Which means it becomes extraordinarily difficult to safely set up
> and read /proc/mounts etc, etc.  Because malloc and huge portions of
> supporting code are now verbotten.
> 
> Several of the namespaces require a non-trivial amount of set-up work
> from within.  The mount namespace requires calling mount, umount, and
> pivot_root.  The network namespace requires making netlink calls over
> sockets.
> 
> The work arounds involving multi-process synchronization and setns after
> fork are quite ugly.
> 
> It would be nice if at least malloc and C++ new were safe (and
> documented as safe) after fork in a pthread environment.  That would go
> a long ways to allowing running interesting set up code without having
> to jump through hoops.

I'm pretty sure malloc is in fact thread-safe.  For some time I was
paranoid about file table operations (like fopen) and mutexed them
all, but we eventually realized that those are also thread-safe.

The only hoop we run through in lxc that I can think of is that we
use pthread_atfork() to drop any mutexes which some thread may have
been holding at fork time.  With that, we have Çaglar doing some
rather impressive concurrency runs using the lxc-go api, and afaik
no issues currently (except in libnih/libdbus, which we have to
paper over by mutexing it, sadly).

-serge