[lxc-devel] Working with glibc (PID/TID caches).

[Eric W. Biederman]

"Carlos O'Donell" <carlos at redhat.com> writes:

> On 08/06/2014 05:46 PM, Serge Hallyn wrote:
>> Quoting Eric W. Biederman (ebiederm at xmission.com):
>>> Serge Hallyn <serge.hallyn at ubuntu.com> writes:
>
> Reviving this super-old discussion because it went in a direction
> that I didn't expect.
>
>>>> Quoting Eric W. Biederman (ebiederm at xmission.com):
>>>>> It would be nice if at least malloc and C++ new were safe (and
>>>>> documented as safe) after fork in a pthread environment.  That would go
>>>>> a long ways to allowing running interesting set up code without having
>>>>> to jump through hoops.
>>>>
>>>> I'm pretty sure malloc is in fact thread-safe.  For some time I was
>>>> paranoid about file table operations (like fopen) and mutexed them
>>>> all, but we eventually realized that those are also thread-safe.
>>>
>>> The dangerous part I was referring to is what happens when someone
>>> holds a mutex and calls fork(3) or clone(3).
>>>
>>> I had missed the existence of pthread_atfork and malloc in glibc
>>> very clearly has atfork handlers so malloc should be safe after
>>> fork(3).
>
> Correct.
>
>>> However I justed tested it and clone(3) without CLONE_VM does not
>>> call the atfork handlers which is a nasty problem to work with.
>> 
>> Fascinating!
>
> The semantics of clone are already well established and we aren't
> going to change them.
>
> If you remain within the boundaries of pthreads then pthread_atfork
> does most of what you want.
>
> Using clone pushes you outside of that boundary, which is why the
> solution is so complicated.
>
>>> ***** Can libc please be modified to call the atfork handlers from
>>>       clone(3) without CLONE_VM.
>>>
>>>       Pretty Please with sugar on top.
>
> No, but we can create another interface with a different name that
> does something similar, but we'll have to think about it more generically
> and see what can be designed.
>
>>>> The only hoop we run through in lxc that I can think of is that we
>>>> use pthread_atfork() to drop any mutexes which some thread may have
>>>> been holding at fork time.  With that, we have Çaglar doing some
>>>> rather impressive concurrency runs using the lxc-go api, and afaik
>>>> no issues currently (except in libnih/libdbus, which we have to
>>>> paper over by mutexing it, sadly).
>>>
>>> I had failed to notice the pthread_atfork handlers and those definitely
>>> solve (or at least make it possible to solve) the nasties that pthreads
>>> introduce into fork.
>>>
>>> If we can get those same fixes for clone(3) that would be a real help.
>> 
>> Yes, the lack of that is very non-obvious and certainly could've bitten
>> us in lxc.
>> 
>>> Serge are you using unshare in liblxc right now?  Using clone(3) where
>>> it doesn't call the pthread_atfork handlers is starting to scare me a
>>> little.
>> 
>> In lxc, we almost always fork followed by unshare.  We only clone(2)
>> when we are starting the actual container (or the reboot-support test
>> before that), and we insist on being single-threaded before that (for
>> console reasons).  This is of course precisely because clone(2) is so
>> much more trouble to use than fork(), so look, it's a feature :)
>
> Worse is that clone impacts thread-local storage in C and C++, so you
> may not even have access to TLS maintained by the core runtime after
> the clone. Calling clone goes behind the back of the threading library.
>
> Are there any specific reasons you use clone instead of fork?
>
> Such reasons would help inform a new API design.

So we could specify flags that create namespaces aka: CLONE_NEWNS,
CLONE_NEWUTS, CLONE_NEWIPC, CLONE_NEWUSER, CLONE_NEWPID, CLONE_NEWNET

We are talking about container creation after all.

At the same time since we are creating a new virtual address space it
would be handy if we didn't need to do all of the work to set up a new
stack as the existing stack is perfectly functional.

Eric