[lxc-devel] setns/fork(glibc) weird interaction, lxc-attach may hang sporadically

Serge Hallyn serge.hallyn at ubuntu.com
Tue Apr 23 15:31:09 UTC 2013


Quoting Christian Seiler (christian at iwakd.de):
> Hi there,
> 
> I've found a problem that comes from the interaction between setns()
> and glibc's implementation of fork() (at least on x86_64). It is rather
> complicated to describe, and forgive me if my mail is very long. I
> know of several ways how to address the problem but wanted to have a
> discussion first before actually changing something.
> 
> For starters, let's read some glibc source code, just because it's
> so much fun. ;-)

thanks for the detailed explanation :)  Fun indeed!

...
> But we don't know against what version of glibc lxc-attach is going to
> be build, so we need a better solution in the short term:
> 
>   - Call the fork syscall directly via syscall(__NR_fork)
>     Since lxc-attach is single-threaded anyway, the things glibc does

lxc-attach isn't but at some point we'll want attach in the API, and
that is not single-threaded.

>     aren't strictly needed, although the cache for getpid() is not
>     updated afterwards.
> 
>   - Use clone(), where apparently, if I read the glibc source correctly,
>     the wrapper doesn't use that logic. Makes life a bit more
>     complicated because we need an extra function and can't just check
>     the return value of a syscall. Advantages: glibc's clone() wrapper
>     does seem to update the pid cache according to the source [2] and
>     we could use CLONE_PARENT to keep only one process around (make the
>     middle one exit immediately) and make the synchronization logic a
>     bit simpler.
> 
>   - Do something else?
> 
> The only possible complication could arrive from the fact that the
> pthread_atfork() handlers are not called with either the fork() syscall
> or the clone() function. This is relevant in cases where the host uses
> e.g. nss-ldap. That being said, we do exec() away pretty fast (and any
> open fds or such nss implementations will have O_CLOEXEC set), but we
> will do that only after invoking getpwuid() to get the shell if no
> command was specified. If a socket or the such is still open then, nss
> could produce some weird effects since it's the same socket as that of
> the parent process. On the other hand, lxc-attach doesn't do nss
> lookups at all before fork()ing away, so this is probably a non-issue.
> 
> My suggestion would be to use clone() and to use that occasion to
> simplify the synchronization logic between all of those processes by
> using CLONE_PARENT. If I can get an agreement on this, I'll implement
> this, but I wanted to hear some thoughts in advance.
> 

Agreed.

Thanks!

-serge




More information about the lxc-devel mailing list