[lxc-devel] lxc_monitord - monitor exiting

[Dwight Engen]

On Mon, 6 May 2013 13:06:43 -0400
Dwight Engen <dwight.engen at oracle.com> wrote:

> Hi Çağlar,
> 
> Thanks for the test program, I can sort of recreate it here with that,
> although once I lxc-stop them all, lxc-monitord does go away. I put a
> debug in lxc_wait() to see that the client always close the fd to the
> monitord and they all were closed so I'm not sure why lxc-monitord
> isn't seeing the accepted fd coming back from epoll to close. Still
> investigating...

Okay, so I debugged this and the problem is basically down to lxc not
being thread aware. With your test program we get multiple threads in
lxcapi_start() simultaneously in the daemonize case. One of them forks
while another one has the client fd to the monitor open and thus the fd
gets duped by the fork and that is the client fd that holds lxc-monitord
open until the container shuts down.

Çağlar you could try out the following patch, it essentially serializes
container startup from a thread perspective. I haven't tested it
thoroughly, but it did fix the problem here. Right now lxc doesn't
support threaded use, so you may run into other things as well.
Depending on our stance on thread support in lxc, you may need to do
the serialization in the threaded app. I guess another
alternative is that initially we could just thread serialize at the API
(big lxc lock).

--

diff --git a/src/lxc/lxccontainer.c b/src/lxc/lxccontainer.c
index 04a9208..f464fdb 100644
--- a/src/lxc/lxccontainer.c
+++ b/src/lxc/lxccontainer.c
@@ -18,6 +18,7 @@
  */
 
 #define _GNU_SOURCE
+#include <pthread.h>
 #include <unistd.h>
 #include <sys/types.h>
 #include <sys/wait.h>
@@ -346,6 +347,7 @@ static bool wait_on_daemonized_start(struct lxc_container *c)
  * I can't decide if it'd be more convenient for callers if we accept '...',
  * or a null-terminated array (i.e. execl vs execv)
  */
+static pthread_mutex_t start_mutex = PTHREAD_MUTEX_INITIALIZER;
 static bool lxcapi_start(struct lxc_container *c, int useinit, char * const argv[])
 {
 	int ret;
@@ -391,13 +393,24 @@ static bool lxcapi_start(struct lxc_container *c, int useinit, char * const argv
 		if (!lxc_container_get(c))
 			return false;
 		lxc_monitord_spawn(c->config_path);
+
+		ret = pthread_mutex_lock(&start_mutex);
+		if (ret != 0) {
+			ERROR("pthread_mutex_lock returned:%d %s", ret, strerror(ret));
+			return false;
+		}
 		pid_t pid = fork();
 		if (pid < 0) {
 			lxc_container_put(c);
+			pthread_mutex_unlock(&start_mutex);
 			return false;
 		}
-		if (pid != 0)
-			return wait_on_daemonized_start(c);
+		if (pid != 0) {
+			ret = wait_on_daemonized_start(c);
+			pthread_mutex_unlock(&start_mutex);
+			return ret;
+		}
+		pthread_mutex_unlock(&start_mutex);
 		/* second fork to be reparented by init */
 		pid = fork();
 		if (pid < 0) {