<div dir="ltr">Hi Dwight,<div><br></div><div style>Couple of quick comments, I needed following to make it compile under ubuntu 12.10 otherwise gcc whines about ignoring return values of those calls.</div><div style><br></div>
<div style><div>diff --git a/src/lxc/lxc_monitord.c b/src/lxc/lxc_monitord.c</div><div>index 1d20d67..9d83673 100644</div><div>--- a/src/lxc/lxc_monitord.c</div><div>+++ b/src/lxc/lxc_monitord.c</div><div>@@ -309,7 +309,11 @@ int main(int argc, char *argv[])</div>
<div> }</div><div> </div><div> /* sync with parent */</div><div>- write(pipefd, "S", 1);</div><div>+ if (write(pipefd, "S", 1) < 0) {</div><div>+ ERROR("write failed");</div>
<div>+ goto out;</div><div>+ }</div><div>+ </div><div> close(pipefd);</div><div> </div><div> if (lxc_monitord_mainloop_add(&mon)) {</div><div>diff --git a/src/lxc/monitor.c b/src/lxc/monitor.c</div>
<div>index 0438fbc..803f98f 100644</div><div>--- a/src/lxc/monitor.c</div><div>+++ b/src/lxc/monitor.c</div><div>@@ -158,6 +158,7 @@ int lxc_monitord_spawn(const char *lxcpath)</div><div> pid_t pid1,pid2;</div><div>
int pipefd[2];</div><div> char pipefd_str[10];</div><div>+ FILE *fd;</div><div> </div><div> char * const args[] = {</div><div> "/usr/bin/lxc-monitord",</div><div>@@ -188,7 +189,10 @@ int lxc_monitord_spawn(const char *lxcpath)</div>
<div> char c;</div><div> /* wait for daemon to create socket */</div><div> close(pipefd[1]);</div><div>- read(pipefd[0], &c, 1);</div><div>+ if (read(pipefd[0], &c, 1) < 0) {</div>
<div>+ SYSERROR("read failed");</div><div>+ exit(EXIT_FAILURE);</div><div>+ }</div><div> close(pipefd[0]);</div><div> exit(EXIT_SUCCESS);</div>
<div> }</div><div>@@ -198,9 +202,21 @@ int lxc_monitord_spawn(const char *lxcpath)</div><div> SYSERROR("failed to setsid");</div><div> exit(EXIT_FAILURE);</div><div> }</div>
<div>- freopen("/dev/null", "r", stdin);</div><div>- freopen("/dev/null", "w", stdout);</div><div>- freopen("/dev/null", "w", stderr);</div><div>
+ fd = freopen("/dev/null", "r", stdin);</div><div>+ if (!fd) {</div><div>+ SYSERROR("freopen failed");</div><div>+ exit(EXIT_FAILURE);</div><div>+ }</div>
<div>+ fd = freopen("/dev/null", "w", stdout);</div><div>+ if (!fd) {</div><div>+ SYSERROR("freopen failed");</div><div>+ exit(EXIT_FAILURE);</div><div>
+ }</div><div>+ fd = freopen("/dev/null", "w", stderr);</div><div>+ if (!fd) {</div><div>+ SYSERROR("freopen failed");</div><div>+ exit(EXIT_FAILURE);</div>
<div>+ }</div><div> close(pipefd[0]);</div><div> sprintf(pipefd_str, "%d", pipefd[1]);</div><div> execvp(args[0], args);</div><div><br></div><div style>and time to time (I think that only happens if monitord wasn't running initially) I get following while starting 10 containers in parallel</div>
<div style><br></div><div style><div>[caglar@qgq:~/Project/lxc/examples] sudo ./concurrent_start </div><div>Starting the container (2)...</div><div>Starting the container (6)...</div><div>Starting the container (1)...</div>
<div>lxc_container: Invalid argument - client failed to receive lxc_msg (monitord died?)</div><div>Starting the container (2) failed...</div><div>Starting the container (0)...</div><div>Starting the container (3)...</div>
<div>Starting the container (5)...</div><div>Starting the container (7)...</div><div>Starting the container (8)...</div><div>Starting the container (9)...</div><div>Starting the container (4)...</div><div><br></div><div style>
Best,</div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Apr 18, 2013 at 2:47 PM, Dwight Engen <span dir="ltr"><<a href="mailto:dwight.engen@oracle.com" target="_blank">dwight.engen@oracle.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On Thu, 18 Apr 2013 14:22:08 -0400<br>
S.Çağlar Onur <<a href="mailto:caglar@10ur.org">caglar@10ur.org</a>> wrote:<br>
<br>
> Hey Dwight,<br>
><br>
> This sounds great, I'll give it a try ASAP. In the meantime ,if you<br>
> want to test it yourself, you can grab go bindings from github via<br>
> "go get <a href="http://github.com/caglar10ur/lxc" target="_blank">github.com/caglar10ur/lxc</a>" but please keep in your mind that<br>
> you need my other two patches to start/stop work reliably [1]/[2].<br>
> <a href="https://github.com/caglar10ur/lxc/tree/master/examples" target="_blank">https://github.com/caglar10ur/lxc/tree/master/examples</a> directory<br>
> contains concurrent versions of create/start/stop/shutdown to test.<br>
><br>
> [1]<br>
> <a href="https://github.com/caglar10ur/lxc-upstream/commit/db60b9c38083e8ed553f3cbdd96ad410bd5eb7ed.patch" target="_blank">https://github.com/caglar10ur/lxc-upstream/commit/db60b9c38083e8ed553f3cbdd96ad410bd5eb7ed.patch</a><br>
> [2]<br>
> <a href="https://github.com/caglar10ur/lxc-upstream/commit/97d4daf4cc5016bcb01c72e5cf9fe3189e9df4f5.patch" target="_blank">https://github.com/caglar10ur/lxc-upstream/commit/97d4daf4cc5016bcb01c72e5cf9fe3189e9df4f5.patch</a><br>
><br>
> Cheers,<br>
<br>
</div>Thanks Caglar, I may give them a try as I've just built go the other<br>
day to try out docker.<br>
<div class="HOEnZb"><div class="h5"><br>
> On Thu, Apr 18, 2013 at 2:09 PM, Dwight Engen<br>
> <<a href="mailto:dwight.engen@oracle.com">dwight.engen@oracle.com</a>>wrote:<br>
><br>
> > After the recent discussions on the lxc monitor, I took a closer<br>
> > look at it. It seems like Serge was resigned to having some sort of<br>
> > daemon as having multicast unix domain supported by the kernel is<br>
> > not going to happen any time soon, and doing it over loopback has<br>
> > some complications too.<br>
> ><br>
> > Initially I thought it would be nice to just have lxc-start "be the<br>
> > daemon" as Stéphane had suggested. I tried this out, putting the fds<br>
> > for the consumer clients (lxc-monitors) in lxc_mainloop , but this<br>
> > is problematic for the scenario where lxc-monitor runs first and we<br>
> > miss some startup states.<br>
> ><br>
> > So here is what I'm proposing: when lxc-monitor starts, it attempts<br>
> > to start lxc-monitord. lxc-monitord creates a fifo and a socket on<br>
> > lxcpath.<br>
> ><br>
> > Producers (containers) wishing to send messages open the fifo and<br>
> > write their lxc_msg's on it. The reason I used a fifo is so that we<br>
> > get permission checks on the opening of it. Right now it is created<br>
> > 0600. This fixes the problem we have today where anyone can open<br>
> > the unix socket and send state changes in.<br>
> ><br>
> > Consumers (lxc-monitor) connect to the unix socket and each gets a<br>
> > copy of the messages. lxc-monitord can do a PEERCRED to ensure the<br>
> > client should be able to get these messages, this is one benefit to<br>
> > using unix domain. I don't think it would be difficult to write a<br>
> > lxc-monitor like client that would bridge the messages onto d-bus.<br>
> ><br>
> > When lxc-monitord hasn't had a client for 30 seconds it will exit.<br>
> ><br>
> > Attached is a proof of concept patch for your comments. It works for<br>
> > the common cases that I've tried but I know it needs some cleanup<br>
> > and testing if we like the approach. The monitor.c part is hard to<br>
> > read because that file is mostly rewritten.<br>
> ><br>
> > I need to test S.Çağlar Onur's scenario of parallel startup with his<br>
> > test script. We need to decide if we want to monitor per lxcpath or<br>
> > per container, today it is per path and I think that being able to<br>
> > do lxc-monitor * would be very useful.<br>
> ><br>
> > --<br>
> ><br>
> > Signed-off-by: Dwight Engen <<a href="mailto:dwight.engen@oracle.com">dwight.engen@oracle.com</a>><br>
> > ---<br>
> > src/lxc/Makefile.am | 2 +<br>
> > src/lxc/af_unix.c | 5 +-<br>
> > src/lxc/lxc_monitor.c | 2 +<br>
> > src/lxc/lxc_monitord.c | 334<br>
> > +++++++++++++++++++++++++++++++++++++++++++++++++<br>
> > src/lxc/mainloop.c | 11 +-<br>
> > src/lxc/mainloop.h | 7 ++<br>
> > src/lxc/monitor.c | 180 ++++++++++++++++----------<br>
> > src/lxc/monitor.h | 11 +-<br>
> > src/lxc/start.c | 7 +-<br>
> > 9 files changed, 484 insertions(+), 75 deletions(-)<br>
> > create mode 100644 src/lxc/lxc_monitord.c<br>
> ><br>
> > diff --git a/src/lxc/Makefile.am b/src/lxc/Makefile.am<br>
> > index ebeca466..1fa0fa8 100644<br>
> > --- a/src/lxc/Makefile.am<br>
> > +++ b/src/lxc/Makefile.am<br>
> > @@ -150,6 +150,7 @@ bin_PROGRAMS = \<br>
> > lxc-start \<br>
> > lxc-execute \<br>
> > lxc-monitor \<br>
> > + lxc-monitord \<br>
> > lxc-wait \<br>
> > lxc-console \<br>
> > lxc-freeze \<br>
> > @@ -181,6 +182,7 @@ lxc_freeze_SOURCES = lxc_freeze.c<br>
> > lxc_info_SOURCES = lxc_info.c<br>
> > lxc_init_SOURCES = lxc_init.c<br>
> > lxc_monitor_SOURCES = lxc_monitor.c<br>
> > +lxc_monitord_SOURCES = lxc_monitord.c<br>
> > lxc_restart_SOURCES = lxc_restart.c<br>
> > lxc_start_SOURCES = lxc_start.c<br>
> > lxc_stop_SOURCES = lxc_stop.c<br>
> > diff --git a/src/lxc/af_unix.c b/src/lxc/af_unix.c<br>
> > index eff13d4..ceef340 100644<br>
> > --- a/src/lxc/af_unix.c<br>
> > +++ b/src/lxc/af_unix.c<br>
> > @@ -52,6 +52,7 @@ int lxc_af_unix_open(const char *path, int type,<br>
> > int flags)<br>
> ><br>
> > addr.sun_family = AF_UNIX;<br>
> > /* copy entire buffer in case of abstract socket */<br>
> > + /* FIXME: this is unsafe if path is too long... */<br>
> > memcpy(addr.sun_path, path,<br>
> > path[0]?strlen(path):sizeof(addr.sun_path));<br>
> ><br>
> > @@ -61,7 +62,7 @@ int lxc_af_unix_open(const char *path, int type,<br>
> > int flags)<br>
> > errno = tmp;<br>
> > return -1;<br>
> > }<br>
> > -<br>
> > +<br>
> > if (type == SOCK_STREAM && listen(fd, 100)) {<br>
> > int tmp = errno;<br>
> > close(fd);<br>
> > @@ -76,7 +77,7 @@ int lxc_af_unix_close(int fd)<br>
> > {<br>
> > struct sockaddr_un addr;<br>
> > socklen_t addrlen = sizeof(addr);<br>
> > -<br>
> > +<br>
> > if (!getsockname(fd, (struct sockaddr *)&addr, &addrlen) &&<br>
> > addr.sun_path[0])<br>
> > unlink(addr.sun_path);<br>
> > diff --git a/src/lxc/lxc_monitor.c b/src/lxc/lxc_monitor.c<br>
> > index 8c15869..0ca829f 100644<br>
> > --- a/src/lxc/lxc_monitor.c<br>
> > +++ b/src/lxc/lxc_monitor.c<br>
> > @@ -87,6 +87,8 @@ int main(int argc, char *argv[])<br>
> > return -1;<br>
> > }<br>
> ><br>
> > + lxc_monitord_spawn(my_args.lxcpath);<br>
> > +<br>
> > fd = lxc_monitor_open(my_args.lxcpath);<br>
> > if (fd < 0)<br>
> > return -1;<br>
> > diff --git a/src/lxc/lxc_monitord.c b/src/lxc/lxc_monitord.c<br>
> > new file mode 100644<br>
> > index 0000000..3e9c708<br>
> > --- /dev/null<br>
> > +++ b/src/lxc/lxc_monitord.c<br>
> > @@ -0,0 +1,334 @@<br>
> > +/*<br>
> > + * lxc: linux Container library<br>
> > + *<br>
> > + * Copyright © 2012 Oracle.<br>
> > + *<br>
> > + * Authors:<br>
> > + * Dwight Engen <<a href="mailto:dwight.engen@oracle.com">dwight.engen@oracle.com</a>><br>
> > + *<br>
> > + * This library is free software; you can redistribute it and/or<br>
> > + * modify it under the terms of the GNU Lesser General Public<br>
> > + * License as published by the Free Software Foundation; either<br>
> > + * version 2.1 of the License, or (at your option) any later<br>
> > version.<br>
> > + *<br>
> > + * This library is distributed in the hope that it will be useful,<br>
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the<br>
> > GNU<br>
> > + * Lesser General Public License for more details.<br>
> > + *<br>
> > + * You should have received a copy of the GNU Lesser General Public<br>
> > + * License along with this library; if not, write to the Free<br>
> > Software<br>
> > + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA<br>
> > 02111-1307 USA<br>
> > + */<br>
> > +<br>
> > +#define _GNU_SOURCE<br>
> > +#include <stdio.h><br>
> > +#include <signal.h><br>
> > +#include <errno.h><br>
> > +#include <unistd.h><br>
> > +#include <string.h><br>
> > +#include <stdlib.h><br>
> > +#include <fcntl.h><br>
> > +#include <sys/types.h><br>
> > +#include <sys/stat.h><br>
> > +#include <sys/param.h><br>
> > +#include <sys/socket.h><br>
> > +#include <sys/un.h><br>
> > +#include <netinet/in.h><br>
> > +#include <net/if.h><br>
> > +<br>
> > +#include <lxc/af_unix.h><br>
> > +#include <lxc/log.h><br>
> > +#include <lxc/mainloop.h><br>
> > +#include <lxc/monitor.h><br>
> > +#include <lxc/utils.h><br>
> > +<br>
> > +lxc_log_define(lxc_monitord, lxc);<br>
> > +<br>
> > +/*<br>
> > + * Defines the structure to store the monitor information<br>
> > + * @fifofd : the file descriptor for publishers (containers)<br>
> > to write state<br>
> > + * @listenfd : the file descriptor for subscribers<br>
> > (lxc-monitors) to connect<br>
> > + * @clientfds : accepted client file descriptors<br>
> > + * @clientfds_cnt: the count of valid fds in clientfds<br>
> > + */<br>
> > +struct lxc_monitor {<br>
> > + int fifofd;<br>
> > + int listenfd;<br>
> > + int clientfds[1024];<br>
> > + int clientfds_cnt;<br>
> > + struct lxc_epoll_descr descr;<br>
> > +};<br>
> > +<br>
> > +static int lxc_monitord_fifo_create(struct lxc_monitor *mon, const<br>
> > char *lxcpath)<br>
> > +{<br>
> > + char fifo_path[PATH_MAX];<br>
> > + int ret;<br>
> > +<br>
> > + ret = snprintf(fifo_path, sizeof(fifo_path), "%s/monitor",<br>
> > lxcpath);<br>
> > + if (ret < 0 || ret >= sizeof(fifo_path)) {<br>
> > + ERROR("lxcpath too long to monitor fifo");<br>
> > + return -1;<br>
> > + }<br>
> > +<br>
> > + ret = mknod(fifo_path, S_IFIFO|S_IRUSR|S_IWUSR, 0);<br>
> > + if (ret < 0) {<br>
> > + INFO("existing monitor fifo at %s, monitor already<br>
> > running?", fifo_path);<br>
> > + return -1;<br>
> > + }<br>
> > + mon->fifofd = open(fifo_path, O_RDWR);<br>
> > + if (mon->fifofd < 0) {<br>
> > + ERROR("failed to open monitor fifo");<br>
> > + return -1;<br>
> > + }<br>
> > + return 0;<br>
> > +}<br>
> > +<br>
> > +static int lxc_monitord_fifo_delete(const char *lxcpath)<br>
> > +{<br>
> > + char fifo_path[PATH_MAX];<br>
> > + int ret;<br>
> > +<br>
> > + ret = snprintf(fifo_path, sizeof(fifo_path), "%s/monitor",<br>
> > lxcpath);<br>
> > + if (ret < 0 || ret >= sizeof(fifo_path)) {<br>
> > + ERROR("lxcpath too long to monitor fifo");<br>
> > + return -1;<br>
> > + }<br>
> > + unlink(fifo_path);<br>
> > + return 0;<br>
> > +}<br>
> > +<br>
> > +static void lxc_monitord_sockfd_remove(struct lxc_monitor *mon,<br>
> > int fd) {<br>
> > + int i;<br>
> > +<br>
> > + lxc_mainloop_del_handler(&mon->descr, fd);<br>
> > + close(fd);<br>
> > +<br>
> > + for (i = 0; i < mon->clientfds_cnt; i++) {<br>
> > + if (mon->clientfds[i] == fd)<br>
> > + break;<br>
> > + }<br>
> > + if (i >= mon->clientfds_cnt) {<br>
> > + CRIT("fd not found");<br>
> > + exit(EXIT_FAILURE);<br>
> > + }<br>
> > + DEBUG("MON: removing client fd:%d memmoving %p %p %lu<br>
> > bytes", fd,<br>
> > + &mon->clientfds[mon->clientfds_cnt],<br>
> > + &mon->clientfds[i],<br>
> > + (&mon->clientfds[mon->clientfds_cnt-1] -<br>
> > &mon->clientfds[i]) * sizeof(int));<br>
> > + memmove(&mon->clientfds[i], &mon->clientfds[i+1],<br>
> > + (&mon->clientfds[mon->clientfds_cnt-1] -<br>
> > &mon->clientfds[i]) * sizeof(int));<br>
> > + mon->clientfds_cnt--;<br>
> > +}<br>
> > +<br>
> > +static int lxc_monitord_sock_handler(int fd, void *data,<br>
> > + struct lxc_epoll_descr *descr)<br>
> > +{<br>
> > + struct lxc_monitor *mon = data;<br>
> > + lxc_monitord_sockfd_remove(mon, fd);<br>
> > + return 0;<br>
> > +}<br>
> > +<br>
> > +static int lxc_monitord_sock_accept(int fd, void *data,<br>
> > + struct lxc_epoll_descr *descr)<br>
> > +{<br>
> > + int ret,clientfd;<br>
> > + struct lxc_monitor *mon = data;<br>
> > +<br>
> > + ret = -1;<br>
> > + clientfd = accept(fd, NULL, 0);<br>
> > + if (clientfd < 0) {<br>
> > + SYSERROR("failed to accept connection");<br>
> > + goto out;<br>
> > + }<br>
> > +<br>
> > + if (fcntl(clientfd, F_SETFD, FD_CLOEXEC)) {<br>
> > + SYSERROR("failed to set close-on-exec on incoming<br>
> > connection");<br>
> > + goto err1;<br>
> > + }<br>
> > +<br>
> > + /* FIXME: check the connected clients credentials here,<br>
> > drop if we don't<br>
> > + * like them.<br>
> > + int on = 1;<br>
> > + if (setsockopt(clientfd, SOL_SOCKET, SO_PASSCRED, &on,<br>
> > sizeof(on))) {<br>
> > + ERROR("failed to enable credential on socket");<br>
> > + goto err1;<br>
> > + }<br>
> > + */<br>
> > +<br>
> > + ret = lxc_mainloop_add_handler(&mon->descr, clientfd,<br>
> > + lxc_monitord_sock_handler,<br>
> > mon);<br>
> > + if (ret) {<br>
> > + ERROR("failed to add socket handler");<br>
> > + goto err1;<br>
> > + }<br>
> > + mon->clientfds[mon->clientfds_cnt++] = clientfd;<br>
> > + INFO("accepted new client fd:%d clientfds_cnt:%d", clientfd,<br>
> > mon->clientfds_cnt);<br>
> > + goto out;<br>
> > +<br>
> > +err1:<br>
> > + close(clientfd);<br>
> > +out:<br>
> > + return ret;<br>
> > +}<br>
> > +<br>
> > +static int lxc_monitord_sock_create(struct lxc_monitor *mon, const<br>
> > char *lxcpath)<br>
> > +{<br>
> > + struct sockaddr_un addr;<br>
> > + int fd;<br>
> > +<br>
> > + if (lxc_monitor_sock_name(lxcpath, &addr) < 0)<br>
> > + return -1;<br>
> > +<br>
> > + fd = lxc_af_unix_open(addr.sun_path, SOCK_STREAM, O_TRUNC);<br>
> > + if (fd < 0) {<br>
> > + ERROR("failed to open unix socket : %s",<br>
> > strerror(errno));<br>
> > + return -1;<br>
> > + }<br>
> > +<br>
> > + mon->listenfd = fd;<br>
> > + return 0;<br>
> > +}<br>
> > +<br>
> > +static int lxc_monitord_sock_delete(const char *lxcpath)<br>
> > +{<br>
> > + struct sockaddr_un addr;<br>
> > +<br>
> > + if (lxc_monitor_sock_name(lxcpath, &addr) < 0)<br>
> > + return -1;<br>
> > + if (addr.sun_path[0])<br>
> > + unlink(addr.sun_path);<br>
> > + return 0;<br>
> > +}<br>
> > +<br>
> > +static int lxc_monitord_create(struct lxc_monitor *mon, const char<br>
> > *lxcpath)<br>
> > +{<br>
> > + int ret;<br>
> > +<br>
> > + ret = lxc_monitord_fifo_create(mon, lxcpath);<br>
> > + if (ret < 0)<br>
> > + return ret;<br>
> > +<br>
> > + ret = lxc_monitord_sock_create(mon, lxcpath);<br>
> > + return ret;<br>
> > +}<br>
> > +<br>
> > +static void lxc_monitord_delete(struct lxc_monitor *mon, const char<br>
> > *lxcpath)<br>
> > +{<br>
> > + int i;<br>
> > +<br>
> > + lxc_mainloop_del_handler(&mon->descr, mon->listenfd);<br>
> > + close(mon->listenfd);<br>
> > + lxc_monitord_sock_delete(lxcpath);<br>
> > +<br>
> > + lxc_mainloop_del_handler(&mon->descr, mon->fifofd);<br>
> > + close(mon->fifofd);<br>
> > + lxc_monitord_fifo_delete(lxcpath);<br>
> > +<br>
> > + for (i = 0; i < mon->clientfds_cnt; i++) {<br>
> > + lxc_mainloop_del_handler(&mon->descr,<br>
> > mon->clientfds[i]);<br>
> > + close(mon->clientfds[i]);<br>
> > + }<br>
> > + mon->clientfds_cnt = 0;<br>
> > +}<br>
> > +<br>
> > +static int lxc_monitord_fifo_handler(int fd, void *data,<br>
> > + struct lxc_epoll_descr *descr)<br>
> > +{<br>
> > + int ret,i;<br>
> > + struct lxc_msg msglxc;<br>
> > + struct lxc_monitor *mon = data;<br>
> > +<br>
> > + ret = read(fd, &msglxc, sizeof(msglxc));<br>
> > + if (ret != sizeof(msglxc)) {<br>
> > + SYSERROR("read fifo failed : %s", strerror(errno));<br>
> > + return 1;<br>
> > + }<br>
> > +<br>
> > + for (i = 0; i < mon->clientfds_cnt; i++) {<br>
> > + ret = write(mon->clientfds[i], &msglxc,<br>
> > sizeof(msglxc));<br>
> > + if (ret < 0) {<br>
> > + ERROR("write failed to client sock,<br>
> > removing");<br>
> > + lxc_monitord_sockfd_remove(mon, i);<br>
> > + }<br>
> > + }<br>
> > +<br>
> > + return 0;<br>
> > +}<br>
> > +<br>
> > +static int lxc_monitord_mainloop_add(struct lxc_monitor *mon)<br>
> > +{<br>
> > + int ret;<br>
> > +<br>
> > + ret = lxc_mainloop_add_handler(&mon->descr, mon->fifofd,<br>
> > + lxc_monitord_fifo_handler,<br>
> > mon);<br>
> > + if (ret < 0) {<br>
> > + ERROR("failed to add to mainloop monitor handler for<br>
> > fifo");<br>
> > + return -1;<br>
> > + }<br>
> > +<br>
> > + ret = lxc_mainloop_add_handler(&mon->descr, mon->listenfd,<br>
> > + lxc_monitord_sock_accept,<br>
> > mon);<br>
> > + if (ret < 0) {<br>
> > + ERROR("failed to add to mainloop monitor handler for<br>
> > listen socket");<br>
> > + return -1;<br>
> > + }<br>
> > +<br>
> > + return 0;<br>
> > +}<br>
> > +<br>
> > +static void lxc_monitord_sig_handler(int sig)<br>
> > +{<br>
> > + INFO("caught signal %d", sig);<br>
> > +}<br>
> > +<br>
> > +int main(int argc, char *argv[])<br>
> > +{<br>
> > + struct lxc_monitor mon;<br>
> > + int ret,pipefd;<br>
> > + char *lxcpath = argv[1];<br>
> > + char logpath[PATH_MAX];<br>
> > +<br>
> > + signal(SIGTERM, lxc_monitord_sig_handler);<br>
> > + snprintf(logpath, sizeof(logpath), "%s/lxc-monitord.log",<br>
> > lxcpath);<br>
> > + ret = lxc_log_init(NULL, logpath, "INFO", "lxc-monitord",<br>
> > 0);<br>
> > + if (ret)<br>
> > + return ret;<br>
> > +<br>
> > + pipefd = atoi(argv[2]);<br>
> > +<br>
> > + ret = EXIT_FAILURE;<br>
> > + memset(&mon, 0, sizeof(mon));<br>
> > + if (lxc_mainloop_open(&mon.descr)) {<br>
> > + ERROR("failed to create mainloop");<br>
> > + goto out;<br>
> > + }<br>
> > +<br>
> > + if (lxc_monitord_create(&mon, lxcpath)) {<br>
> > + goto out;<br>
> > + }<br>
> > +<br>
> > + /* sync with parent */<br>
> > + write(pipefd, "S", 1);<br>
> > + close(pipefd);<br>
> > +<br>
> > + if (lxc_monitord_mainloop_add(&mon)) {<br>
> > + ERROR("failed to add mainloop handlers");<br>
> > + goto out;<br>
> > + }<br>
> > +<br>
> > + NOTICE("monitoring lxcpath %s", lxcpath);<br>
> > + for(;;) {<br>
> > + ret = lxc_mainloop_timeout(&mon.descr, 1000 * 30);<br>
> > + if (mon.clientfds_cnt <= 0)<br>
> > + {<br>
> > + NOTICE("no clients for 30 seconds,<br>
> > exiting");<br>
> > + break;<br>
> > + }<br>
> > + }<br>
> > +<br>
> > + lxc_monitord_delete(&mon, lxcpath);<br>
> > + lxc_mainloop_close(&mon.descr);<br>
> > +out:<br>
> > + return ret;<br>
> > +}<br>
> > diff --git a/src/lxc/mainloop.c b/src/lxc/mainloop.c<br>
> > index 975215d..4805dcd 100644<br>
> > --- a/src/lxc/mainloop.c<br>
> > +++ b/src/lxc/mainloop.c<br>
> > @@ -38,7 +38,7 @@ struct mainloop_handler {<br>
> ><br>
> > #define MAX_EVENTS 10<br>
> ><br>
> > -int lxc_mainloop(struct lxc_epoll_descr *descr)<br>
> > +int lxc_mainloop_timeout(struct lxc_epoll_descr *descr, int<br>
> > timeout_ms) {<br>
> > int i, nfds;<br>
> > struct mainloop_handler *handler;<br>
> > @@ -46,7 +46,7 @@ int lxc_mainloop(struct lxc_epoll_descr *descr)<br>
> ><br>
> > for (;;) {<br>
> ><br>
> > - nfds = epoll_wait(descr->epfd, events, MAX_EVENTS,<br>
> > -1);<br>
> > + nfds = epoll_wait(descr->epfd, events, MAX_EVENTS,<br>
> > timeout_ms);<br>
> > if (nfds < 0) {<br>
> > if (errno == EINTR)<br>
> > continue;<br>
> > @@ -64,11 +64,18 @@ int lxc_mainloop(struct lxc_epoll_descr *descr)<br>
> > return 0;<br>
> > }<br>
> ><br>
> > + if (nfds == 0 && timeout_ms != 0)<br>
> > + return 0;<br>
> > +<br>
> > if (lxc_list_empty(&descr->handlers))<br>
> > return 0;<br>
> > }<br>
> > }<br>
> ><br>
> > +int lxc_mainloop(struct lxc_epoll_descr *descr) {<br>
> > + return lxc_mainloop_timeout(descr, -1);<br>
> > +}<br>
> > +<br>
> > int lxc_mainloop_add_handler(struct lxc_epoll_descr *descr, int fd,<br>
> > lxc_mainloop_callback_t callback, void<br>
> > *data) {<br>
> > diff --git a/src/lxc/mainloop.h b/src/lxc/mainloop.h<br>
> > index 6b16242..c79c4fd 100644<br>
> > --- a/src/lxc/mainloop.h<br>
> > +++ b/src/lxc/mainloop.h<br>
> > @@ -21,6 +21,9 @@<br>
> > * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA<br>
> > 02111-1307 USA */<br>
> ><br>
> > +#ifndef _mainloop_h<br>
> > +#define _mainloop_h<br>
> > +<br>
> > #include "list.h"<br>
> ><br>
> > struct lxc_epoll_descr {<br>
> > @@ -33,6 +36,8 @@ typedef int (*lxc_mainloop_callback_t)(int fd,<br>
> > void *data,<br>
> ><br>
> > extern int lxc_mainloop(struct lxc_epoll_descr *descr);<br>
> ><br>
> > +extern int lxc_mainloop_timeout(struct lxc_epoll_descr *descr, int<br>
> > timeout_ms);<br>
> > +<br>
> > extern int lxc_mainloop_add_handler(struct lxc_epoll_descr *descr,<br>
> > int fd, lxc_mainloop_callback_t callback,<br>
> > void *data);<br>
> > @@ -42,3 +47,5 @@ extern int lxc_mainloop_del_handler(struct<br>
> > lxc_epoll_descr *descr, int fd);<br>
> > extern int lxc_mainloop_open(struct lxc_epoll_descr *descr);<br>
> ><br>
> > extern int lxc_mainloop_close(struct lxc_epoll_descr *descr);<br>
> > +<br>
> > +#endif<br>
> > diff --git a/src/lxc/monitor.c b/src/lxc/monitor.c<br>
> > index afdaf67..0438fbc 100644<br>
> > --- a/src/lxc/monitor.c<br>
> > +++ b/src/lxc/monitor.c<br>
> > @@ -20,6 +20,7 @@<br>
> > * License along with this library; if not, write to the Free<br>
> > Software<br>
> > * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA<br>
> > 02111-1307 USA */<br>
> > +<br>
> > #include <stdio.h><br>
> > #include <errno.h><br>
> > #include <unistd.h><br>
> > @@ -30,7 +31,7 @@<br>
> > #include <sys/stat.h><br>
> > #include <sys/param.h><br>
> > #include <sys/socket.h><br>
> > -#include <sys/un.h><br>
> > +#include <sys/wait.h><br>
> > #include <netinet/in.h><br>
> > #include <net/if.h><br>
> ><br>
> > @@ -43,35 +44,24 @@<br>
> ><br>
> > lxc_log_define(lxc_monitor, lxc);<br>
> ><br>
> > -#ifndef UNIX_PATH_MAX<br>
> > -#define UNIX_PATH_MAX 108<br>
> > -#endif<br>
> > -<br>
> > -static void lxc_monitor_send(struct lxc_msg *msg, const char<br>
> > *lxcpath) +/* routines used by monitor publishers (containers) */<br>
> > +static void lxc_monitor_fifo_send(struct lxc_msg *msg, const char<br>
> > *lxcpath)<br>
> > {<br>
> > - int fd;<br>
> > - struct sockaddr_un addr = { .sun_family = AF_UNIX };<br>
> > - char *offset = &addr.sun_path[1];<br>
> > - size_t ret, len;<br>
> > + int fd,ret;<br>
> > + char fifo_path[PATH_MAX];<br>
> ><br>
> > - /*<br>
> > - * addr.sun_path is only 108 bytes.<br>
> > - * should we take a hash of lxcpath? a subset of it?<br>
> > - */<br>
> > - len = sizeof(addr.sun_path) - 1;<br>
> > - ret = snprintf(offset, len, "%s/lxc-monitor", lxcpath);<br>
> > - if (ret < 0 || ret >= len) {<br>
> > - ERROR("lxcpath too long to open monitor");<br>
> > + //FIXME: BuildAssert(sizeof(*msg) < PIPE_BUF), "write not<br>
> > guaranteed atomic");<br>
> > + ret = snprintf(fifo_path, sizeof(fifo_path), "%s/monitor",<br>
> > lxcpath);<br>
> > + if (ret < 0 || ret >= sizeof(fifo_path)) {<br>
> > + ERROR("lxcpath too long to open monitor fifo");<br>
> > return;<br>
> > }<br>
> > -<br>
> > - fd = socket(PF_UNIX, SOCK_DGRAM, 0);<br>
> > - if (fd < 0)<br>
> > + fd = open(fifo_path, O_WRONLY);<br>
> > + if (fd < 0) {<br>
> > + ERROR("failed to open monitor fifo %s", fifo_path);<br>
> > return;<br>
> > -<br>
> > - sendto(fd, msg, sizeof(*msg), 0,<br>
> > - (const struct sockaddr *)&addr, sizeof(addr));<br>
> > -<br>
> > + }<br>
> > + ret = write(fd, msg, sizeof(*msg));<br>
> > close(fd);<br>
> > }<br>
> ><br>
> > @@ -82,72 +72,74 @@ void lxc_monitor_send_state(const char *name,<br>
> > lxc_state_t state, const char *lxc<br>
> > strncpy(<a href="http://msg.name" target="_blank">msg.name</a>, name, sizeof(<a href="http://msg.name" target="_blank">msg.name</a>));<br>
> > <a href="http://msg.name" target="_blank">msg.name</a>[sizeof(<a href="http://msg.name" target="_blank">msg.name</a>) - 1] = 0;<br>
> ><br>
> > - lxc_monitor_send(&msg, lxcpath);<br>
> > + lxc_monitor_fifo_send(&msg, lxcpath);<br>
> > }<br>
> ><br>
> > -int lxc_monitor_open(const char *lxcpath)<br>
> > +<br>
> > +/* routines used by monitor subscribers (lxc-monitor) */<br>
> > +int lxc_monitor_close(int fd)<br>
> > {<br>
> > - struct sockaddr_un addr = { .sun_family = AF_UNIX };<br>
> > - char *offset = &addr.sun_path[1];<br>
> > - int fd;<br>
> > - size_t ret, len;<br>
> > + return close(fd);<br>
> > +}<br>
> ><br>
> > - /*<br>
> > - * addr.sun_path is only 108 bytes.<br>
> > - * should we take a hash of lxcpath? a subset of it?<br>
> > +int lxc_monitor_sock_name(const char *lxcpath, struct sockaddr_un<br>
> > *addr) {<br>
> > + size_t len,ret;<br>
> > + char *sockname = &addr->sun_path[0]; // 1 for abstract<br>
> > +<br>
> > + /* addr.sun_path is only 108 bytes.<br>
> > + * should we take a hash of lxcpath? a subset of it?<br>
> > ftok()? none<br>
> > + * are guaranteed.<br>
> > */<br>
> > - len = sizeof(addr.sun_path) - 1;<br>
> > - ret = snprintf(offset, len, "%s/lxc-monitor", lxcpath);<br>
> > + memset(addr, 0, sizeof(*addr));<br>
> > + addr->sun_family = AF_UNIX;<br>
> > + len = sizeof(addr->sun_path) - 1;<br>
> > + ret = snprintf(sockname, len, "%s/monitor-sock", lxcpath);<br>
> > if (ret < 0 || ret >= len) {<br>
> > - ERROR("lxcpath too long to open monitor");<br>
> > + ERROR("lxcpath too long for unix socket");<br>
> > return -1;<br>
> > }<br>
> > + return 0;<br>
> > +}<br>
> > +<br>
> > +int lxc_monitor_open(const char *lxcpath)<br>
> > +{<br>
> > + struct sockaddr_un addr;<br>
> > + size_t ret;<br>
> > + int fd;<br>
> ><br>
> > - fd = socket(PF_UNIX, SOCK_DGRAM, 0);<br>
> > + if (lxc_monitor_sock_name(lxcpath, &addr) < 0)<br>
> > + return -1;<br>
> > +<br>
> > + fd = socket(PF_UNIX, SOCK_STREAM, 0);<br>
> > if (fd < 0) {<br>
> > ERROR("socket : %s", strerror(errno));<br>
> > return -1;<br>
> > }<br>
> ><br>
> > - if (bind(fd, (struct sockaddr *)&addr, sizeof(addr))) {<br>
> > - ERROR("bind : %s", strerror(errno));<br>
> > - close(fd);<br>
> > - return -1;<br>
> > + ret = connect(fd, (struct sockaddr *)&addr, sizeof(addr));<br>
> > + if (ret < 0) {<br>
> > + /* try to spawn a lxc-monitord here instead ? */<br>
> > + ERROR("connect : %s", strerror(errno));<br>
> > + goto err1;<br>
> > }<br>
> > -<br>
> > return fd;<br>
> > +err1:<br>
> > + close(fd);<br>
> > + return ret;<br>
> > }<br>
> ><br>
> > -/* timeout of 0 means return immediately; -1 means wait forever */<br>
> > -int lxc_monitor_read_timeout(int fd, struct lxc_msg *msg, int<br>
> > timeout) +int lxc_monitor_read_timeout(int fd, struct lxc_msg<br>
> > *msglxc, int timeout) {<br>
> > struct sockaddr_un from;<br>
> > socklen_t len = sizeof(from);<br>
> > int ret;<br>
> > - fd_set rfds;<br>
> > - struct timeval tv;<br>
> ><br>
> > - if (timeout != -1) {<br>
> > - FD_ZERO(&rfds);<br>
> > - FD_SET(fd, &rfds);<br>
> > -<br>
> > - tv.tv_sec = timeout;<br>
> > - tv.tv_usec = 0;<br>
> > -<br>
> > - ret = select(fd+1, &rfds, NULL, NULL, &tv);<br>
> > - if (ret == -1)<br>
> > - return -1;<br>
> > - else if (!ret)<br>
> > - return -2; // timed out<br>
> > - }<br>
> > -<br>
> > - ret = recvfrom(fd, msg, sizeof(*msg), 0,<br>
> > + ret = recvfrom(fd, msglxc, sizeof(*msglxc), 0,<br>
> > (struct sockaddr *)&from, &len);<br>
> > - if (ret < 0) {<br>
> > - SYSERROR("failed to receive state");<br>
> > + if (ret <= 0) {<br>
> > + SYSERROR("client failed to receive lxc_msg (monitord<br>
> > died?)");<br>
> > return -1;<br>
> > }<br>
> > -<br>
> > return ret;<br>
> > }<br>
> ><br>
> > @@ -156,7 +148,61 @@ int lxc_monitor_read(int fd, struct lxc_msg<br>
> > *msg) return lxc_monitor_read_timeout(fd, msg, -1);<br>
> > }<br>
> ><br>
> > -int lxc_monitor_close(int fd)<br>
> > +<br>
> > +<br>
> > +/* used to spawn a monitord either on startup of a container, or<br>
> > when<br>
> > + * lxc-monitor starts<br>
> > + */<br>
> > +int lxc_monitord_spawn(const char *lxcpath)<br>
> > {<br>
> > - return close(fd);<br>
> > + pid_t pid1,pid2;<br>
> > + int pipefd[2];<br>
> > + char pipefd_str[10];<br>
> > +<br>
> > + char * const args[] = {<br>
> > + "/usr/bin/lxc-monitord",<br>
> > + (char *)lxcpath,<br>
> > + pipefd_str,<br>
> > + NULL,<br>
> > + };<br>
> > +<br>
> > + /* double fork to avoid zombies when monitord exits */<br>
> > + pid1 = fork();<br>
> > + if (pid1 < 0) {<br>
> > + SYSERROR("failed to fork");<br>
> > + return -1;<br>
> > + }<br>
> > +<br>
> > + if (pid1) {<br>
> > + waitpid(pid1, NULL, 0);<br>
> > + return 0;<br>
> > + }<br>
> > +<br>
> > + if (pipe(pipefd) < 0) {<br>
> > + SYSERROR("failed to create pipe");<br>
> > + exit(EXIT_FAILURE);<br>
> > + }<br>
> > +<br>
> > + pid2 = fork();<br>
> > + if (pid2) {<br>
> > + char c;<br>
> > + /* wait for daemon to create socket */<br>
> > + close(pipefd[1]);<br>
> > + read(pipefd[0], &c, 1);<br>
> > + close(pipefd[0]);<br>
> > + exit(EXIT_SUCCESS);<br>
> > + }<br>
> > +<br>
> > + umask(0);<br>
> > + if (setsid() < 0) {<br>
> > + SYSERROR("failed to setsid");<br>
> > + exit(EXIT_FAILURE);<br>
> > + }<br>
> > + freopen("/dev/null", "r", stdin);<br>
> > + freopen("/dev/null", "w", stdout);<br>
> > + freopen("/dev/null", "w", stderr);<br>
> > + close(pipefd[0]);<br>
> > + sprintf(pipefd_str, "%d", pipefd[1]);<br>
> > + execvp(args[0], args);<br>
> > + exit(EXIT_FAILURE);<br>
> > }<br>
> > diff --git a/src/lxc/monitor.h b/src/lxc/monitor.h<br>
> > index 8bef4c7..faa606c 100644<br>
> > --- a/src/lxc/monitor.h<br>
> > +++ b/src/lxc/monitor.h<br>
> > @@ -24,6 +24,10 @@<br>
> > #define __monitor_h<br>
> ><br>
> > #include <sys/param.h><br>
> > +#include <sys/un.h><br>
> > +<br>
> > +#include <lxc/conf.h><br>
> > +#include <lxc/mainloop.h><br>
> ><br>
> > typedef enum {<br>
> > lxc_msg_state,<br>
> > @@ -32,11 +36,14 @@ typedef enum {<br>
> ><br>
> > struct lxc_msg {<br>
> > lxc_msg_type_t type;<br>
> > - char name[MAXPATHLEN];<br>
> > + char name[NAME_MAX+1];<br>
> > int value;<br>
> > };<br>
> ><br>
> > -void lxc_monitor_send_state(const char *name, lxc_state_t state,<br>
> > +extern int lxc_monitor_open(const char *lxcpath);<br>
> > +extern int lxc_monitor_sock_name(const char *lxcpath, struct<br>
> > sockaddr_un *addr);<br>
> > +extern void lxc_monitor_send_state(const char *name, lxc_state_t<br>
> > state, const char *lxcpath);<br>
> > +extern int lxc_monitord_spawn(const char *lxcpath);<br>
> ><br>
> > #endif<br>
> > diff --git a/src/lxc/start.c b/src/lxc/start.c<br>
> > index aefccd6..9451fb7 100644<br>
> > --- a/src/lxc/start.c<br>
> > +++ b/src/lxc/start.c<br>
> > @@ -429,6 +429,9 @@ struct lxc_handler *lxc_init(const char *name,<br>
> > struct lxc_conf *conf, const char<br>
> > if (lxc_command_init(name, handler, lxcpath))<br>
> > goto out_free_name;<br>
> ><br>
> > + if (lxc_monitord_spawn(lxcpath))<br>
> > + goto out_close_maincmd_fd;<br>
> > +<br>
> > if (lxc_read_seccomp_config(conf) != 0) {<br>
> > ERROR("failed loading seccomp policy");<br>
> > goto out_close_maincmd_fd;<br>
> > @@ -437,7 +440,7 @@ struct lxc_handler *lxc_init(const char *name,<br>
> > struct lxc_conf *conf, const char<br>
> > /* Begin the set the state to STARTING*/<br>
> > if (lxc_set_state(name, handler, STARTING)) {<br>
> > ERROR("failed to set state '%s'",<br>
> > lxc_state2str(STARTING));<br>
> > - goto out_free_name;<br>
> > + goto out_close_maincmd_fd;<br>
> > }<br>
> ><br>
> > /* Start of environment variable setup for hooks */<br>
> > @@ -808,7 +811,7 @@ int lxc_spawn(struct lxc_handler *handler)<br>
> > /* TODO - pass lxc.cgroup.dir (or user's pam cgroup) in for<br>
> > first argument */<br>
> > if ((handler->cgroup = lxc_cgroup_path_create(NULL, name))<br>
> > == NULL) goto out_delete_net;<br>
> > -<br>
> > +<br>
> > if (lxc_cgroup_enter(handler->cgroup, handler->pid) < 0)<br>
> > goto out_delete_net;<br>
> ><br>
> > --<br>
> > 1.8.1.4<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > ------------------------------------------------------------------------------<br>
> > Precog is a next-generation analytics platform capable of advanced<br>
> > analytics on semi-structured data. The platform includes APIs for<br>
> > building apps and a phenomenal toolset for data science. Developers<br>
> > can use our toolset for easy data analysis & visualization. Get a<br>
> > free account!<br>
> > <a href="http://www2.precog.com/precogplatform/slashdotnewsletter" target="_blank">http://www2.precog.com/precogplatform/slashdotnewsletter</a><br>
> > _______________________________________________ Lxc-devel mailing<br>
> > list <a href="mailto:Lxc-devel@lists.sourceforge.net">Lxc-devel@lists.sourceforge.net</a><br>
> > <a href="https://lists.sourceforge.net/lists/listinfo/lxc-devel" target="_blank">https://lists.sourceforge.net/lists/listinfo/lxc-devel</a><br>
> ><br>
><br>
><br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>S.Çağlar Onur <<a href="mailto:caglar@10ur.org">caglar@10ur.org</a>>
</div>