<div dir="ltr">Hi Dwight,<div><br></div><div><div class="gmail_extra"><div class="gmail_quote">On Mon, Apr 22, 2013 at 5:42 PM, Dwight Engen <span dir="ltr"><<a href="mailto:dwight.engen@oracle.com" target="_blank">dwight.engen@oracle.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">[...]<br>
<div class="im">> > > So here is what I'm proposing: when lxc-monitor starts, it<br>
> > > attempts to start lxc-monitord. lxc-monitord creates a fifo and a<br>
> > > socket on lxcpath.<br>
> ><br>
> > Thanks, Dwight. Looks awesome. Some comments below, but I'm only<br>
> > not adding an ack bc you say you want to make some changes first<br>
> > anyway.<br>
> ><br>
> > When you send the next version I'll run it through my testsuite (and<br>
> > presumably ack).<br>
<br>
</div>Serge, this one should be testable so feel free to hit it with your<br>
testsuite if you want, but I'm still sending it as RFC due to the<br>
questions raised below. Caglar, I've tested this pretty thoroughly<br>
with your python parallel start script doing 40 containers<br>
parallel 4 ways (and put into a loop running for a few minutes) and<br>
some other parallel startup cases I made. If you want to test it<br>
yourself too, that'd be great.</blockquote><div><br></div><div style>Yep, it's working great here with my set of tools as well. Awesome work! Thanks for doing this...</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">
> > ...<br>
> > > +static void lxc_monitord_delete(struct lxc_monitor *mon, const<br>
> > > char *lxcpath) +{<br>
> > > + int i;<br>
> > > +<br>
> > > + lxc_mainloop_del_handler(&mon->descr, mon->listenfd);<br>
> > > + close(mon->listenfd);<br>
> ><br>
> > The ordering here might need to change to ensure we don't get any<br>
> > clients hanging between the two steps.<br>
><br>
> Hmm, good point. Have to think about this, there might be a race the<br>
> other way with still having the fd in the epoll set also.<br>
<br>
</div>I think the way I have it is the right order. It looks to me from<br>
the epoll_ctl manpage that we'll get EBADF if fd isn't valid at the<br>
time of EPOLL_CTL_DEL.<br>
<br>
<br>
I have a couple of new questions too :( In the current code there is<br>
the following flow (only when starting a container as daemon through<br>
the api):<br>
<br>
lxcapi_start()<br>
wait_on_daemonized_start()<br>
lxcapi_wait()<br>
lxc_wait()<br>
lxc_monitor_open()<br>
<br>
This is racy with multiple starters all trying to bind the socket<br>
address and is the source of Caglar's original problem. It looks to me<br>
like it is also racy with respect to the child container<br>
setting the state, but there is a timeout safety on the wait side so<br>
we don't hang.<br>
<br>
In the change I'm proposing, I put in a call to lxc_monitord_spawn() in<br>
only this daemon case which should mean there is always a place to<br>
deliver the status and no races between parent and child (because of<br>
sync'ing on the pipe), but has the drawback of having monitord always<br>
get started when the container is started daemonized. This is the only<br>
flow I found that used lxc_wait internally. Stephane, you might want to<br>
comment on this as it looks like you added the<br>
wait_on_daemonized_start() stuff, I'd be happy if my analysis missed<br>
something :) At least monitord will go away after startup, but it'd be<br>
nice not to have to start it at all. Any thoughts?<br>
<br>
The other question is: I do getsockopt SO_PEERCRED and<br>
check against the effective uid which I believe is correct in case<br>
the caller is setuid. What I'm wondering is why the routines in<br>
af_unix.c are passing/checking against the real?</blockquote><div><br></div><div style>I believe you are right and those needs to be replaced with geteuid/getegid to handle elevated privileges </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">
<br>
--<br>
<br>
Signed-off-by: Dwight Engen <<a href="mailto:dwight.engen@oracle.com">dwight.engen@oracle.com</a>><br>
---<br>
</div> .gitignore | 1 +<br>
src/lxc/Makefile.am | 2 +<br>
src/lxc/lxc_console.c | 4 +-<br>
src/lxc/lxc_monitor.c | 2 +<br>
src/lxc/lxc_monitord.c | 377 +++++++++++++++++++++++++++++++++++++++++++++++++<br>
src/lxc/lxccontainer.c | 6 +-<br>
src/lxc/mainloop.c | 7 +-<br>
src/lxc/mainloop.h | 7 +-<br>
src/lxc/monitor.c | 192 ++++++++++++++++++-------<br>
src/lxc/monitor.h | 10 +-<br>
src/lxc/start.c | 4 +-<br>
src/lxc/utils.h | 26 ++++<br>
12 files changed, 574 insertions(+), 64 deletions(-)<br>
create mode 100644 src/lxc/lxc_monitord.c<br>
<br>
diff --git a/.gitignore b/.gitignore<br>
index 905a2dc..c614a75 100644<br>
--- a/.gitignore<br>
+++ b/.gitignore<br>
@@ -52,6 +52,7 @@ src/lxc/lxc-info<br>
src/lxc/lxc-init<br>
src/lxc/lxc-kill<br>
src/lxc/lxc-monitor<br>
+src/lxc/lxc-monitord<br>
src/lxc/lxc-netstat<br>
src/lxc/lxc-ps<br>
src/lxc/lxc-restart<br>
<div class="im">diff --git a/src/lxc/Makefile.am b/src/lxc/Makefile.am<br>
index ebeca466..1fa0fa8 100644<br>
--- a/src/lxc/Makefile.am<br>
+++ b/src/lxc/Makefile.am<br>
@@ -150,6 +150,7 @@ bin_PROGRAMS = \<br>
lxc-start \<br>
lxc-execute \<br>
lxc-monitor \<br>
+ lxc-monitord \<br>
lxc-wait \<br>
lxc-console \<br>
lxc-freeze \<br>
@@ -181,6 +182,7 @@ lxc_freeze_SOURCES = lxc_freeze.c<br>
lxc_info_SOURCES = lxc_info.c<br>
lxc_init_SOURCES = lxc_init.c<br>
lxc_monitor_SOURCES = lxc_monitor.c<br>
+lxc_monitord_SOURCES = lxc_monitord.c<br>
lxc_restart_SOURCES = lxc_restart.c<br>
lxc_start_SOURCES = lxc_start.c<br>
lxc_stop_SOURCES = lxc_stop.c<br>
</div>diff --git a/src/lxc/lxc_console.c b/src/lxc/lxc_console.c<br>
index 643c442..f6659f6 100644<br>
--- a/src/lxc/lxc_console.c<br>
+++ b/src/lxc/lxc_console.c<br>
@@ -241,7 +241,7 @@ Type <Ctrl+%1$c q> to exit the console, \<br>
goto out_mainloop_open;<br>
}<br>
<br>
- err = lxc_mainloop(&descr);<br>
+ err = lxc_mainloop(&descr, -1);<br>
if (err) {<br>
ERROR("mainloop returned an error");<br>
goto out_mainloop_open;<br>
@@ -255,7 +255,7 @@ out_mainloop_open:<br>
out:<br>
/* Restore previous terminal parameter */<br>
tcsetattr(0, TCSAFLUSH, &oldtios);<br>
-<br>
+<br>
/* Return to line it is */<br>
printf("\n");<br>
<div class="im"><br>
diff --git a/src/lxc/lxc_monitor.c b/src/lxc/lxc_monitor.c<br>
index 8c15869..0ca829f 100644<br>
--- a/src/lxc/lxc_monitor.c<br>
+++ b/src/lxc/lxc_monitor.c<br>
@@ -87,6 +87,8 @@ int main(int argc, char *argv[])<br>
return -1;<br>
}<br>
<br>
+ lxc_monitord_spawn(my_args.lxcpath);<br>
+<br>
fd = lxc_monitor_open(my_args.lxcpath);<br>
if (fd < 0)<br>
return -1;<br>
diff --git a/src/lxc/lxc_monitord.c b/src/lxc/lxc_monitord.c<br>
new file mode 100644<br>
</div>index 0000000..8857aec<br>
--- /dev/null<br>
+++ b/src/lxc/lxc_monitord.c<br>
@@ -0,0 +1,377 @@<br>
<div><div class="h5">+/*<br>
+ * lxc: linux Container library<br>
+ *<br>
+ * Copyright © 2012 Oracle.<br>
+ *<br>
+ * Authors:<br>
+ * Dwight Engen <<a href="mailto:dwight.engen@oracle.com">dwight.engen@oracle.com</a>><br>
+ *<br>
+ * This library is free software; you can redistribute it and/or<br>
+ * modify it under the terms of the GNU Lesser General Public<br>
+ * License as published by the Free Software Foundation; either<br>
+ * version 2.1 of the License, or (at your option) any later version.<br>
+ *<br>
+ * This library is distributed in the hope that it will be useful,<br>
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of<br>
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU<br>
+ * Lesser General Public License for more details.<br>
+ *<br>
+ * You should have received a copy of the GNU Lesser General Public<br>
+ * License along with this library; if not, write to the Free Software<br>
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA<br>
+ */<br>
+<br>
+#define _GNU_SOURCE<br>
+#include <stdio.h><br>
+#include <signal.h><br>
+#include <errno.h><br>
+#include <unistd.h><br>
+#include <string.h><br>
+#include <stdlib.h><br>
+#include <fcntl.h><br>
+#include <sys/types.h><br>
+#include <sys/stat.h><br>
+#include <sys/param.h><br>
+#include <sys/socket.h><br>
+#include <sys/un.h><br>
+#include <netinet/in.h><br>
+#include <net/if.h><br>
+<br>
+#include <lxc/af_unix.h><br>
+#include <lxc/log.h><br>
+#include <lxc/mainloop.h><br>
+#include <lxc/monitor.h><br>
+#include <lxc/utils.h><br>
+<br>
+lxc_log_define(lxc_monitord, lxc);<br>
+<br>
</div></div>+static struct lxc_monitor mon;<br>
+<br>
+static void lxc_monitord_cleanup(void);<br>
<div class="im">+<br>
+/*<br>
+ * Defines the structure to store the monitor information<br>
</div>+ * @lxcpath : the path being monitored<br>
<div class="im">+ * @fifofd : the file descriptor for publishers (containers) to write state<br>
+ * @listenfd : the file descriptor for subscribers (lxc-monitors) to connect<br>
+ * @clientfds : accepted client file descriptors<br>
+ * @clientfds_cnt: the count of valid fds in clientfds<br>
</div>+ * @descr : the lxc_mainloop state<br>
+ */<br>
+struct lxc_monitor {<br>
+ const char *lxcpath;<br>
<div class="im">+ int fifofd;<br>
+ int listenfd;<br>
+ int clientfds[1024];<br>
+ int clientfds_cnt;<br>
+ struct lxc_epoll_descr descr;<br>
+};<br>
+<br>
</div>+static int lxc_monitord_fifo_create(struct lxc_monitor *mon)<br>
<div class="im">+{<br>
+ char fifo_path[PATH_MAX];<br>
+ int ret;<br>
+<br>
</div>+ ret = snprintf(fifo_path, sizeof(fifo_path), "%s/monitor-fifo", mon->lxcpath);<br>
<div class="im">+ if (ret < 0 || ret >= sizeof(fifo_path)) {<br>
+ ERROR("lxcpath too long to monitor fifo");<br>
+ return -1;<br>
+ }<br>
+<br>
+ ret = mknod(fifo_path, S_IFIFO|S_IRUSR|S_IWUSR, 0);<br>
+ if (ret < 0) {<br>
</div>+ INFO("monitor fifo %s exists, already running?", fifo_path);<br>
<div class="im">+ return -1;<br>
+ }<br>
+<br>
+ mon->fifofd = open(fifo_path, O_RDWR);<br>
+ if (mon->fifofd < 0) {<br>
</div>+ unlink(fifo_path);<br>
<div class="im">+ ERROR("failed to open monitor fifo");<br>
+ return -1;<br>
+ }<br>
+ return 0;<br>
+}<br>
+<br>
</div>+static int lxc_monitord_fifo_delete(struct lxc_monitor *mon)<br>
<div class="im">+{<br>
+ char fifo_path[PATH_MAX];<br>
+ int ret;<br>
+<br>
</div>+ ret = snprintf(fifo_path, sizeof(fifo_path), "%s/monitor-fifo", mon->lxcpath);<br>
<div class="im">+ if (ret < 0 || ret >= sizeof(fifo_path)) {<br>
+ ERROR("lxcpath too long to monitor fifo");<br>
+ return -1;<br>
+ }<br>
+ unlink(fifo_path);<br>
+ return 0;<br>
+}<br>
+<br>
+static void lxc_monitord_sockfd_remove(struct lxc_monitor *mon, int fd) {<br>
</div>+ int i;<br>
+<br>
+ if (lxc_mainloop_del_handler(&mon->descr, fd))<br>
+ CRIT("fd:%d not found in mainloop", fd);<br>
<div class="im">+ close(fd);<br>
+<br>
+ for (i = 0; i < mon->clientfds_cnt; i++) {<br>
+ if (mon->clientfds[i] == fd)<br>
+ break;<br>
+ }<br>
+ if (i >= mon->clientfds_cnt) {<br>
</div>+ CRIT("fd:%d not found in clients array", fd);<br>
+ lxc_monitord_cleanup();<br>
+ exit(EXIT_FAILURE);<br>
+ }<br>
+<br>
<div class="im">+ memmove(&mon->clientfds[i], &mon->clientfds[i+1],<br>
+ (&mon->clientfds[mon->clientfds_cnt-1] - &mon->clientfds[i]) * sizeof(int));<br>
+ mon->clientfds_cnt--;<br>
+}<br>
+<br>
+static int lxc_monitord_sock_handler(int fd, void *data,<br>
+ struct lxc_epoll_descr *descr)<br>
+{<br>
+ struct lxc_monitor *mon = data;<br>
+<br>
+ lxc_monitord_sockfd_remove(mon, fd);<br>
+ return 0;<br>
+}<br>
+<br>
+static int lxc_monitord_sock_accept(int fd, void *data,<br>
+ struct lxc_epoll_descr *descr)<br>
+{<br>
+ int ret,clientfd;<br>
+ struct lxc_monitor *mon = data;<br>
</div>+ struct ucred cred;<br>
+ socklen_t credsz = sizeof(cred);<br>
<div class="im">+<br>
+ ret = -1;<br>
+ clientfd = accept(fd, NULL, 0);<br>
+ if (clientfd < 0) {<br>
+ SYSERROR("failed to accept connection");<br>
+ goto out;<br>
+ }<br>
+<br>
+ if (fcntl(clientfd, F_SETFD, FD_CLOEXEC)) {<br>
+ SYSERROR("failed to set close-on-exec on incoming connection");<br>
+ goto err1;<br>
+ }<br>
+<br>
</div>+ if (getsockopt(clientfd, SOL_SOCKET, SO_PEERCRED, &cred, &credsz))<br>
+ {<br>
+ ERROR("failed to get credentials on socket");<br>
+ goto err1;<br>
+ }<br>
+ if (cred.uid && cred.uid != geteuid()) {<br>
+ WARN("monitor denied for uid:%d", cred.uid);<br>
+ ret = -EACCES;<br>
<div class="im">+ goto err1;<br>
+ }<br>
+<br>
</div><div class="im">+ ret = lxc_mainloop_add_handler(&mon->descr, clientfd,<br>
+ lxc_monitord_sock_handler, mon);<br>
+ if (ret) {<br>
+ ERROR("failed to add socket handler");<br>
+ goto err1;<br>
+ }<br>
+ mon->clientfds[mon->clientfds_cnt++] = clientfd;<br>
</div>+ INFO("accepted client fd:%d clients:%d", clientfd, mon->clientfds_cnt);<br>
<div class="im">+ goto out;<br>
+<br>
+err1:<br>
+ close(clientfd);<br>
+out:<br>
+ return ret;<br>
+}<br>
+<br>
</div>+static int lxc_monitord_sock_create(struct lxc_monitor *mon)<br>
<div class="im">+{<br>
+ struct sockaddr_un addr;<br>
+ int fd;<br>
+<br>
</div>+ if (lxc_monitor_sock_name(mon->lxcpath, &addr) < 0)<br>
<div class="im">+ return -1;<br>
+<br>
+ fd = lxc_af_unix_open(addr.sun_path, SOCK_STREAM, O_TRUNC);<br>
+ if (fd < 0) {<br>
+ ERROR("failed to open unix socket : %s", strerror(errno));<br>
+ return -1;<br>
+ }<br>
+<br>
+ mon->listenfd = fd;<br>
+ return 0;<br>
+}<br>
+<br>
</div>+static int lxc_monitord_sock_delete(struct lxc_monitor *mon)<br>
<div class="im">+{<br>
+ struct sockaddr_un addr;<br>
+<br>
</div>+ if (lxc_monitor_sock_name(mon->lxcpath, &addr) < 0)<br>
<div class="im">+ return -1;<br>
+ if (addr.sun_path[0])<br>
+ unlink(addr.sun_path);<br>
+ return 0;<br>
+}<br>
+<br>
</div>+static int lxc_monitord_create(struct lxc_monitor *mon)<br>
<div class="im">+{<br>
+ int ret;<br>
+<br>
</div>+ ret = lxc_monitord_fifo_create(mon);<br>
<div class="im">+ if (ret < 0)<br>
+ return ret;<br>
+<br>
</div>+ ret = lxc_monitord_sock_create(mon);<br>
<div class="im">+ return ret;<br>
+}<br>
+<br>
</div>+static void lxc_monitord_delete(struct lxc_monitor *mon)<br>
<div class="im">+{<br>
+ int i;<br>
+<br>
+ lxc_mainloop_del_handler(&mon->descr, mon->listenfd);<br>
+ close(mon->listenfd);<br>
</div>+ lxc_monitord_sock_delete(mon);<br>
+<br>
<div class="im">+ lxc_mainloop_del_handler(&mon->descr, mon->fifofd);<br>
+ close(mon->fifofd);<br>
</div>+ lxc_monitord_fifo_delete(mon);<br>
<div class="im">+<br>
+ for (i = 0; i < mon->clientfds_cnt; i++) {<br>
+ lxc_mainloop_del_handler(&mon->descr, mon->clientfds[i]);<br>
+ close(mon->clientfds[i]);<br>
+ }<br>
+ mon->clientfds_cnt = 0;<br>
+}<br>
+<br>
+static int lxc_monitord_fifo_handler(int fd, void *data,<br>
+ struct lxc_epoll_descr *descr)<br>
+{<br>
+ int ret,i;<br>
+ struct lxc_msg msglxc;<br>
+ struct lxc_monitor *mon = data;<br>
+<br>
+ ret = read(fd, &msglxc, sizeof(msglxc));<br>
+ if (ret != sizeof(msglxc)) {<br>
+ SYSERROR("read fifo failed : %s", strerror(errno));<br>
+ return 1;<br>
+ }<br>
+<br>
+ for (i = 0; i < mon->clientfds_cnt; i++) {<br>
</div>+ DEBUG("writing client fd:%d", mon->clientfds[i]);<br>
<div class="im">+ ret = write(mon->clientfds[i], &msglxc, sizeof(msglxc));<br>
+ if (ret < 0) {<br>
</div>+ ERROR("write failed to client sock:%d %d %s",<br>
+ mon->clientfds[i], errno, strerror(errno));<br>
<div><div class="h5">+ }<br>
+ }<br>
+<br>
+ return 0;<br>
+}<br>
+<br>
+static int lxc_monitord_mainloop_add(struct lxc_monitor *mon)<br>
+{<br>
+ int ret;<br>
+<br>
+ ret = lxc_mainloop_add_handler(&mon->descr, mon->fifofd,<br>
+ lxc_monitord_fifo_handler, mon);<br>
+ if (ret < 0) {<br>
+ ERROR("failed to add to mainloop monitor handler for fifo");<br>
+ return -1;<br>
+ }<br>
+<br>
+ ret = lxc_mainloop_add_handler(&mon->descr, mon->listenfd,<br>
+ lxc_monitord_sock_accept, mon);<br>
+ if (ret < 0) {<br>
+ ERROR("failed to add to mainloop monitor handler for listen socket");<br>
+ return -1;<br>
+ }<br>
+<br>
+ return 0;<br>
+}<br>
+<br>
</div></div>+static void lxc_monitord_cleanup(void)<br>
+{<br>
+ lxc_monitord_delete(&mon);<br>
<div class="im">+}<br>
+<br>
+static void lxc_monitord_sig_handler(int sig)<br>
+{<br>
+ INFO("caught signal %d", sig);<br>
</div>+ lxc_monitord_cleanup();<br>
+ exit(EXIT_SUCCESS);<br>
<div class="im">+}<br>
+<br>
+int main(int argc, char *argv[])<br>
+{<br>
</div><div class="im">+ int ret,pipefd;<br>
+ char *lxcpath = argv[1];<br>
+ char logpath[PATH_MAX];<br>
</div>+ sigset_t mask;<br>
<div class="im">+<br>
+ snprintf(logpath, sizeof(logpath), "%s/lxc-monitord.log", lxcpath);<br>
</div>+ ret = lxc_log_init(NULL, logpath, "NOTICE", "lxc-monitord", 0);<br>
<div class="im">+ if (ret)<br>
+ return ret;<br>
+<br>
+ pipefd = atoi(argv[2]);<br>
+<br>
</div>+ if (sigfillset(&mask) ||<br>
+ sigdelset(&mask, SIGILL) ||<br>
+ sigdelset(&mask, SIGSEGV) ||<br>
+ sigdelset(&mask, SIGBUS) ||<br>
+ sigdelset(&mask, SIGTERM) ||<br>
+ sigprocmask(SIG_BLOCK, &mask, NULL)) {<br>
+ SYSERROR("failed to set signal mask");<br>
<div class="im">+ return -1;<br>
+ }<br>
+<br>
</div>+ signal(SIGILL, lxc_monitord_sig_handler);<br>
+ signal(SIGSEGV, lxc_monitord_sig_handler);<br>
+ signal(SIGBUS, lxc_monitord_sig_handler);<br>
+ signal(SIGTERM, lxc_monitord_sig_handler);<br>
<div class="im">+<br>
+ ret = EXIT_FAILURE;<br>
+ memset(&mon, 0, sizeof(mon));<br>
</div>+ mon.lxcpath = lxcpath;<br>
<div class="im">+ if (lxc_mainloop_open(&mon.descr)) {<br>
+ ERROR("failed to create mainloop");<br>
+ goto out;<br>
+ }<br>
+<br>
</div>+ if (lxc_monitord_create(&mon)) {<br>
<div class="im">+ goto out;<br>
+ }<br>
+<br>
</div>+ /* sync with parent, we're ignoring the return from write<br>
+ * because regardless if it works or not, the following<br>
+ * close will sync us with the parent process. the<br>
+ * if-empty-statement construct is to quiet the<br>
+ * warn-unused-result warning.<br>
+ */<br>
+ if (write(pipefd, "S", 1)) ;<br>
<div class="im">+ close(pipefd);<br>
+<br>
+ if (lxc_monitord_mainloop_add(&mon)) {<br>
+ ERROR("failed to add mainloop handlers");<br>
+ goto out;<br>
+ }<br>
+<br>
</div>+ NOTICE("monitoring lxcpath %s", mon.lxcpath);<br>
+ for(;;) {<br>
+ ret = lxc_mainloop(&mon.descr, 1000 * 30);<br>
<div class="im">+ if (mon.clientfds_cnt <= 0)<br>
+ {<br>
+ NOTICE("no clients for 30 seconds, exiting");<br>
+ break;<br>
+ }<br>
+ }<br>
+<br>
</div>+ lxc_mainloop_close(&mon.descr);<br>
+ lxc_monitord_cleanup();<br>
+ ret = EXIT_SUCCESS;<br>
+ NOTICE("monitor exiting");<br>
<div class="im">+out:<br>
+ return ret;<br>
+}<br>
</div>diff --git a/src/lxc/lxccontainer.c b/src/lxc/lxccontainer.c<br>
index 1df6a98..53765b0 100644<br>
--- a/src/lxc/lxccontainer.c<br>
+++ b/src/lxc/lxccontainer.c<br>
@@ -32,6 +32,7 @@<br>
#include <sys/wait.h><br>
#include <errno.h><br>
#include <lxc/utils.h><br>
+#include <lxc/monitor.h><br>
<br>
lxc_log_define(lxc_container, lxc);<br>
<br>
@@ -370,6 +371,7 @@ static bool lxcapi_start(struct lxc_container *c, int useinit, char * const argv<br>
if (daemonize) {<br>
if (!lxc_container_get(c))<br>
return false;<br>
+ lxc_monitord_spawn(c->config_path);<br>
pid_t pid = fork();<br>
if (pid < 0) {<br>
lxc_container_put(c);<br>
@@ -560,7 +562,7 @@ static bool lxcapi_create(struct lxc_container *c, char *t, char *const argv[])<br>
}<br>
<br>
/* container is already created if we have a config and rootfs.path is accessible */<br>
- if (lxcapi_is_defined(c) && c->lxc_conf && c->lxc_conf->rootfs.path && access(c->lxc_conf->rootfs.path, F_OK) == 0)<br>
+ if (lxcapi_is_defined(c) && c->lxc_conf && c->lxc_conf->rootfs.path && access(c->lxc_conf->rootfs.path, F_OK) == 0)<br>
goto out;<br>
<br>
/* we're going to fork. but since we'll wait for our child, we<br>
@@ -826,7 +828,7 @@ static bool lxcapi_destroy(struct lxc_container *c)<br>
return false;<br>
<br>
/* container is already destroyed if we don't have a config and rootfs.path is not accessible */<br>
- if (!lxcapi_is_defined(c) && (!c->lxc_conf || !c->lxc_conf->rootfs.path || access(c->lxc_conf->rootfs.path, F_OK) != 0))<br>
+ if (!lxcapi_is_defined(c) && (!c->lxc_conf || !c->lxc_conf->rootfs.path || access(c->lxc_conf->rootfs.path, F_OK) != 0))<br>
return false;<br>
<br>
pid = fork();<br>
diff --git a/src/lxc/mainloop.c b/src/lxc/mainloop.c<br>
index 975215d..d9ab5d1 100644<br>
<div class="im">--- a/src/lxc/mainloop.c<br>
+++ b/src/lxc/mainloop.c<br>
@@ -38,7 +38,7 @@ struct mainloop_handler {<br>
<br>
#define MAX_EVENTS 10<br>
<br>
-int lxc_mainloop(struct lxc_epoll_descr *descr)<br>
</div>+int lxc_mainloop(struct lxc_epoll_descr *descr, int timeout_ms)<br>
<div class="im"> {<br>
int i, nfds;<br>
struct mainloop_handler *handler;<br>
@@ -46,7 +46,7 @@ int lxc_mainloop(struct lxc_epoll_descr *descr)<br>
<br>
for (;;) {<br>
<br>
- nfds = epoll_wait(descr->epfd, events, MAX_EVENTS, -1);<br>
+ nfds = epoll_wait(descr->epfd, events, MAX_EVENTS, timeout_ms);<br>
if (nfds < 0) {<br>
if (errno == EINTR)<br>
continue;<br>
</div>@@ -64,6 +64,9 @@ int lxc_mainloop(struct lxc_epoll_descr *descr)<br>
<div class="im"> return 0;<br>
}<br>
<br>
+ if (nfds == 0 && timeout_ms != 0)<br>
+ return 0;<br>
+<br>
if (lxc_list_empty(&descr->handlers))<br>
return 0;<br>
}<br>
</div>diff --git a/src/lxc/mainloop.h b/src/lxc/mainloop.h<br>
index 6b16242..ec87569 100644<br>
<div class="im">--- a/src/lxc/mainloop.h<br>
+++ b/src/lxc/mainloop.h<br>
@@ -21,6 +21,9 @@<br>
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA<br>
*/<br>
<br>
+#ifndef _mainloop_h<br>
+#define _mainloop_h<br>
+<br>
#include "list.h"<br>
<br>
struct lxc_epoll_descr {<br>
</div>@@ -31,7 +34,7 @@ struct lxc_epoll_descr {<br>
<div class="im"> typedef int (*lxc_mainloop_callback_t)(int fd, void *data,<br>
</div> struct lxc_epoll_descr *descr);<br>
<br>
-extern int lxc_mainloop(struct lxc_epoll_descr *descr);<br>
+extern int lxc_mainloop(struct lxc_epoll_descr *descr, int timeout_ms);<br>
<div class="im"><br>
extern int lxc_mainloop_add_handler(struct lxc_epoll_descr *descr, int fd,<br>
lxc_mainloop_callback_t callback,<br>
</div>@@ -42,3 +45,5 @@ extern int lxc_mainloop_del_handler(struct lxc_epoll_descr *descr, int fd);<br>
<div class="im"> extern int lxc_mainloop_open(struct lxc_epoll_descr *descr);<br>
<br>
extern int lxc_mainloop_close(struct lxc_epoll_descr *descr);<br>
+<br>
+#endif<br>
diff --git a/src/lxc/monitor.c b/src/lxc/monitor.c<br>
</div>index afdaf67..e7044f6 100644<br>
--- a/src/lxc/monitor.c<br>
+++ b/src/lxc/monitor.c<br>
@@ -5,6 +5,7 @@<br>
*<br>
* Authors:<br>
* Daniel Lezcano <daniel.lezcano at <a href="http://free.fr" target="_blank">free.fr</a>><br>
<div class="im">+ * Dwight Engen <<a href="mailto:dwight.engen@oracle.com">dwight.engen@oracle.com</a>><br>
</div> *<br>
<div class="im"> * This library is free software; you can redistribute it and/or<br>
</div><div class="im"> * modify it under the terms of the GNU Lesser General Public<br>
</div>@@ -20,6 +21,7 @@<br>
<div class="im"> * License along with this library; if not, write to the Free Software<br>
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA<br>
*/<br>
+<br>
#include <stdio.h><br>
#include <errno.h><br>
#include <unistd.h><br>
</div>@@ -30,7 +32,7 @@<br>
<div class="im"> #include <sys/stat.h><br>
#include <sys/param.h><br>
#include <sys/socket.h><br>
-#include <sys/un.h><br>
+#include <sys/wait.h><br>
#include <netinet/in.h><br>
#include <net/if.h><br>
<br>
</div>@@ -40,37 +42,36 @@<br>
#include <lxc/log.h><br>
#include <lxc/state.h><br>
#include <lxc/monitor.h><br>
+#include <lxc/utils.h><br>
<div class="im"><br>
lxc_log_define(lxc_monitor, lxc);<br>
<br>
-#ifndef UNIX_PATH_MAX<br>
-#define UNIX_PATH_MAX 108<br>
-#endif<br>
-<br>
-static void lxc_monitor_send(struct lxc_msg *msg, const char *lxcpath)<br>
+/* routines used by monitor publishers (containers) */<br>
+static void lxc_monitor_fifo_send(struct lxc_msg *msg, const char *lxcpath)<br>
{<br>
- int fd;<br>
</div><div class="im">- struct sockaddr_un addr = { .sun_family = AF_UNIX };<br>
- char *offset = &addr.sun_path[1];<br>
</div>- size_t ret, len;<br>
-<br>
<div class="im">- /*<br>
- * addr.sun_path is only 108 bytes.<br>
- * should we take a hash of lxcpath? a subset of it?<br>
</div><div class="im">- */<br>
- len = sizeof(addr.sun_path) - 1;<br>
- ret = snprintf(offset, len, "%s/lxc-monitor", lxcpath);<br>
- if (ret < 0 || ret >= len) {<br>
- ERROR("lxcpath too long to open monitor");<br>
</div><div class="im">+ int fd,ret;<br>
+ char fifo_path[PATH_MAX];<br>
</div>+<br>
+ BUILD_BUG_ON(sizeof(*msg) > PIPE_BUF); /* write not guaranteed atomic */<br>
+ ret = snprintf(fifo_path, sizeof(fifo_path), "%s/monitor-fifo", lxcpath);<br>
<div class="im">+ if (ret < 0 || ret >= sizeof(fifo_path)) {<br>
+ ERROR("lxcpath too long to open monitor fifo");<br>
return;<br>
}<br>
<br>
- fd = socket(PF_UNIX, SOCK_DGRAM, 0);<br>
- if (fd < 0)<br>
+ fd = open(fifo_path, O_WRONLY);<br>
+ if (fd < 0) {<br>
</div>+ /* it is normal for this open to fail when there is no monitor<br>
+ * running, so we don't log it<br>
+ */<br>
return;<br>
+ }<br>
<div class="im"><br>
- sendto(fd, msg, sizeof(*msg), 0,<br>
- (const struct sockaddr *)&addr, sizeof(addr));<br>
</div><div class="im">+ ret = write(fd, msg, sizeof(*msg));<br>
</div>+ if (ret != sizeof(*msg)) {<br>
+ SYSERROR("failed to write monitor fifo %s", fifo_path);<br>
+ return;<br>
+ }<br>
<br>
close(fd);<br>
}<br>
@@ -82,50 +83,74 @@ void lxc_monitor_send_state(const char *name, lxc_state_t state, const char *lxc<br>
<div class="im"> strncpy(<a href="http://msg.name" target="_blank">msg.name</a>, name, sizeof(<a href="http://msg.name" target="_blank">msg.name</a>));<br>
<a href="http://msg.name" target="_blank">msg.name</a>[sizeof(<a href="http://msg.name" target="_blank">msg.name</a>) - 1] = 0;<br>
<br>
- lxc_monitor_send(&msg, lxcpath);<br>
+ lxc_monitor_fifo_send(&msg, lxcpath);<br>
}<br>
<br>
-int lxc_monitor_open(const char *lxcpath)<br>
+<br>
</div><div class="im">+/* routines used by monitor subscribers (lxc-monitor) */<br>
+int lxc_monitor_close(int fd)<br>
{<br>
- struct sockaddr_un addr = { .sun_family = AF_UNIX };<br>
- char *offset = &addr.sun_path[1];<br>
- int fd;<br>
- size_t ret, len;<br>
</div>-<br>
<div class="im">- /*<br>
- * addr.sun_path is only 108 bytes.<br>
- * should we take a hash of lxcpath? a subset of it?<br>
</div>+ return close(fd);<br>
+}<br>
<div class="im">+<br>
+int lxc_monitor_sock_name(const char *lxcpath, struct sockaddr_un *addr) {<br>
</div>+ size_t len;<br>
+ int ret;<br>
<div class="im">+ char *sockname = &addr->sun_path[0]; // 1 for abstract<br>
+<br>
+ /* addr.sun_path is only 108 bytes.<br>
</div>+ * should we take a hash of lxcpath? a subset of it? ftok()? we need<br>
+ * to make sure it is unique.<br>
<div class="im"> */<br>
- len = sizeof(addr.sun_path) - 1;<br>
- ret = snprintf(offset, len, "%s/lxc-monitor", lxcpath);<br>
+ memset(addr, 0, sizeof(*addr));<br>
+ addr->sun_family = AF_UNIX;<br>
+ len = sizeof(addr->sun_path) - 1;<br>
+ ret = snprintf(sockname, len, "%s/monitor-sock", lxcpath);<br>
if (ret < 0 || ret >= len) {<br>
- ERROR("lxcpath too long to open monitor");<br>
+ ERROR("lxcpath too long for unix socket");<br>
return -1;<br>
}<br>
+ return 0;<br>
+}<br>
<br>
</div><div class="im">- fd = socket(PF_UNIX, SOCK_DGRAM, 0);<br>
</div><div class="im">+int lxc_monitor_open(const char *lxcpath)<br>
+{<br>
+ struct sockaddr_un addr;<br>
</div>+ int fd,ret;<br>
+ int retry,backoff_ms[] = {10, 50, 100};<br>
<div class="im">+<br>
+ if (lxc_monitor_sock_name(lxcpath, &addr) < 0)<br>
+ return -1;<br>
+<br>
</div><div class="im">+ fd = socket(PF_UNIX, SOCK_STREAM, 0);<br>
if (fd < 0) {<br>
ERROR("socket : %s", strerror(errno));<br>
return -1;<br>
}<br>
<br>
- if (bind(fd, (struct sockaddr *)&addr, sizeof(addr))) {<br>
- ERROR("bind : %s", strerror(errno));<br>
- close(fd);<br>
- return -1;<br>
</div>+ for (retry = 0; retry < sizeof(backoff_ms)/sizeof(backoff_ms[0]); retry++) {<br>
<div class="im">+ ret = connect(fd, (struct sockaddr *)&addr, sizeof(addr));<br>
</div>+ if (ret == 0 || errno != ECONNREFUSED)<br>
+ break;<br>
+ ERROR("connect : backing off %d", backoff_ms[retry]);<br>
+ usleep(backoff_ms[retry] * 1000);<br>
}<br>
<br>
+ if (ret < 0) {<br>
<div class="im">+ ERROR("connect : %s", strerror(errno));<br>
+ goto err1;<br>
</div>+ }<br>
<div class="im"> return fd;<br>
+err1:<br>
+ close(fd);<br>
+ return ret;<br>
}<br>
<br>
-/* timeout of 0 means return immediately; -1 means wait forever */<br>
-int lxc_monitor_read_timeout(int fd, struct lxc_msg *msg, int timeout)<br>
+int lxc_monitor_read_timeout(int fd, struct lxc_msg *msglxc, int timeout)<br>
{<br>
</div>- struct sockaddr_un from;<br>
- socklen_t len = sizeof(from);<br>
- int ret;<br>
fd_set rfds;<br>
struct timeval tv;<br>
+ int ret;<br>
<br>
if (timeout != -1) {<br>
FD_ZERO(&rfds);<br>
@@ -141,13 +166,12 @@ int lxc_monitor_read_timeout(int fd, struct lxc_msg *msg, int timeout)<br>
<div class="im"> return -2; // timed out<br>
}<br>
<br>
</div><div class="im">- ret = recvfrom(fd, msg, sizeof(*msg), 0,<br>
</div>- (struct sockaddr *)&from, &len);<br>
<div class="im">- if (ret < 0) {<br>
- SYSERROR("failed to receive state");<br>
</div>+ ret = recv(fd, msglxc, sizeof(*msglxc), 0);<br>
<div class="im">+ if (ret <= 0) {<br>
</div>+ SYSERROR("client failed to recv (monitord died?) %s",<br>
+ strerror(errno));<br>
<div class="im"> return -1;<br>
}<br>
-<br>
return ret;<br>
}<br>
<br>
</div>@@ -156,7 +180,69 @@ int lxc_monitor_read(int fd, struct lxc_msg *msg)<br>
<div class="im"> return lxc_monitor_read_timeout(fd, msg, -1);<br>
}<br>
<br>
-int lxc_monitor_close(int fd)<br>
+<br>
+<br>
</div>+/* used to spawn a monitord either on startup of a daemon container, or when<br>
<div><div class="h5">+ * lxc-monitor starts<br>
+ */<br>
+int lxc_monitord_spawn(const char *lxcpath)<br>
{<br>
- return close(fd);<br>
+ pid_t pid1,pid2;<br>
+ int pipefd[2];<br>
+ char pipefd_str[10];<br>
+<br>
+ char * const args[] = {<br>
+ "/usr/bin/lxc-monitord",<br>
+ (char *)lxcpath,<br>
+ pipefd_str,<br>
+ NULL,<br>
+ };<br>
+<br>
+ /* double fork to avoid zombies when monitord exits */<br>
+ pid1 = fork();<br>
+ if (pid1 < 0) {<br>
+ SYSERROR("failed to fork");<br>
+ return -1;<br>
+ }<br>
+<br>
+ if (pid1) {<br>
+ waitpid(pid1, NULL, 0);<br>
+ return 0;<br>
+ }<br>
+<br>
+ if (pipe(pipefd) < 0) {<br>
+ SYSERROR("failed to create pipe");<br>
+ exit(EXIT_FAILURE);<br>
+ }<br>
+<br>
+ pid2 = fork();<br>
+ if (pid2) {<br>
+ char c;<br>
+ /* wait for daemon to create socket */<br>
+ close(pipefd[1]);<br>
</div></div>+ /* sync with child, we're ignoring the return from read<br>
+ * because regardless if it works or not, either way we've<br>
+ * synced with the child process. the if-empty-statement<br>
+ * construct is to quiet the warn-unused-result warning.<br>
+ */<br>
+ if (read(pipefd[0], &c, 1)) ;<br>
<div class="im">+ close(pipefd[0]);<br>
+ exit(EXIT_SUCCESS);<br>
+ }<br>
+<br>
+ umask(0);<br>
+ if (setsid() < 0) {<br>
+ SYSERROR("failed to setsid");<br>
+ exit(EXIT_FAILURE);<br>
+ }<br>
</div>+ close(0);<br>
+ close(1);<br>
+ close(2);<br>
+ open("/dev/null", O_RDONLY);<br>
+ open("/dev/null", O_RDWR);<br>
+ open("/dev/null", O_RDWR);<br>
<div class="im">+ close(pipefd[0]);<br>
+ sprintf(pipefd_str, "%d", pipefd[1]);<br>
+ execvp(args[0], args);<br>
+ exit(EXIT_FAILURE);<br>
}<br>
diff --git a/src/lxc/monitor.h b/src/lxc/monitor.h<br>
</div>index 8bef4c7..cd59ee8 100644<br>
--- a/src/lxc/monitor.h<br>
+++ b/src/lxc/monitor.h<br>
@@ -24,6 +24,9 @@<br>
<div class="im"> #define __monitor_h<br>
<br>
#include <sys/param.h><br>
+#include <sys/un.h><br>
+<br>
+#include <lxc/conf.h><br>
<br>
</div> typedef enum {<br>
lxc_msg_state,<br>
@@ -32,11 +35,14 @@ typedef enum {<br>
<div class="im"><br>
struct lxc_msg {<br>
lxc_msg_type_t type;<br>
- char name[MAXPATHLEN];<br>
+ char name[NAME_MAX+1];<br>
int value;<br>
};<br>
<br>
-void lxc_monitor_send_state(const char *name, lxc_state_t state,<br>
+extern int lxc_monitor_open(const char *lxcpath);<br>
+extern int lxc_monitor_sock_name(const char *lxcpath, struct sockaddr_un *addr);<br>
+extern void lxc_monitor_send_state(const char *name, lxc_state_t state,<br>
const char *lxcpath);<br>
+extern int lxc_monitord_spawn(const char *lxcpath);<br>
<br>
#endif<br>
</div>diff --git a/src/lxc/start.c b/src/lxc/start.c<br>
index 0a0cc40..fd96d4f 100644<br>
--- a/src/lxc/start.c<br>
+++ b/src/lxc/start.c<br>
@@ -390,7 +390,7 @@ int lxc_poll(const char *name, struct lxc_handler *handler)<br>
#endif<br>
}<br>
<br>
- return lxc_mainloop(&descr);<br>
+ return lxc_mainloop(&descr, -1);<br>
<br>
out_mainloop_open:<br>
lxc_mainloop_close(&descr);<br>
@@ -808,7 +808,7 @@ int lxc_spawn(struct lxc_handler *handler)<br>
<div class="im"> /* TODO - pass lxc.cgroup.dir (or user's pam cgroup) in for first argument */<br>
if ((handler->cgroup = lxc_cgroup_path_create(NULL, name)) == NULL)<br>
goto out_delete_net;<br>
-<br>
+<br>
if (lxc_cgroup_enter(handler->cgroup, handler->pid) < 0)<br>
goto out_delete_net;<br>
<br>
</div>diff --git a/src/lxc/utils.h b/src/lxc/utils.h<br>
index 8954503..8e6a748 100644<br>
--- a/src/lxc/utils.h<br>
+++ b/src/lxc/utils.h<br>
@@ -32,4 +32,30 @@ extern int mkdir_p(const char *dir, mode_t mode);<br>
*/<br>
extern const char *default_lxc_path(void);<br>
<br>
+/**<br>
+ * BUILD_BUG_ON - break compile if a condition is true.<br>
+ * @condition: the condition which the compiler should know is false.<br>
+ *<br>
+ * If you have some code which relies on certain constants being equal, or<br>
+ * other compile-time-evaluated condition, you should use BUILD_BUG_ON to<br>
+ * detect if someone changes it.<br>
+ *<br>
+ * The implementation uses gcc's reluctance to create a negative array, but<br>
+ * gcc (as of 4.4) only emits that error for obvious cases (eg. not arguments<br>
+ * to inline functions). So as a fallback we use the optimizer; if it can't<br>
+ * prove the condition is false, it will cause a link error on the undefined<br>
+ * "__build_bug_on_failed". This error message can be harder to track down<br>
+ * though, hence the two different methods.<br>
+ */<br>
+#ifndef __OPTIMIZE__<br>
+#define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)]))<br>
+#else<br>
+extern int __build_bug_on_failed;<br>
+#define BUILD_BUG_ON(condition) \<br>
+ do { \<br>
+ ((void)sizeof(char[1 - 2*!!(condition)])); \<br>
+ if (condition) __build_bug_on_failed = 1; \<br>
+ } while(0)<br>
+#endif<br>
+<br>
#endif<br>
<span class="HOEnZb"><font color="#888888">--<br>
1.8.1.4<br>
<br>
</font></span></blockquote></div><br clear="all"><div style>Cheers,</div>-- <br>S.Çağlar Onur <<a href="mailto:caglar@10ur.org">caglar@10ur.org</a>>
</div></div></div>