[lxc-devel] [RFC PATCH] allow multiple monitor clients

S.Çağlar Onur caglar at 10ur.org
Thu Apr 18 19:02:15 UTC 2013


Hi Dwight,

Couple of quick comments, I needed following to make it compile under
ubuntu 12.10 otherwise gcc whines about ignoring return values of those
calls.

diff --git a/src/lxc/lxc_monitord.c b/src/lxc/lxc_monitord.c
index 1d20d67..9d83673 100644
--- a/src/lxc/lxc_monitord.c
+++ b/src/lxc/lxc_monitord.c
@@ -309,7 +309,11 @@ int main(int argc, char *argv[])
        }

        /* sync with parent */
-       write(pipefd, "S", 1);
+       if (write(pipefd, "S", 1) < 0) {
+               ERROR("write failed");
+               goto out;
+       }
+
        close(pipefd);

        if (lxc_monitord_mainloop_add(&mon)) {
diff --git a/src/lxc/monitor.c b/src/lxc/monitor.c
index 0438fbc..803f98f 100644
--- a/src/lxc/monitor.c
+++ b/src/lxc/monitor.c
@@ -158,6 +158,7 @@ int lxc_monitord_spawn(const char *lxcpath)
        pid_t pid1,pid2;
        int pipefd[2];
        char pipefd_str[10];
+       FILE *fd;

        char * const args[] = {
                "/usr/bin/lxc-monitord",
@@ -188,7 +189,10 @@ int lxc_monitord_spawn(const char *lxcpath)
                char c;
                /* wait for daemon to create socket */
                close(pipefd[1]);
-               read(pipefd[0], &c, 1);
+               if (read(pipefd[0], &c, 1) < 0) {
+                       SYSERROR("read failed");
+                       exit(EXIT_FAILURE);
+               }
                close(pipefd[0]);
                exit(EXIT_SUCCESS);
        }
@@ -198,9 +202,21 @@ int lxc_monitord_spawn(const char *lxcpath)
                SYSERROR("failed to setsid");
                exit(EXIT_FAILURE);
        }
-       freopen("/dev/null", "r", stdin);
-       freopen("/dev/null", "w", stdout);
-       freopen("/dev/null", "w", stderr);
+       fd = freopen("/dev/null", "r", stdin);
+       if (!fd) {
+               SYSERROR("freopen failed");
+               exit(EXIT_FAILURE);
+       }
+       fd = freopen("/dev/null", "w", stdout);
+       if (!fd) {
+               SYSERROR("freopen failed");
+               exit(EXIT_FAILURE);
+       }
+       fd = freopen("/dev/null", "w", stderr);
+       if (!fd) {
+               SYSERROR("freopen failed");
+               exit(EXIT_FAILURE);
+       }
        close(pipefd[0]);
        sprintf(pipefd_str, "%d", pipefd[1]);
        execvp(args[0], args);

and time to time (I think that only happens if monitord wasn't running
initially) I get following while starting 10 containers in parallel

[caglar at qgq:~/Project/lxc/examples] sudo ./concurrent_start
Starting the container (2)...
Starting the container (6)...
Starting the container (1)...
lxc_container: Invalid argument - client failed to receive lxc_msg
(monitord died?)
Starting the container (2) failed...
Starting the container (0)...
Starting the container (3)...
Starting the container (5)...
Starting the container (7)...
Starting the container (8)...
Starting the container (9)...
Starting the container (4)...

Best,


On Thu, Apr 18, 2013 at 2:47 PM, Dwight Engen <dwight.engen at oracle.com>wrote:

> On Thu, 18 Apr 2013 14:22:08 -0400
> S.Çağlar Onur <caglar at 10ur.org> wrote:
>
> > Hey Dwight,
> >
> > This sounds great, I'll give it a try ASAP. In the meantime ,if you
> > want to test it yourself, you can grab go bindings from github via
> > "go get github.com/caglar10ur/lxc" but please keep in your mind that
> > you need my other two patches to start/stop work reliably [1]/[2].
> > https://github.com/caglar10ur/lxc/tree/master/examples directory
> > contains concurrent versions of create/start/stop/shutdown to test.
> >
> > [1]
> >
> https://github.com/caglar10ur/lxc-upstream/commit/db60b9c38083e8ed553f3cbdd96ad410bd5eb7ed.patch
> > [2]
> >
> https://github.com/caglar10ur/lxc-upstream/commit/97d4daf4cc5016bcb01c72e5cf9fe3189e9df4f5.patch
> >
> > Cheers,
>
> Thanks Caglar, I may give them a try as I've just built go the other
> day to try out docker.
>
> > On Thu, Apr 18, 2013 at 2:09 PM, Dwight Engen
> > <dwight.engen at oracle.com>wrote:
> >
> > > After the recent discussions on the lxc monitor, I took a closer
> > > look at it. It seems like Serge was resigned to having some sort of
> > > daemon as having multicast unix domain supported by the kernel is
> > > not going to happen any time soon, and doing it over loopback has
> > > some complications too.
> > >
> > > Initially I thought it would be nice to just have lxc-start "be the
> > > daemon" as Stéphane had suggested. I tried this out, putting the fds
> > > for the consumer clients (lxc-monitors) in lxc_mainloop , but this
> > > is problematic for the scenario where lxc-monitor runs first and we
> > > miss some startup states.
> > >
> > > So here is what I'm proposing: when lxc-monitor starts, it attempts
> > > to start lxc-monitord. lxc-monitord creates a fifo and a socket on
> > > lxcpath.
> > >
> > > Producers (containers) wishing to send messages open the fifo and
> > > write their lxc_msg's on it. The reason I used a fifo is so that we
> > > get permission checks on the opening of it. Right now it is created
> > > 0600. This fixes the problem we have today where anyone can open
> > > the unix socket and send state changes in.
> > >
> > > Consumers (lxc-monitor) connect to the unix socket and each gets a
> > > copy of the messages. lxc-monitord can do a PEERCRED to ensure the
> > > client should be able to get these messages, this is one benefit to
> > > using unix domain. I don't think it would be difficult to write a
> > > lxc-monitor like client that would bridge the messages onto d-bus.
> > >
> > > When lxc-monitord hasn't had a client for 30 seconds it will exit.
> > >
> > > Attached is a proof of concept patch for your comments. It works for
> > > the common cases that I've tried but I know it needs some cleanup
> > > and testing if we like the approach. The monitor.c part is hard to
> > > read because that file is mostly rewritten.
> > >
> > > I need to test S.Çağlar Onur's scenario of parallel startup with his
> > > test script. We need to decide if we want to monitor per lxcpath or
> > > per container, today it is per path and I think that being able to
> > > do lxc-monitor * would be very useful.
> > >
> > > --
> > >
> > > Signed-off-by: Dwight Engen <dwight.engen at oracle.com>
> > > ---
> > >  src/lxc/Makefile.am    |   2 +
> > >  src/lxc/af_unix.c      |   5 +-
> > >  src/lxc/lxc_monitor.c  |   2 +
> > >  src/lxc/lxc_monitord.c | 334
> > > +++++++++++++++++++++++++++++++++++++++++++++++++
> > >  src/lxc/mainloop.c     |  11 +-
> > >  src/lxc/mainloop.h     |   7 ++
> > >  src/lxc/monitor.c      | 180 ++++++++++++++++----------
> > >  src/lxc/monitor.h      |  11 +-
> > >  src/lxc/start.c        |   7 +-
> > >  9 files changed, 484 insertions(+), 75 deletions(-)
> > >  create mode 100644 src/lxc/lxc_monitord.c
> > >
> > > diff --git a/src/lxc/Makefile.am b/src/lxc/Makefile.am
> > > index ebeca466..1fa0fa8 100644
> > > --- a/src/lxc/Makefile.am
> > > +++ b/src/lxc/Makefile.am
> > > @@ -150,6 +150,7 @@ bin_PROGRAMS = \
> > >         lxc-start \
> > >         lxc-execute \
> > >         lxc-monitor \
> > > +       lxc-monitord \
> > >         lxc-wait \
> > >         lxc-console \
> > >         lxc-freeze \
> > > @@ -181,6 +182,7 @@ lxc_freeze_SOURCES = lxc_freeze.c
> > >  lxc_info_SOURCES = lxc_info.c
> > >  lxc_init_SOURCES = lxc_init.c
> > >  lxc_monitor_SOURCES = lxc_monitor.c
> > > +lxc_monitord_SOURCES = lxc_monitord.c
> > >  lxc_restart_SOURCES = lxc_restart.c
> > >  lxc_start_SOURCES = lxc_start.c
> > >  lxc_stop_SOURCES = lxc_stop.c
> > > diff --git a/src/lxc/af_unix.c b/src/lxc/af_unix.c
> > > index eff13d4..ceef340 100644
> > > --- a/src/lxc/af_unix.c
> > > +++ b/src/lxc/af_unix.c
> > > @@ -52,6 +52,7 @@ int lxc_af_unix_open(const char *path, int type,
> > > int flags)
> > >
> > >         addr.sun_family = AF_UNIX;
> > >         /* copy entire buffer in case of abstract socket */
> > > +       /* FIXME: this is unsafe if path is too long... */
> > >         memcpy(addr.sun_path, path,
> > >                path[0]?strlen(path):sizeof(addr.sun_path));
> > >
> > > @@ -61,7 +62,7 @@ int lxc_af_unix_open(const char *path, int type,
> > > int flags)
> > >                 errno = tmp;
> > >                 return -1;
> > >         }
> > > -
> > > +
> > >         if (type == SOCK_STREAM && listen(fd, 100)) {
> > >                 int tmp = errno;
> > >                 close(fd);
> > > @@ -76,7 +77,7 @@ int lxc_af_unix_close(int fd)
> > >  {
> > >         struct sockaddr_un addr;
> > >         socklen_t addrlen = sizeof(addr);
> > > -
> > > +
> > >         if (!getsockname(fd, (struct sockaddr *)&addr, &addrlen) &&
> > >             addr.sun_path[0])
> > >                 unlink(addr.sun_path);
> > > diff --git a/src/lxc/lxc_monitor.c b/src/lxc/lxc_monitor.c
> > > index 8c15869..0ca829f 100644
> > > --- a/src/lxc/lxc_monitor.c
> > > +++ b/src/lxc/lxc_monitor.c
> > > @@ -87,6 +87,8 @@ int main(int argc, char *argv[])
> > >                 return -1;
> > >         }
> > >
> > > +       lxc_monitord_spawn(my_args.lxcpath);
> > > +
> > >         fd = lxc_monitor_open(my_args.lxcpath);
> > >         if (fd < 0)
> > >                 return -1;
> > > diff --git a/src/lxc/lxc_monitord.c b/src/lxc/lxc_monitord.c
> > > new file mode 100644
> > > index 0000000..3e9c708
> > > --- /dev/null
> > > +++ b/src/lxc/lxc_monitord.c
> > > @@ -0,0 +1,334 @@
> > > +/*
> > > + * lxc: linux Container library
> > > + *
> > > + * Copyright © 2012 Oracle.
> > > + *
> > > + * Authors:
> > > + * Dwight Engen <dwight.engen at oracle.com>
> > > + *
> > > + * This library is free software; you can redistribute it and/or
> > > + * modify it under the terms of the GNU Lesser General Public
> > > + * License as published by the Free Software Foundation; either
> > > + * version 2.1 of the License, or (at your option) any later
> > > version.
> > > + *
> > > + * This library is distributed in the hope that it will be useful,
> > > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > > GNU
> > > + * Lesser General Public License for more details.
> > > + *
> > > + * You should have received a copy of the GNU Lesser General Public
> > > + * License along with this library; if not, write to the Free
> > > Software
> > > + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> > > 02111-1307 USA
> > > + */
> > > +
> > > +#define _GNU_SOURCE
> > > +#include <stdio.h>
> > > +#include <signal.h>
> > > +#include <errno.h>
> > > +#include <unistd.h>
> > > +#include <string.h>
> > > +#include <stdlib.h>
> > > +#include <fcntl.h>
> > > +#include <sys/types.h>
> > > +#include <sys/stat.h>
> > > +#include <sys/param.h>
> > > +#include <sys/socket.h>
> > > +#include <sys/un.h>
> > > +#include <netinet/in.h>
> > > +#include <net/if.h>
> > > +
> > > +#include <lxc/af_unix.h>
> > > +#include <lxc/log.h>
> > > +#include <lxc/mainloop.h>
> > > +#include <lxc/monitor.h>
> > > +#include <lxc/utils.h>
> > > +
> > > +lxc_log_define(lxc_monitord, lxc);
> > > +
> > > +/*
> > > + * Defines the structure to store the monitor information
> > > + * @fifofd       : the file descriptor for publishers (containers)
> > > to write state
> > > + * @listenfd     : the file descriptor for subscribers
> > > (lxc-monitors) to connect
> > > + * @clientfds    : accepted client file descriptors
> > > + * @clientfds_cnt: the count of valid fds in clientfds
> > > + */
> > > +struct lxc_monitor {
> > > +       int fifofd;
> > > +       int listenfd;
> > > +       int clientfds[1024];
> > > +       int clientfds_cnt;
> > > +       struct lxc_epoll_descr descr;
> > > +};
> > > +
> > > +static int lxc_monitord_fifo_create(struct lxc_monitor *mon, const
> > > char *lxcpath)
> > > +{
> > > +       char fifo_path[PATH_MAX];
> > > +       int ret;
> > > +
> > > +       ret = snprintf(fifo_path, sizeof(fifo_path), "%s/monitor",
> > > lxcpath);
> > > +       if (ret < 0 || ret >= sizeof(fifo_path)) {
> > > +               ERROR("lxcpath too long to monitor fifo");
> > > +               return -1;
> > > +       }
> > > +
> > > +       ret = mknod(fifo_path, S_IFIFO|S_IRUSR|S_IWUSR, 0);
> > > +       if (ret < 0) {
> > > +               INFO("existing monitor fifo at %s, monitor already
> > > running?", fifo_path);
> > > +               return -1;
> > > +       }
> > > +       mon->fifofd = open(fifo_path, O_RDWR);
> > > +       if (mon->fifofd < 0) {
> > > +               ERROR("failed to open monitor fifo");
> > > +               return -1;
> > > +       }
> > > +       return 0;
> > > +}
> > > +
> > > +static int lxc_monitord_fifo_delete(const char *lxcpath)
> > > +{
> > > +       char fifo_path[PATH_MAX];
> > > +       int ret;
> > > +
> > > +       ret = snprintf(fifo_path, sizeof(fifo_path), "%s/monitor",
> > > lxcpath);
> > > +       if (ret < 0 || ret >= sizeof(fifo_path)) {
> > > +               ERROR("lxcpath too long to monitor fifo");
> > > +               return -1;
> > > +       }
> > > +       unlink(fifo_path);
> > > +       return 0;
> > > +}
> > > +
> > > +static void lxc_monitord_sockfd_remove(struct lxc_monitor *mon,
> > > int fd) {
> > > +       int i;
> > > +
> > > +       lxc_mainloop_del_handler(&mon->descr, fd);
> > > +       close(fd);
> > > +
> > > +       for (i = 0; i < mon->clientfds_cnt; i++) {
> > > +               if (mon->clientfds[i] == fd)
> > > +                       break;
> > > +       }
> > > +       if (i >= mon->clientfds_cnt) {
> > > +               CRIT("fd not found");
> > > +               exit(EXIT_FAILURE);
> > > +       }
> > > +       DEBUG("MON: removing client fd:%d memmoving %p %p %lu
> > > bytes", fd,
> > > +               &mon->clientfds[mon->clientfds_cnt],
> > > +               &mon->clientfds[i],
> > > +               (&mon->clientfds[mon->clientfds_cnt-1] -
> > > &mon->clientfds[i]) * sizeof(int));
> > > +       memmove(&mon->clientfds[i], &mon->clientfds[i+1],
> > > +               (&mon->clientfds[mon->clientfds_cnt-1] -
> > > &mon->clientfds[i]) * sizeof(int));
> > > +       mon->clientfds_cnt--;
> > > +}
> > > +
> > > +static int lxc_monitord_sock_handler(int fd, void *data,
> > > +                                    struct lxc_epoll_descr *descr)
> > > +{
> > > +       struct lxc_monitor *mon = data;
> > > +       lxc_monitord_sockfd_remove(mon, fd);
> > > +       return 0;
> > > +}
> > > +
> > > +static int lxc_monitord_sock_accept(int fd, void *data,
> > > +                                   struct lxc_epoll_descr *descr)
> > > +{
> > > +       int ret,clientfd;
> > > +       struct lxc_monitor *mon = data;
> > > +
> > > +       ret = -1;
> > > +       clientfd = accept(fd, NULL, 0);
> > > +       if (clientfd < 0) {
> > > +               SYSERROR("failed to accept connection");
> > > +               goto out;
> > > +       }
> > > +
> > > +       if (fcntl(clientfd, F_SETFD, FD_CLOEXEC)) {
> > > +               SYSERROR("failed to set close-on-exec on incoming
> > > connection");
> > > +               goto err1;
> > > +       }
> > > +
> > > +       /* FIXME: check the connected clients credentials here,
> > > drop if we don't
> > > +        * like them.
> > > +       int on = 1;
> > > +       if (setsockopt(clientfd, SOL_SOCKET, SO_PASSCRED, &on,
> > > sizeof(on))) {
> > > +               ERROR("failed to enable credential on socket");
> > > +               goto err1;
> > > +       }
> > > +        */
> > > +
> > > +       ret = lxc_mainloop_add_handler(&mon->descr, clientfd,
> > > +                                      lxc_monitord_sock_handler,
> > > mon);
> > > +       if (ret) {
> > > +               ERROR("failed to add socket handler");
> > > +               goto err1;
> > > +       }
> > > +       mon->clientfds[mon->clientfds_cnt++] = clientfd;
> > > +       INFO("accepted new client fd:%d clientfds_cnt:%d", clientfd,
> > > mon->clientfds_cnt);
> > > +       goto out;
> > > +
> > > +err1:
> > > +       close(clientfd);
> > > +out:
> > > +       return ret;
> > > +}
> > > +
> > > +static int lxc_monitord_sock_create(struct lxc_monitor *mon, const
> > > char *lxcpath)
> > > +{
> > > +       struct sockaddr_un addr;
> > > +       int fd;
> > > +
> > > +       if (lxc_monitor_sock_name(lxcpath, &addr) < 0)
> > > +               return -1;
> > > +
> > > +       fd = lxc_af_unix_open(addr.sun_path, SOCK_STREAM, O_TRUNC);
> > > +       if (fd < 0) {
> > > +               ERROR("failed to open unix socket : %s",
> > > strerror(errno));
> > > +               return -1;
> > > +       }
> > > +
> > > +       mon->listenfd = fd;
> > > +       return 0;
> > > +}
> > > +
> > > +static int lxc_monitord_sock_delete(const char *lxcpath)
> > > +{
> > > +       struct sockaddr_un addr;
> > > +
> > > +       if (lxc_monitor_sock_name(lxcpath, &addr) < 0)
> > > +               return -1;
> > > +       if (addr.sun_path[0])
> > > +               unlink(addr.sun_path);
> > > +       return 0;
> > > +}
> > > +
> > > +static int lxc_monitord_create(struct lxc_monitor *mon, const char
> > > *lxcpath)
> > > +{
> > > +       int ret;
> > > +
> > > +       ret = lxc_monitord_fifo_create(mon, lxcpath);
> > > +       if (ret < 0)
> > > +               return ret;
> > > +
> > > +       ret = lxc_monitord_sock_create(mon, lxcpath);
> > > +       return ret;
> > > +}
> > > +
> > > +static void lxc_monitord_delete(struct lxc_monitor *mon, const char
> > > *lxcpath)
> > > +{
> > > +       int i;
> > > +
> > > +       lxc_mainloop_del_handler(&mon->descr, mon->listenfd);
> > > +       close(mon->listenfd);
> > > +       lxc_monitord_sock_delete(lxcpath);
> > > +
> > > +       lxc_mainloop_del_handler(&mon->descr, mon->fifofd);
> > > +       close(mon->fifofd);
> > > +       lxc_monitord_fifo_delete(lxcpath);
> > > +
> > > +       for (i = 0; i < mon->clientfds_cnt; i++) {
> > > +               lxc_mainloop_del_handler(&mon->descr,
> > > mon->clientfds[i]);
> > > +               close(mon->clientfds[i]);
> > > +       }
> > > +       mon->clientfds_cnt = 0;
> > > +}
> > > +
> > > +static int lxc_monitord_fifo_handler(int fd, void *data,
> > > +                                    struct lxc_epoll_descr *descr)
> > > +{
> > > +       int ret,i;
> > > +       struct lxc_msg msglxc;
> > > +       struct lxc_monitor *mon = data;
> > > +
> > > +       ret = read(fd, &msglxc, sizeof(msglxc));
> > > +       if (ret != sizeof(msglxc)) {
> > > +               SYSERROR("read fifo failed : %s", strerror(errno));
> > > +               return 1;
> > > +       }
> > > +
> > > +       for (i = 0; i < mon->clientfds_cnt; i++) {
> > > +               ret = write(mon->clientfds[i], &msglxc,
> > > sizeof(msglxc));
> > > +               if (ret < 0) {
> > > +                       ERROR("write failed to client sock,
> > > removing");
> > > +                       lxc_monitord_sockfd_remove(mon, i);
> > > +               }
> > > +       }
> > > +
> > > +       return 0;
> > > +}
> > > +
> > > +static int lxc_monitord_mainloop_add(struct lxc_monitor *mon)
> > > +{
> > > +       int ret;
> > > +
> > > +       ret = lxc_mainloop_add_handler(&mon->descr, mon->fifofd,
> > > +                                      lxc_monitord_fifo_handler,
> > > mon);
> > > +       if (ret < 0) {
> > > +               ERROR("failed to add to mainloop monitor handler for
> > > fifo");
> > > +               return -1;
> > > +       }
> > > +
> > > +       ret = lxc_mainloop_add_handler(&mon->descr, mon->listenfd,
> > > +                                      lxc_monitord_sock_accept,
> > > mon);
> > > +       if (ret < 0) {
> > > +               ERROR("failed to add to mainloop monitor handler for
> > > listen socket");
> > > +               return -1;
> > > +       }
> > > +
> > > +       return 0;
> > > +}
> > > +
> > > +static void lxc_monitord_sig_handler(int sig)
> > > +{
> > > +       INFO("caught signal %d", sig);
> > > +}
> > > +
> > > +int main(int argc, char *argv[])
> > > +{
> > > +       struct lxc_monitor mon;
> > > +       int ret,pipefd;
> > > +       char *lxcpath = argv[1];
> > > +       char logpath[PATH_MAX];
> > > +
> > > +       signal(SIGTERM, lxc_monitord_sig_handler);
> > > +       snprintf(logpath, sizeof(logpath), "%s/lxc-monitord.log",
> > > lxcpath);
> > > +       ret = lxc_log_init(NULL, logpath, "INFO", "lxc-monitord",
> > > 0);
> > > +       if (ret)
> > > +               return ret;
> > > +
> > > +       pipefd = atoi(argv[2]);
> > > +
> > > +       ret = EXIT_FAILURE;
> > > +       memset(&mon, 0, sizeof(mon));
> > > +       if (lxc_mainloop_open(&mon.descr)) {
> > > +               ERROR("failed to create mainloop");
> > > +               goto out;
> > > +       }
> > > +
> > > +       if (lxc_monitord_create(&mon, lxcpath)) {
> > > +               goto out;
> > > +       }
> > > +
> > > +       /* sync with parent */
> > > +       write(pipefd, "S", 1);
> > > +       close(pipefd);
> > > +
> > > +       if (lxc_monitord_mainloop_add(&mon)) {
> > > +               ERROR("failed to add mainloop handlers");
> > > +               goto out;
> > > +       }
> > > +
> > > +       NOTICE("monitoring lxcpath %s", lxcpath);
> > > +       for(;;) {
> > > +               ret = lxc_mainloop_timeout(&mon.descr, 1000 * 30);
> > > +               if (mon.clientfds_cnt <= 0)
> > > +               {
> > > +                       NOTICE("no clients for 30 seconds,
> > > exiting");
> > > +                       break;
> > > +               }
> > > +       }
> > > +
> > > +       lxc_monitord_delete(&mon, lxcpath);
> > > +       lxc_mainloop_close(&mon.descr);
> > > +out:
> > > +       return ret;
> > > +}
> > > diff --git a/src/lxc/mainloop.c b/src/lxc/mainloop.c
> > > index 975215d..4805dcd 100644
> > > --- a/src/lxc/mainloop.c
> > > +++ b/src/lxc/mainloop.c
> > > @@ -38,7 +38,7 @@ struct mainloop_handler {
> > >
> > >  #define MAX_EVENTS 10
> > >
> > > -int lxc_mainloop(struct lxc_epoll_descr *descr)
> > > +int lxc_mainloop_timeout(struct lxc_epoll_descr *descr, int
> > > timeout_ms) {
> > >         int i, nfds;
> > >         struct mainloop_handler *handler;
> > > @@ -46,7 +46,7 @@ int lxc_mainloop(struct lxc_epoll_descr *descr)
> > >
> > >         for (;;) {
> > >
> > > -               nfds = epoll_wait(descr->epfd, events, MAX_EVENTS,
> > > -1);
> > > +               nfds = epoll_wait(descr->epfd, events, MAX_EVENTS,
> > > timeout_ms);
> > >                 if (nfds < 0) {
> > >                         if (errno == EINTR)
> > >                                 continue;
> > > @@ -64,11 +64,18 @@ int lxc_mainloop(struct lxc_epoll_descr *descr)
> > >                                 return 0;
> > >                 }
> > >
> > > +               if (nfds == 0 && timeout_ms != 0)
> > > +                       return 0;
> > > +
> > >                 if (lxc_list_empty(&descr->handlers))
> > >                         return 0;
> > >         }
> > >  }
> > >
> > > +int lxc_mainloop(struct lxc_epoll_descr *descr) {
> > > +       return lxc_mainloop_timeout(descr, -1);
> > > +}
> > > +
> > >  int lxc_mainloop_add_handler(struct lxc_epoll_descr *descr, int fd,
> > >                              lxc_mainloop_callback_t callback, void
> > > *data) {
> > > diff --git a/src/lxc/mainloop.h b/src/lxc/mainloop.h
> > > index 6b16242..c79c4fd 100644
> > > --- a/src/lxc/mainloop.h
> > > +++ b/src/lxc/mainloop.h
> > > @@ -21,6 +21,9 @@
> > >   * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> > > 02111-1307 USA */
> > >
> > > +#ifndef _mainloop_h
> > > +#define _mainloop_h
> > > +
> > >  #include "list.h"
> > >
> > >  struct lxc_epoll_descr {
> > > @@ -33,6 +36,8 @@ typedef int (*lxc_mainloop_callback_t)(int fd,
> > > void *data,
> > >
> > >  extern int lxc_mainloop(struct lxc_epoll_descr *descr);
> > >
> > > +extern int lxc_mainloop_timeout(struct lxc_epoll_descr *descr, int
> > > timeout_ms);
> > > +
> > >  extern int lxc_mainloop_add_handler(struct lxc_epoll_descr *descr,
> > > int fd, lxc_mainloop_callback_t callback,
> > >                                     void *data);
> > > @@ -42,3 +47,5 @@ extern int lxc_mainloop_del_handler(struct
> > > lxc_epoll_descr *descr, int fd);
> > >  extern int lxc_mainloop_open(struct lxc_epoll_descr *descr);
> > >
> > >  extern int lxc_mainloop_close(struct lxc_epoll_descr *descr);
> > > +
> > > +#endif
> > > diff --git a/src/lxc/monitor.c b/src/lxc/monitor.c
> > > index afdaf67..0438fbc 100644
> > > --- a/src/lxc/monitor.c
> > > +++ b/src/lxc/monitor.c
> > > @@ -20,6 +20,7 @@
> > >   * License along with this library; if not, write to the Free
> > > Software
> > >   * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> > > 02111-1307 USA */
> > > +
> > >  #include <stdio.h>
> > >  #include <errno.h>
> > >  #include <unistd.h>
> > > @@ -30,7 +31,7 @@
> > >  #include <sys/stat.h>
> > >  #include <sys/param.h>
> > >  #include <sys/socket.h>
> > > -#include <sys/un.h>
> > > +#include <sys/wait.h>
> > >  #include <netinet/in.h>
> > >  #include <net/if.h>
> > >
> > > @@ -43,35 +44,24 @@
> > >
> > >  lxc_log_define(lxc_monitor, lxc);
> > >
> > > -#ifndef UNIX_PATH_MAX
> > > -#define UNIX_PATH_MAX 108
> > > -#endif
> > > -
> > > -static void lxc_monitor_send(struct lxc_msg *msg, const char
> > > *lxcpath) +/* routines used by monitor publishers (containers) */
> > > +static void lxc_monitor_fifo_send(struct lxc_msg *msg, const char
> > > *lxcpath)
> > >  {
> > > -       int fd;
> > > -       struct sockaddr_un addr = { .sun_family = AF_UNIX };
> > > -       char *offset = &addr.sun_path[1];
> > > -       size_t ret, len;
> > > +       int fd,ret;
> > > +       char fifo_path[PATH_MAX];
> > >
> > > -       /*
> > > -        * addr.sun_path is only 108 bytes.
> > > -        * should we take a hash of lxcpath?  a subset of it?
> > > -        */
> > > -       len = sizeof(addr.sun_path) - 1;
> > > -       ret = snprintf(offset, len, "%s/lxc-monitor", lxcpath);
> > > -       if (ret < 0 || ret >= len) {
> > > -               ERROR("lxcpath too long to open monitor");
> > > +       //FIXME: BuildAssert(sizeof(*msg) < PIPE_BUF), "write not
> > > guaranteed atomic");
> > > +       ret = snprintf(fifo_path, sizeof(fifo_path), "%s/monitor",
> > > lxcpath);
> > > +       if (ret < 0 || ret >= sizeof(fifo_path)) {
> > > +               ERROR("lxcpath too long to open monitor fifo");
> > >                 return;
> > >         }
> > > -
> > > -       fd = socket(PF_UNIX, SOCK_DGRAM, 0);
> > > -       if (fd < 0)
> > > +       fd = open(fifo_path, O_WRONLY);
> > > +       if (fd < 0) {
> > > +               ERROR("failed to open monitor fifo %s", fifo_path);
> > >                 return;
> > > -
> > > -       sendto(fd, msg, sizeof(*msg), 0,
> > > -              (const struct sockaddr *)&addr, sizeof(addr));
> > > -
> > > +       }
> > > +       ret = write(fd, msg, sizeof(*msg));
> > >         close(fd);
> > >  }
> > >
> > > @@ -82,72 +72,74 @@ void lxc_monitor_send_state(const char *name,
> > > lxc_state_t state, const char *lxc
> > >         strncpy(msg.name, name, sizeof(msg.name));
> > >         msg.name[sizeof(msg.name) - 1] = 0;
> > >
> > > -       lxc_monitor_send(&msg, lxcpath);
> > > +       lxc_monitor_fifo_send(&msg, lxcpath);
> > >  }
> > >
> > > -int lxc_monitor_open(const char *lxcpath)
> > > +
> > > +/* routines used by monitor subscribers (lxc-monitor) */
> > > +int lxc_monitor_close(int fd)
> > >  {
> > > -       struct sockaddr_un addr = { .sun_family = AF_UNIX };
> > > -       char *offset = &addr.sun_path[1];
> > > -       int fd;
> > > -       size_t ret, len;
> > > +       return close(fd);
> > > +}
> > >
> > > -       /*
> > > -        * addr.sun_path is only 108 bytes.
> > > -        * should we take a hash of lxcpath?  a subset of it?
> > > +int lxc_monitor_sock_name(const char *lxcpath, struct sockaddr_un
> > > *addr) {
> > > +       size_t len,ret;
> > > +       char *sockname = &addr->sun_path[0]; // 1 for abstract
> > > +
> > > +       /* addr.sun_path is only 108 bytes.
> > > +        * should we take a hash of lxcpath?  a subset of it?
> > > ftok()? none
> > > +        * are guaranteed.
> > >          */
> > > -       len = sizeof(addr.sun_path) - 1;
> > > -       ret = snprintf(offset, len, "%s/lxc-monitor", lxcpath);
> > > +       memset(addr, 0, sizeof(*addr));
> > > +       addr->sun_family = AF_UNIX;
> > > +       len = sizeof(addr->sun_path) - 1;
> > > +       ret = snprintf(sockname, len, "%s/monitor-sock", lxcpath);
> > >         if (ret < 0 || ret >= len) {
> > > -               ERROR("lxcpath too long to open monitor");
> > > +               ERROR("lxcpath too long for unix socket");
> > >                 return -1;
> > >         }
> > > +       return 0;
> > > +}
> > > +
> > > +int lxc_monitor_open(const char *lxcpath)
> > > +{
> > > +       struct sockaddr_un addr;
> > > +       size_t ret;
> > > +       int fd;
> > >
> > > -       fd = socket(PF_UNIX, SOCK_DGRAM, 0);
> > > +       if (lxc_monitor_sock_name(lxcpath, &addr) < 0)
> > > +               return -1;
> > > +
> > > +       fd = socket(PF_UNIX, SOCK_STREAM, 0);
> > >         if (fd < 0) {
> > >                 ERROR("socket : %s", strerror(errno));
> > >                 return -1;
> > >         }
> > >
> > > -       if (bind(fd, (struct sockaddr *)&addr, sizeof(addr))) {
> > > -               ERROR("bind : %s", strerror(errno));
> > > -               close(fd);
> > > -               return -1;
> > > +       ret = connect(fd, (struct sockaddr *)&addr, sizeof(addr));
> > > +       if (ret < 0) {
> > > +               /* try to spawn a lxc-monitord here instead ? */
> > > +               ERROR("connect : %s", strerror(errno));
> > > +               goto err1;
> > >         }
> > > -
> > >         return fd;
> > > +err1:
> > > +       close(fd);
> > > +       return ret;
> > >  }
> > >
> > > -/* timeout of 0 means return immediately;  -1 means wait forever */
> > > -int lxc_monitor_read_timeout(int fd, struct lxc_msg *msg, int
> > > timeout) +int lxc_monitor_read_timeout(int fd, struct lxc_msg
> > > *msglxc, int timeout) {
> > >         struct sockaddr_un from;
> > >         socklen_t len = sizeof(from);
> > >         int ret;
> > > -       fd_set rfds;
> > > -       struct timeval tv;
> > >
> > > -       if (timeout != -1) {
> > > -               FD_ZERO(&rfds);
> > > -               FD_SET(fd, &rfds);
> > > -
> > > -               tv.tv_sec = timeout;
> > > -               tv.tv_usec = 0;
> > > -
> > > -               ret = select(fd+1, &rfds, NULL, NULL, &tv);
> > > -               if (ret == -1)
> > > -                       return -1;
> > > -               else if (!ret)
> > > -                       return -2;  // timed out
> > > -       }
> > > -
> > > -       ret = recvfrom(fd, msg, sizeof(*msg), 0,
> > > +       ret = recvfrom(fd, msglxc, sizeof(*msglxc), 0,
> > >                        (struct sockaddr *)&from, &len);
> > > -       if (ret < 0) {
> > > -               SYSERROR("failed to receive state");
> > > +       if (ret <= 0) {
> > > +               SYSERROR("client failed to receive lxc_msg (monitord
> > > died?)");
> > >                 return -1;
> > >         }
> > > -
> > >         return ret;
> > >  }
> > >
> > > @@ -156,7 +148,61 @@ int lxc_monitor_read(int fd, struct lxc_msg
> > > *msg) return lxc_monitor_read_timeout(fd, msg, -1);
> > >  }
> > >
> > > -int lxc_monitor_close(int fd)
> > > +
> > > +
> > > +/* used to spawn a monitord either on startup of a container, or
> > > when
> > > + * lxc-monitor starts
> > > + */
> > > +int lxc_monitord_spawn(const char *lxcpath)
> > >  {
> > > -       return close(fd);
> > > +       pid_t pid1,pid2;
> > > +       int pipefd[2];
> > > +       char pipefd_str[10];
> > > +
> > > +       char * const args[] = {
> > > +               "/usr/bin/lxc-monitord",
> > > +               (char *)lxcpath,
> > > +               pipefd_str,
> > > +               NULL,
> > > +       };
> > > +
> > > +       /* double fork to avoid zombies when monitord exits */
> > > +       pid1 = fork();
> > > +       if (pid1 < 0) {
> > > +               SYSERROR("failed to fork");
> > > +               return -1;
> > > +       }
> > > +
> > > +       if (pid1) {
> > > +               waitpid(pid1, NULL, 0);
> > > +               return 0;
> > > +       }
> > > +
> > > +       if (pipe(pipefd) < 0) {
> > > +               SYSERROR("failed to create pipe");
> > > +               exit(EXIT_FAILURE);
> > > +       }
> > > +
> > > +       pid2 = fork();
> > > +       if (pid2) {
> > > +               char c;
> > > +               /* wait for daemon to create socket */
> > > +               close(pipefd[1]);
> > > +               read(pipefd[0], &c, 1);
> > > +               close(pipefd[0]);
> > > +               exit(EXIT_SUCCESS);
> > > +       }
> > > +
> > > +       umask(0);
> > > +       if (setsid() < 0) {
> > > +               SYSERROR("failed to setsid");
> > > +               exit(EXIT_FAILURE);
> > > +       }
> > > +       freopen("/dev/null", "r", stdin);
> > > +       freopen("/dev/null", "w", stdout);
> > > +       freopen("/dev/null", "w", stderr);
> > > +       close(pipefd[0]);
> > > +       sprintf(pipefd_str, "%d", pipefd[1]);
> > > +       execvp(args[0], args);
> > > +       exit(EXIT_FAILURE);
> > >  }
> > > diff --git a/src/lxc/monitor.h b/src/lxc/monitor.h
> > > index 8bef4c7..faa606c 100644
> > > --- a/src/lxc/monitor.h
> > > +++ b/src/lxc/monitor.h
> > > @@ -24,6 +24,10 @@
> > >  #define __monitor_h
> > >
> > >  #include <sys/param.h>
> > > +#include <sys/un.h>
> > > +
> > > +#include <lxc/conf.h>
> > > +#include <lxc/mainloop.h>
> > >
> > >  typedef enum {
> > >         lxc_msg_state,
> > > @@ -32,11 +36,14 @@ typedef enum {
> > >
> > >  struct lxc_msg {
> > >         lxc_msg_type_t type;
> > > -       char name[MAXPATHLEN];
> > > +       char name[NAME_MAX+1];
> > >         int value;
> > >  };
> > >
> > > -void lxc_monitor_send_state(const char *name, lxc_state_t state,
> > > +extern int lxc_monitor_open(const char *lxcpath);
> > > +extern int lxc_monitor_sock_name(const char *lxcpath, struct
> > > sockaddr_un *addr);
> > > +extern void lxc_monitor_send_state(const char *name, lxc_state_t
> > > state, const char *lxcpath);
> > > +extern int lxc_monitord_spawn(const char *lxcpath);
> > >
> > >  #endif
> > > diff --git a/src/lxc/start.c b/src/lxc/start.c
> > > index aefccd6..9451fb7 100644
> > > --- a/src/lxc/start.c
> > > +++ b/src/lxc/start.c
> > > @@ -429,6 +429,9 @@ struct lxc_handler *lxc_init(const char *name,
> > > struct lxc_conf *conf, const char
> > >         if (lxc_command_init(name, handler, lxcpath))
> > >                 goto out_free_name;
> > >
> > > +       if (lxc_monitord_spawn(lxcpath))
> > > +               goto out_close_maincmd_fd;
> > > +
> > >         if (lxc_read_seccomp_config(conf) != 0) {
> > >                 ERROR("failed loading seccomp policy");
> > >                 goto out_close_maincmd_fd;
> > > @@ -437,7 +440,7 @@ struct lxc_handler *lxc_init(const char *name,
> > > struct lxc_conf *conf, const char
> > >         /* Begin the set the state to STARTING*/
> > >         if (lxc_set_state(name, handler, STARTING)) {
> > >                 ERROR("failed to set state '%s'",
> > > lxc_state2str(STARTING));
> > > -               goto out_free_name;
> > > +               goto out_close_maincmd_fd;
> > >         }
> > >
> > >         /* Start of environment variable setup for hooks */
> > > @@ -808,7 +811,7 @@ int lxc_spawn(struct lxc_handler *handler)
> > >         /* TODO - pass lxc.cgroup.dir (or user's pam cgroup) in for
> > > first argument */
> > >         if ((handler->cgroup = lxc_cgroup_path_create(NULL, name))
> > > == NULL) goto out_delete_net;
> > > -
> > > +
> > >         if (lxc_cgroup_enter(handler->cgroup, handler->pid) < 0)
> > >                 goto out_delete_net;
> > >
> > > --
> > > 1.8.1.4
> > >
> > >
> > >
> > >
> > >
> ------------------------------------------------------------------------------
> > > Precog is a next-generation analytics platform capable of advanced
> > > analytics on semi-structured data. The platform includes APIs for
> > > building apps and a phenomenal toolset for data science. Developers
> > > can use our toolset for easy data analysis & visualization. Get a
> > > free account!
> > > http://www2.precog.com/precogplatform/slashdotnewsletter
> > > _______________________________________________ Lxc-devel mailing
> > > list Lxc-devel at lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/lxc-devel
> > >
> >
> >
> >
>
>


-- 
S.Çağlar Onur <caglar at 10ur.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130418/b2a4aedf/attachment.html>


More information about the lxc-devel mailing list