[lxc-devel] [RFC PATCH] allow multiple monitor clients
Dwight Engen
dwight.engen at oracle.com
Thu Apr 18 19:20:37 UTC 2013
Thanks Çağlar! I'll look into those things.
On Thu, 18 Apr 2013 15:08:19 -0400
S.Çağlar Onur <caglar at 10ur.org> wrote:
> Hi,
>
> I think I got it why, most likely recvfrom gets an EINTR there like
> select.
>
> int lxc_monitor_read_timeout(int fd, struct lxc_msg *msglxc, int
> timeout) {
> struct sockaddr_un from;
> socklen_t len = sizeof(from);
> int ret;
>
> ret = recvfrom(fd, msglxc, sizeof(*msglxc), 0,
> (struct sockaddr *)&from, &len);
> if (ret <= 0) {
> SYSERROR("client failed to receive lxc_msg (monitord
> died?)");
> return -1;
> }
> return ret;
> }
>
>
>
>
> On Thu, Apr 18, 2013 at 3:02 PM, S.Çağlar Onur <caglar at 10ur.org>
> wrote:
>
> > Hi Dwight,
> >
> > Couple of quick comments, I needed following to make it compile
> > under ubuntu 12.10 otherwise gcc whines about ignoring return
> > values of those calls.
> >
> > diff --git a/src/lxc/lxc_monitord.c b/src/lxc/lxc_monitord.c
> > index 1d20d67..9d83673 100644
> > --- a/src/lxc/lxc_monitord.c
> > +++ b/src/lxc/lxc_monitord.c
> > @@ -309,7 +309,11 @@ int main(int argc, char *argv[])
> > }
> >
> > /* sync with parent */
> > - write(pipefd, "S", 1);
> > + if (write(pipefd, "S", 1) < 0) {
> > + ERROR("write failed");
> > + goto out;
> > + }
> > +
> > close(pipefd);
> >
> > if (lxc_monitord_mainloop_add(&mon)) {
> > diff --git a/src/lxc/monitor.c b/src/lxc/monitor.c
> > index 0438fbc..803f98f 100644
> > --- a/src/lxc/monitor.c
> > +++ b/src/lxc/monitor.c
> > @@ -158,6 +158,7 @@ int lxc_monitord_spawn(const char *lxcpath)
> > pid_t pid1,pid2;
> > int pipefd[2];
> > char pipefd_str[10];
> > + FILE *fd;
> >
> > char * const args[] = {
> > "/usr/bin/lxc-monitord",
> > @@ -188,7 +189,10 @@ int lxc_monitord_spawn(const char *lxcpath)
> > char c;
> > /* wait for daemon to create socket */
> > close(pipefd[1]);
> > - read(pipefd[0], &c, 1);
> > + if (read(pipefd[0], &c, 1) < 0) {
> > + SYSERROR("read failed");
> > + exit(EXIT_FAILURE);
> > + }
> > close(pipefd[0]);
> > exit(EXIT_SUCCESS);
> > }
> > @@ -198,9 +202,21 @@ int lxc_monitord_spawn(const char *lxcpath)
> > SYSERROR("failed to setsid");
> > exit(EXIT_FAILURE);
> > }
> > - freopen("/dev/null", "r", stdin);
> > - freopen("/dev/null", "w", stdout);
> > - freopen("/dev/null", "w", stderr);
> > + fd = freopen("/dev/null", "r", stdin);
> > + if (!fd) {
> > + SYSERROR("freopen failed");
> > + exit(EXIT_FAILURE);
> > + }
> > + fd = freopen("/dev/null", "w", stdout);
> > + if (!fd) {
> > + SYSERROR("freopen failed");
> > + exit(EXIT_FAILURE);
> > + }
> > + fd = freopen("/dev/null", "w", stderr);
> > + if (!fd) {
> > + SYSERROR("freopen failed");
> > + exit(EXIT_FAILURE);
> > + }
> > close(pipefd[0]);
> > sprintf(pipefd_str, "%d", pipefd[1]);
> > execvp(args[0], args);
> >
> > and time to time (I think that only happens if monitord wasn't
> > running initially) I get following while starting 10 containers in
> > parallel
> >
> > [caglar at qgq:~/Project/lxc/examples] sudo ./concurrent_start
> > Starting the container (2)...
> > Starting the container (6)...
> > Starting the container (1)...
> > lxc_container: Invalid argument - client failed to receive lxc_msg
> > (monitord died?)
> > Starting the container (2) failed...
> > Starting the container (0)...
> > Starting the container (3)...
> > Starting the container (5)...
> > Starting the container (7)...
> > Starting the container (8)...
> > Starting the container (9)...
> > Starting the container (4)...
> >
> > Best,
> >
> >
> > On Thu, Apr 18, 2013 at 2:47 PM, Dwight Engen
> > <dwight.engen at oracle.com>wrote:
> >
> >> On Thu, 18 Apr 2013 14:22:08 -0400
> >> S.Çağlar Onur <caglar at 10ur.org> wrote:
> >>
> >> > Hey Dwight,
> >> >
> >> > This sounds great, I'll give it a try ASAP. In the meantime ,if
> >> > you want to test it yourself, you can grab go bindings from
> >> > github via "go get github.com/caglar10ur/lxc" but please keep in
> >> > your mind that you need my other two patches to start/stop work
> >> > reliably [1]/[2].
> >> > https://github.com/caglar10ur/lxc/tree/master/examples directory
> >> > contains concurrent versions of create/start/stop/shutdown to
> >> > test.
> >> >
> >> > [1]
> >> >
> >> https://github.com/caglar10ur/lxc-upstream/commit/db60b9c38083e8ed553f3cbdd96ad410bd5eb7ed.patch
> >> > [2]
> >> >
> >> https://github.com/caglar10ur/lxc-upstream/commit/97d4daf4cc5016bcb01c72e5cf9fe3189e9df4f5.patch
> >> >
> >> > Cheers,
> >>
> >> Thanks Caglar, I may give them a try as I've just built go the
> >> other day to try out docker.
> >>
> >> > On Thu, Apr 18, 2013 at 2:09 PM, Dwight Engen
> >> > <dwight.engen at oracle.com>wrote:
> >> >
> >> > > After the recent discussions on the lxc monitor, I took a
> >> > > closer look at it. It seems like Serge was resigned to having
> >> > > some sort of daemon as having multicast unix domain supported
> >> > > by the kernel is not going to happen any time soon, and doing
> >> > > it over loopback has some complications too.
> >> > >
> >> > > Initially I thought it would be nice to just have lxc-start
> >> > > "be the daemon" as Stéphane had suggested. I tried this out,
> >> > > putting the fds for the consumer clients (lxc-monitors) in
> >> > > lxc_mainloop , but this is problematic for the scenario where
> >> > > lxc-monitor runs first and we miss some startup states.
> >> > >
> >> > > So here is what I'm proposing: when lxc-monitor starts, it
> >> > > attempts to start lxc-monitord. lxc-monitord creates a fifo
> >> > > and a socket on lxcpath.
> >> > >
> >> > > Producers (containers) wishing to send messages open the fifo
> >> > > and write their lxc_msg's on it. The reason I used a fifo is
> >> > > so that we get permission checks on the opening of it. Right
> >> > > now it is created 0600. This fixes the problem we have today
> >> > > where anyone can open the unix socket and send state changes
> >> > > in.
> >> > >
> >> > > Consumers (lxc-monitor) connect to the unix socket and each
> >> > > gets a copy of the messages. lxc-monitord can do a PEERCRED to
> >> > > ensure the client should be able to get these messages, this
> >> > > is one benefit to using unix domain. I don't think it would be
> >> > > difficult to write a lxc-monitor like client that would bridge
> >> > > the messages onto d-bus.
> >> > >
> >> > > When lxc-monitord hasn't had a client for 30 seconds it will
> >> > > exit.
> >> > >
> >> > > Attached is a proof of concept patch for your comments. It
> >> > > works for the common cases that I've tried but I know it needs
> >> > > some cleanup and testing if we like the approach. The
> >> > > monitor.c part is hard to read because that file is mostly
> >> > > rewritten.
> >> > >
> >> > > I need to test S.Çağlar Onur's scenario of parallel startup
> >> > > with his test script. We need to decide if we want to monitor
> >> > > per lxcpath or per container, today it is per path and I think
> >> > > that being able to do lxc-monitor * would be very useful.
> >> > >
> >> > > --
> >> > >
> >> > > Signed-off-by: Dwight Engen <dwight.engen at oracle.com>
> >> > > ---
> >> > > src/lxc/Makefile.am | 2 +
> >> > > src/lxc/af_unix.c | 5 +-
> >> > > src/lxc/lxc_monitor.c | 2 +
> >> > > src/lxc/lxc_monitord.c | 334
> >> > > +++++++++++++++++++++++++++++++++++++++++++++++++
> >> > > src/lxc/mainloop.c | 11 +-
> >> > > src/lxc/mainloop.h | 7 ++
> >> > > src/lxc/monitor.c | 180 ++++++++++++++++----------
> >> > > src/lxc/monitor.h | 11 +-
> >> > > src/lxc/start.c | 7 +-
> >> > > 9 files changed, 484 insertions(+), 75 deletions(-)
> >> > > create mode 100644 src/lxc/lxc_monitord.c
> >> > >
> >> > > diff --git a/src/lxc/Makefile.am b/src/lxc/Makefile.am
> >> > > index ebeca466..1fa0fa8 100644
> >> > > --- a/src/lxc/Makefile.am
> >> > > +++ b/src/lxc/Makefile.am
> >> > > @@ -150,6 +150,7 @@ bin_PROGRAMS = \
> >> > > lxc-start \
> >> > > lxc-execute \
> >> > > lxc-monitor \
> >> > > + lxc-monitord \
> >> > > lxc-wait \
> >> > > lxc-console \
> >> > > lxc-freeze \
> >> > > @@ -181,6 +182,7 @@ lxc_freeze_SOURCES = lxc_freeze.c
> >> > > lxc_info_SOURCES = lxc_info.c
> >> > > lxc_init_SOURCES = lxc_init.c
> >> > > lxc_monitor_SOURCES = lxc_monitor.c
> >> > > +lxc_monitord_SOURCES = lxc_monitord.c
> >> > > lxc_restart_SOURCES = lxc_restart.c
> >> > > lxc_start_SOURCES = lxc_start.c
> >> > > lxc_stop_SOURCES = lxc_stop.c
> >> > > diff --git a/src/lxc/af_unix.c b/src/lxc/af_unix.c
> >> > > index eff13d4..ceef340 100644
> >> > > --- a/src/lxc/af_unix.c
> >> > > +++ b/src/lxc/af_unix.c
> >> > > @@ -52,6 +52,7 @@ int lxc_af_unix_open(const char *path, int
> >> > > type, int flags)
> >> > >
> >> > > addr.sun_family = AF_UNIX;
> >> > > /* copy entire buffer in case of abstract socket */
> >> > > + /* FIXME: this is unsafe if path is too long... */
> >> > > memcpy(addr.sun_path, path,
> >> > > path[0]?strlen(path):sizeof(addr.sun_path));
> >> > >
> >> > > @@ -61,7 +62,7 @@ int lxc_af_unix_open(const char *path, int
> >> > > type, int flags)
> >> > > errno = tmp;
> >> > > return -1;
> >> > > }
> >> > > -
> >> > > +
> >> > > if (type == SOCK_STREAM && listen(fd, 100)) {
> >> > > int tmp = errno;
> >> > > close(fd);
> >> > > @@ -76,7 +77,7 @@ int lxc_af_unix_close(int fd)
> >> > > {
> >> > > struct sockaddr_un addr;
> >> > > socklen_t addrlen = sizeof(addr);
> >> > > -
> >> > > +
> >> > > if (!getsockname(fd, (struct sockaddr *)&addr,
> >> > > &addrlen) && addr.sun_path[0])
> >> > > unlink(addr.sun_path);
> >> > > diff --git a/src/lxc/lxc_monitor.c b/src/lxc/lxc_monitor.c
> >> > > index 8c15869..0ca829f 100644
> >> > > --- a/src/lxc/lxc_monitor.c
> >> > > +++ b/src/lxc/lxc_monitor.c
> >> > > @@ -87,6 +87,8 @@ int main(int argc, char *argv[])
> >> > > return -1;
> >> > > }
> >> > >
> >> > > + lxc_monitord_spawn(my_args.lxcpath);
> >> > > +
> >> > > fd = lxc_monitor_open(my_args.lxcpath);
> >> > > if (fd < 0)
> >> > > return -1;
> >> > > diff --git a/src/lxc/lxc_monitord.c b/src/lxc/lxc_monitord.c
> >> > > new file mode 100644
> >> > > index 0000000..3e9c708
> >> > > --- /dev/null
> >> > > +++ b/src/lxc/lxc_monitord.c
> >> > > @@ -0,0 +1,334 @@
> >> > > +/*
> >> > > + * lxc: linux Container library
> >> > > + *
> >> > > + * Copyright © 2012 Oracle.
> >> > > + *
> >> > > + * Authors:
> >> > > + * Dwight Engen <dwight.engen at oracle.com>
> >> > > + *
> >> > > + * This library is free software; you can redistribute it
> >> > > and/or
> >> > > + * modify it under the terms of the GNU Lesser General Public
> >> > > + * License as published by the Free Software Foundation;
> >> > > either
> >> > > + * version 2.1 of the License, or (at your option) any later
> >> > > version.
> >> > > + *
> >> > > + * This library is distributed in the hope that it will be
> >> > > useful,
> >> > > + * but WITHOUT ANY WARRANTY; without even the implied
> >> > > warranty of
> >> > > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
> >> > > the GNU
> >> > > + * Lesser General Public License for more details.
> >> > > + *
> >> > > + * You should have received a copy of the GNU Lesser General
> >> > > Public
> >> > > + * License along with this library; if not, write to the Free
> >> > > Software
> >> > > + * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> >> > > 02111-1307 USA
> >> > > + */
> >> > > +
> >> > > +#define _GNU_SOURCE
> >> > > +#include <stdio.h>
> >> > > +#include <signal.h>
> >> > > +#include <errno.h>
> >> > > +#include <unistd.h>
> >> > > +#include <string.h>
> >> > > +#include <stdlib.h>
> >> > > +#include <fcntl.h>
> >> > > +#include <sys/types.h>
> >> > > +#include <sys/stat.h>
> >> > > +#include <sys/param.h>
> >> > > +#include <sys/socket.h>
> >> > > +#include <sys/un.h>
> >> > > +#include <netinet/in.h>
> >> > > +#include <net/if.h>
> >> > > +
> >> > > +#include <lxc/af_unix.h>
> >> > > +#include <lxc/log.h>
> >> > > +#include <lxc/mainloop.h>
> >> > > +#include <lxc/monitor.h>
> >> > > +#include <lxc/utils.h>
> >> > > +
> >> > > +lxc_log_define(lxc_monitord, lxc);
> >> > > +
> >> > > +/*
> >> > > + * Defines the structure to store the monitor information
> >> > > + * @fifofd : the file descriptor for publishers
> >> > > (containers) to write state
> >> > > + * @listenfd : the file descriptor for subscribers
> >> > > (lxc-monitors) to connect
> >> > > + * @clientfds : accepted client file descriptors
> >> > > + * @clientfds_cnt: the count of valid fds in clientfds
> >> > > + */
> >> > > +struct lxc_monitor {
> >> > > + int fifofd;
> >> > > + int listenfd;
> >> > > + int clientfds[1024];
> >> > > + int clientfds_cnt;
> >> > > + struct lxc_epoll_descr descr;
> >> > > +};
> >> > > +
> >> > > +static int lxc_monitord_fifo_create(struct lxc_monitor *mon,
> >> > > const char *lxcpath)
> >> > > +{
> >> > > + char fifo_path[PATH_MAX];
> >> > > + int ret;
> >> > > +
> >> > > + ret = snprintf(fifo_path, sizeof(fifo_path),
> >> > > "%s/monitor", lxcpath);
> >> > > + if (ret < 0 || ret >= sizeof(fifo_path)) {
> >> > > + ERROR("lxcpath too long to monitor fifo");
> >> > > + return -1;
> >> > > + }
> >> > > +
> >> > > + ret = mknod(fifo_path, S_IFIFO|S_IRUSR|S_IWUSR, 0);
> >> > > + if (ret < 0) {
> >> > > + INFO("existing monitor fifo at %s, monitor
> >> > > already running?", fifo_path);
> >> > > + return -1;
> >> > > + }
> >> > > + mon->fifofd = open(fifo_path, O_RDWR);
> >> > > + if (mon->fifofd < 0) {
> >> > > + ERROR("failed to open monitor fifo");
> >> > > + return -1;
> >> > > + }
> >> > > + return 0;
> >> > > +}
> >> > > +
> >> > > +static int lxc_monitord_fifo_delete(const char *lxcpath)
> >> > > +{
> >> > > + char fifo_path[PATH_MAX];
> >> > > + int ret;
> >> > > +
> >> > > + ret = snprintf(fifo_path, sizeof(fifo_path),
> >> > > "%s/monitor", lxcpath);
> >> > > + if (ret < 0 || ret >= sizeof(fifo_path)) {
> >> > > + ERROR("lxcpath too long to monitor fifo");
> >> > > + return -1;
> >> > > + }
> >> > > + unlink(fifo_path);
> >> > > + return 0;
> >> > > +}
> >> > > +
> >> > > +static void lxc_monitord_sockfd_remove(struct lxc_monitor
> >> > > *mon, int fd) {
> >> > > + int i;
> >> > > +
> >> > > + lxc_mainloop_del_handler(&mon->descr, fd);
> >> > > + close(fd);
> >> > > +
> >> > > + for (i = 0; i < mon->clientfds_cnt; i++) {
> >> > > + if (mon->clientfds[i] == fd)
> >> > > + break;
> >> > > + }
> >> > > + if (i >= mon->clientfds_cnt) {
> >> > > + CRIT("fd not found");
> >> > > + exit(EXIT_FAILURE);
> >> > > + }
> >> > > + DEBUG("MON: removing client fd:%d memmoving %p %p %lu
> >> > > bytes", fd,
> >> > > + &mon->clientfds[mon->clientfds_cnt],
> >> > > + &mon->clientfds[i],
> >> > > + (&mon->clientfds[mon->clientfds_cnt-1] -
> >> > > &mon->clientfds[i]) * sizeof(int));
> >> > > + memmove(&mon->clientfds[i], &mon->clientfds[i+1],
> >> > > + (&mon->clientfds[mon->clientfds_cnt-1] -
> >> > > &mon->clientfds[i]) * sizeof(int));
> >> > > + mon->clientfds_cnt--;
> >> > > +}
> >> > > +
> >> > > +static int lxc_monitord_sock_handler(int fd, void *data,
> >> > > + struct lxc_epoll_descr
> >> > > *descr) +{
> >> > > + struct lxc_monitor *mon = data;
> >> > > + lxc_monitord_sockfd_remove(mon, fd);
> >> > > + return 0;
> >> > > +}
> >> > > +
> >> > > +static int lxc_monitord_sock_accept(int fd, void *data,
> >> > > + struct lxc_epoll_descr
> >> > > *descr) +{
> >> > > + int ret,clientfd;
> >> > > + struct lxc_monitor *mon = data;
> >> > > +
> >> > > + ret = -1;
> >> > > + clientfd = accept(fd, NULL, 0);
> >> > > + if (clientfd < 0) {
> >> > > + SYSERROR("failed to accept connection");
> >> > > + goto out;
> >> > > + }
> >> > > +
> >> > > + if (fcntl(clientfd, F_SETFD, FD_CLOEXEC)) {
> >> > > + SYSERROR("failed to set close-on-exec on
> >> > > incoming connection");
> >> > > + goto err1;
> >> > > + }
> >> > > +
> >> > > + /* FIXME: check the connected clients credentials here,
> >> > > drop if we don't
> >> > > + * like them.
> >> > > + int on = 1;
> >> > > + if (setsockopt(clientfd, SOL_SOCKET, SO_PASSCRED, &on,
> >> > > sizeof(on))) {
> >> > > + ERROR("failed to enable credential on socket");
> >> > > + goto err1;
> >> > > + }
> >> > > + */
> >> > > +
> >> > > + ret = lxc_mainloop_add_handler(&mon->descr, clientfd,
> >> > > +
> >> > > lxc_monitord_sock_handler, mon);
> >> > > + if (ret) {
> >> > > + ERROR("failed to add socket handler");
> >> > > + goto err1;
> >> > > + }
> >> > > + mon->clientfds[mon->clientfds_cnt++] = clientfd;
> >> > > + INFO("accepted new client fd:%d clientfds_cnt:%d",
> >> > > clientfd, mon->clientfds_cnt);
> >> > > + goto out;
> >> > > +
> >> > > +err1:
> >> > > + close(clientfd);
> >> > > +out:
> >> > > + return ret;
> >> > > +}
> >> > > +
> >> > > +static int lxc_monitord_sock_create(struct lxc_monitor *mon,
> >> > > const char *lxcpath)
> >> > > +{
> >> > > + struct sockaddr_un addr;
> >> > > + int fd;
> >> > > +
> >> > > + if (lxc_monitor_sock_name(lxcpath, &addr) < 0)
> >> > > + return -1;
> >> > > +
> >> > > + fd = lxc_af_unix_open(addr.sun_path, SOCK_STREAM,
> >> > > O_TRUNC);
> >> > > + if (fd < 0) {
> >> > > + ERROR("failed to open unix socket : %s",
> >> > > strerror(errno));
> >> > > + return -1;
> >> > > + }
> >> > > +
> >> > > + mon->listenfd = fd;
> >> > > + return 0;
> >> > > +}
> >> > > +
> >> > > +static int lxc_monitord_sock_delete(const char *lxcpath)
> >> > > +{
> >> > > + struct sockaddr_un addr;
> >> > > +
> >> > > + if (lxc_monitor_sock_name(lxcpath, &addr) < 0)
> >> > > + return -1;
> >> > > + if (addr.sun_path[0])
> >> > > + unlink(addr.sun_path);
> >> > > + return 0;
> >> > > +}
> >> > > +
> >> > > +static int lxc_monitord_create(struct lxc_monitor *mon, const
> >> > > char *lxcpath)
> >> > > +{
> >> > > + int ret;
> >> > > +
> >> > > + ret = lxc_monitord_fifo_create(mon, lxcpath);
> >> > > + if (ret < 0)
> >> > > + return ret;
> >> > > +
> >> > > + ret = lxc_monitord_sock_create(mon, lxcpath);
> >> > > + return ret;
> >> > > +}
> >> > > +
> >> > > +static void lxc_monitord_delete(struct lxc_monitor *mon,
> >> > > const char *lxcpath)
> >> > > +{
> >> > > + int i;
> >> > > +
> >> > > + lxc_mainloop_del_handler(&mon->descr, mon->listenfd);
> >> > > + close(mon->listenfd);
> >> > > + lxc_monitord_sock_delete(lxcpath);
> >> > > +
> >> > > + lxc_mainloop_del_handler(&mon->descr, mon->fifofd);
> >> > > + close(mon->fifofd);
> >> > > + lxc_monitord_fifo_delete(lxcpath);
> >> > > +
> >> > > + for (i = 0; i < mon->clientfds_cnt; i++) {
> >> > > + lxc_mainloop_del_handler(&mon->descr,
> >> > > mon->clientfds[i]);
> >> > > + close(mon->clientfds[i]);
> >> > > + }
> >> > > + mon->clientfds_cnt = 0;
> >> > > +}
> >> > > +
> >> > > +static int lxc_monitord_fifo_handler(int fd, void *data,
> >> > > + struct lxc_epoll_descr
> >> > > *descr) +{
> >> > > + int ret,i;
> >> > > + struct lxc_msg msglxc;
> >> > > + struct lxc_monitor *mon = data;
> >> > > +
> >> > > + ret = read(fd, &msglxc, sizeof(msglxc));
> >> > > + if (ret != sizeof(msglxc)) {
> >> > > + SYSERROR("read fifo failed : %s",
> >> > > strerror(errno));
> >> > > + return 1;
> >> > > + }
> >> > > +
> >> > > + for (i = 0; i < mon->clientfds_cnt; i++) {
> >> > > + ret = write(mon->clientfds[i], &msglxc,
> >> > > sizeof(msglxc));
> >> > > + if (ret < 0) {
> >> > > + ERROR("write failed to client sock,
> >> > > removing");
> >> > > + lxc_monitord_sockfd_remove(mon, i);
> >> > > + }
> >> > > + }
> >> > > +
> >> > > + return 0;
> >> > > +}
> >> > > +
> >> > > +static int lxc_monitord_mainloop_add(struct lxc_monitor *mon)
> >> > > +{
> >> > > + int ret;
> >> > > +
> >> > > + ret = lxc_mainloop_add_handler(&mon->descr,
> >> > > mon->fifofd,
> >> > > +
> >> > > lxc_monitord_fifo_handler, mon);
> >> > > + if (ret < 0) {
> >> > > + ERROR("failed to add to mainloop monitor
> >> > > handler for fifo");
> >> > > + return -1;
> >> > > + }
> >> > > +
> >> > > + ret = lxc_mainloop_add_handler(&mon->descr,
> >> > > mon->listenfd,
> >> > > +
> >> > > lxc_monitord_sock_accept, mon);
> >> > > + if (ret < 0) {
> >> > > + ERROR("failed to add to mainloop monitor
> >> > > handler for listen socket");
> >> > > + return -1;
> >> > > + }
> >> > > +
> >> > > + return 0;
> >> > > +}
> >> > > +
> >> > > +static void lxc_monitord_sig_handler(int sig)
> >> > > +{
> >> > > + INFO("caught signal %d", sig);
> >> > > +}
> >> > > +
> >> > > +int main(int argc, char *argv[])
> >> > > +{
> >> > > + struct lxc_monitor mon;
> >> > > + int ret,pipefd;
> >> > > + char *lxcpath = argv[1];
> >> > > + char logpath[PATH_MAX];
> >> > > +
> >> > > + signal(SIGTERM, lxc_monitord_sig_handler);
> >> > > + snprintf(logpath, sizeof(logpath),
> >> > > "%s/lxc-monitord.log", lxcpath);
> >> > > + ret = lxc_log_init(NULL, logpath, "INFO",
> >> > > "lxc-monitord", 0);
> >> > > + if (ret)
> >> > > + return ret;
> >> > > +
> >> > > + pipefd = atoi(argv[2]);
> >> > > +
> >> > > + ret = EXIT_FAILURE;
> >> > > + memset(&mon, 0, sizeof(mon));
> >> > > + if (lxc_mainloop_open(&mon.descr)) {
> >> > > + ERROR("failed to create mainloop");
> >> > > + goto out;
> >> > > + }
> >> > > +
> >> > > + if (lxc_monitord_create(&mon, lxcpath)) {
> >> > > + goto out;
> >> > > + }
> >> > > +
> >> > > + /* sync with parent */
> >> > > + write(pipefd, "S", 1);
> >> > > + close(pipefd);
> >> > > +
> >> > > + if (lxc_monitord_mainloop_add(&mon)) {
> >> > > + ERROR("failed to add mainloop handlers");
> >> > > + goto out;
> >> > > + }
> >> > > +
> >> > > + NOTICE("monitoring lxcpath %s", lxcpath);
> >> > > + for(;;) {
> >> > > + ret = lxc_mainloop_timeout(&mon.descr, 1000 *
> >> > > 30);
> >> > > + if (mon.clientfds_cnt <= 0)
> >> > > + {
> >> > > + NOTICE("no clients for 30 seconds,
> >> > > exiting");
> >> > > + break;
> >> > > + }
> >> > > + }
> >> > > +
> >> > > + lxc_monitord_delete(&mon, lxcpath);
> >> > > + lxc_mainloop_close(&mon.descr);
> >> > > +out:
> >> > > + return ret;
> >> > > +}
> >> > > diff --git a/src/lxc/mainloop.c b/src/lxc/mainloop.c
> >> > > index 975215d..4805dcd 100644
> >> > > --- a/src/lxc/mainloop.c
> >> > > +++ b/src/lxc/mainloop.c
> >> > > @@ -38,7 +38,7 @@ struct mainloop_handler {
> >> > >
> >> > > #define MAX_EVENTS 10
> >> > >
> >> > > -int lxc_mainloop(struct lxc_epoll_descr *descr)
> >> > > +int lxc_mainloop_timeout(struct lxc_epoll_descr *descr, int
> >> > > timeout_ms) {
> >> > > int i, nfds;
> >> > > struct mainloop_handler *handler;
> >> > > @@ -46,7 +46,7 @@ int lxc_mainloop(struct lxc_epoll_descr
> >> > > *descr)
> >> > >
> >> > > for (;;) {
> >> > >
> >> > > - nfds = epoll_wait(descr->epfd, events,
> >> > > MAX_EVENTS, -1);
> >> > > + nfds = epoll_wait(descr->epfd, events,
> >> > > MAX_EVENTS, timeout_ms);
> >> > > if (nfds < 0) {
> >> > > if (errno == EINTR)
> >> > > continue;
> >> > > @@ -64,11 +64,18 @@ int lxc_mainloop(struct lxc_epoll_descr
> >> > > *descr) return 0;
> >> > > }
> >> > >
> >> > > + if (nfds == 0 && timeout_ms != 0)
> >> > > + return 0;
> >> > > +
> >> > > if (lxc_list_empty(&descr->handlers))
> >> > > return 0;
> >> > > }
> >> > > }
> >> > >
> >> > > +int lxc_mainloop(struct lxc_epoll_descr *descr) {
> >> > > + return lxc_mainloop_timeout(descr, -1);
> >> > > +}
> >> > > +
> >> > > int lxc_mainloop_add_handler(struct lxc_epoll_descr *descr,
> >> > > int fd, lxc_mainloop_callback_t callback, void
> >> > > *data) {
> >> > > diff --git a/src/lxc/mainloop.h b/src/lxc/mainloop.h
> >> > > index 6b16242..c79c4fd 100644
> >> > > --- a/src/lxc/mainloop.h
> >> > > +++ b/src/lxc/mainloop.h
> >> > > @@ -21,6 +21,9 @@
> >> > > * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> >> > > 02111-1307 USA */
> >> > >
> >> > > +#ifndef _mainloop_h
> >> > > +#define _mainloop_h
> >> > > +
> >> > > #include "list.h"
> >> > >
> >> > > struct lxc_epoll_descr {
> >> > > @@ -33,6 +36,8 @@ typedef int (*lxc_mainloop_callback_t)(int
> >> > > fd, void *data,
> >> > >
> >> > > extern int lxc_mainloop(struct lxc_epoll_descr *descr);
> >> > >
> >> > > +extern int lxc_mainloop_timeout(struct lxc_epoll_descr
> >> > > *descr, int timeout_ms);
> >> > > +
> >> > > extern int lxc_mainloop_add_handler(struct lxc_epoll_descr
> >> > > *descr, int fd, lxc_mainloop_callback_t callback,
> >> > > void *data);
> >> > > @@ -42,3 +47,5 @@ extern int lxc_mainloop_del_handler(struct
> >> > > lxc_epoll_descr *descr, int fd);
> >> > > extern int lxc_mainloop_open(struct lxc_epoll_descr *descr);
> >> > >
> >> > > extern int lxc_mainloop_close(struct lxc_epoll_descr *descr);
> >> > > +
> >> > > +#endif
> >> > > diff --git a/src/lxc/monitor.c b/src/lxc/monitor.c
> >> > > index afdaf67..0438fbc 100644
> >> > > --- a/src/lxc/monitor.c
> >> > > +++ b/src/lxc/monitor.c
> >> > > @@ -20,6 +20,7 @@
> >> > > * License along with this library; if not, write to the Free
> >> > > Software
> >> > > * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
> >> > > 02111-1307 USA */
> >> > > +
> >> > > #include <stdio.h>
> >> > > #include <errno.h>
> >> > > #include <unistd.h>
> >> > > @@ -30,7 +31,7 @@
> >> > > #include <sys/stat.h>
> >> > > #include <sys/param.h>
> >> > > #include <sys/socket.h>
> >> > > -#include <sys/un.h>
> >> > > +#include <sys/wait.h>
> >> > > #include <netinet/in.h>
> >> > > #include <net/if.h>
> >> > >
> >> > > @@ -43,35 +44,24 @@
> >> > >
> >> > > lxc_log_define(lxc_monitor, lxc);
> >> > >
> >> > > -#ifndef UNIX_PATH_MAX
> >> > > -#define UNIX_PATH_MAX 108
> >> > > -#endif
> >> > > -
> >> > > -static void lxc_monitor_send(struct lxc_msg *msg, const char
> >> > > *lxcpath) +/* routines used by monitor publishers (containers)
> >> > > */ +static void lxc_monitor_fifo_send(struct lxc_msg *msg,
> >> > > const char *lxcpath)
> >> > > {
> >> > > - int fd;
> >> > > - struct sockaddr_un addr = { .sun_family = AF_UNIX };
> >> > > - char *offset = &addr.sun_path[1];
> >> > > - size_t ret, len;
> >> > > + int fd,ret;
> >> > > + char fifo_path[PATH_MAX];
> >> > >
> >> > > - /*
> >> > > - * addr.sun_path is only 108 bytes.
> >> > > - * should we take a hash of lxcpath? a subset of it?
> >> > > - */
> >> > > - len = sizeof(addr.sun_path) - 1;
> >> > > - ret = snprintf(offset, len, "%s/lxc-monitor", lxcpath);
> >> > > - if (ret < 0 || ret >= len) {
> >> > > - ERROR("lxcpath too long to open monitor");
> >> > > + //FIXME: BuildAssert(sizeof(*msg) < PIPE_BUF), "write
> >> > > not guaranteed atomic");
> >> > > + ret = snprintf(fifo_path, sizeof(fifo_path),
> >> > > "%s/monitor", lxcpath);
> >> > > + if (ret < 0 || ret >= sizeof(fifo_path)) {
> >> > > + ERROR("lxcpath too long to open monitor fifo");
> >> > > return;
> >> > > }
> >> > > -
> >> > > - fd = socket(PF_UNIX, SOCK_DGRAM, 0);
> >> > > - if (fd < 0)
> >> > > + fd = open(fifo_path, O_WRONLY);
> >> > > + if (fd < 0) {
> >> > > + ERROR("failed to open monitor fifo %s",
> >> > > fifo_path); return;
> >> > > -
> >> > > - sendto(fd, msg, sizeof(*msg), 0,
> >> > > - (const struct sockaddr *)&addr, sizeof(addr));
> >> > > -
> >> > > + }
> >> > > + ret = write(fd, msg, sizeof(*msg));
> >> > > close(fd);
> >> > > }
> >> > >
> >> > > @@ -82,72 +72,74 @@ void lxc_monitor_send_state(const char
> >> > > *name, lxc_state_t state, const char *lxc
> >> > > strncpy(msg.name, name, sizeof(msg.name));
> >> > > msg.name[sizeof(msg.name) - 1] = 0;
> >> > >
> >> > > - lxc_monitor_send(&msg, lxcpath);
> >> > > + lxc_monitor_fifo_send(&msg, lxcpath);
> >> > > }
> >> > >
> >> > > -int lxc_monitor_open(const char *lxcpath)
> >> > > +
> >> > > +/* routines used by monitor subscribers (lxc-monitor) */
> >> > > +int lxc_monitor_close(int fd)
> >> > > {
> >> > > - struct sockaddr_un addr = { .sun_family = AF_UNIX };
> >> > > - char *offset = &addr.sun_path[1];
> >> > > - int fd;
> >> > > - size_t ret, len;
> >> > > + return close(fd);
> >> > > +}
> >> > >
> >> > > - /*
> >> > > - * addr.sun_path is only 108 bytes.
> >> > > - * should we take a hash of lxcpath? a subset of it?
> >> > > +int lxc_monitor_sock_name(const char *lxcpath, struct
> >> > > sockaddr_un *addr) {
> >> > > + size_t len,ret;
> >> > > + char *sockname = &addr->sun_path[0]; // 1 for abstract
> >> > > +
> >> > > + /* addr.sun_path is only 108 bytes.
> >> > > + * should we take a hash of lxcpath? a subset of it?
> >> > > ftok()? none
> >> > > + * are guaranteed.
> >> > > */
> >> > > - len = sizeof(addr.sun_path) - 1;
> >> > > - ret = snprintf(offset, len, "%s/lxc-monitor", lxcpath);
> >> > > + memset(addr, 0, sizeof(*addr));
> >> > > + addr->sun_family = AF_UNIX;
> >> > > + len = sizeof(addr->sun_path) - 1;
> >> > > + ret = snprintf(sockname, len, "%s/monitor-sock",
> >> > > lxcpath); if (ret < 0 || ret >= len) {
> >> > > - ERROR("lxcpath too long to open monitor");
> >> > > + ERROR("lxcpath too long for unix socket");
> >> > > return -1;
> >> > > }
> >> > > + return 0;
> >> > > +}
> >> > > +
> >> > > +int lxc_monitor_open(const char *lxcpath)
> >> > > +{
> >> > > + struct sockaddr_un addr;
> >> > > + size_t ret;
> >> > > + int fd;
> >> > >
> >> > > - fd = socket(PF_UNIX, SOCK_DGRAM, 0);
> >> > > + if (lxc_monitor_sock_name(lxcpath, &addr) < 0)
> >> > > + return -1;
> >> > > +
> >> > > + fd = socket(PF_UNIX, SOCK_STREAM, 0);
> >> > > if (fd < 0) {
> >> > > ERROR("socket : %s", strerror(errno));
> >> > > return -1;
> >> > > }
> >> > >
> >> > > - if (bind(fd, (struct sockaddr *)&addr, sizeof(addr))) {
> >> > > - ERROR("bind : %s", strerror(errno));
> >> > > - close(fd);
> >> > > - return -1;
> >> > > + ret = connect(fd, (struct sockaddr *)&addr,
> >> > > sizeof(addr));
> >> > > + if (ret < 0) {
> >> > > + /* try to spawn a lxc-monitord here instead ?
> >> > > */
> >> > > + ERROR("connect : %s", strerror(errno));
> >> > > + goto err1;
> >> > > }
> >> > > -
> >> > > return fd;
> >> > > +err1:
> >> > > + close(fd);
> >> > > + return ret;
> >> > > }
> >> > >
> >> > > -/* timeout of 0 means return immediately; -1 means wait
> >> > > forever */ -int lxc_monitor_read_timeout(int fd, struct
> >> > > lxc_msg *msg, int timeout) +int lxc_monitor_read_timeout(int
> >> > > fd, struct lxc_msg *msglxc, int timeout) {
> >> > > struct sockaddr_un from;
> >> > > socklen_t len = sizeof(from);
> >> > > int ret;
> >> > > - fd_set rfds;
> >> > > - struct timeval tv;
> >> > >
> >> > > - if (timeout != -1) {
> >> > > - FD_ZERO(&rfds);
> >> > > - FD_SET(fd, &rfds);
> >> > > -
> >> > > - tv.tv_sec = timeout;
> >> > > - tv.tv_usec = 0;
> >> > > -
> >> > > - ret = select(fd+1, &rfds, NULL, NULL, &tv);
> >> > > - if (ret == -1)
> >> > > - return -1;
> >> > > - else if (!ret)
> >> > > - return -2; // timed out
> >> > > - }
> >> > > -
> >> > > - ret = recvfrom(fd, msg, sizeof(*msg), 0,
> >> > > + ret = recvfrom(fd, msglxc, sizeof(*msglxc), 0,
> >> > > (struct sockaddr *)&from, &len);
> >> > > - if (ret < 0) {
> >> > > - SYSERROR("failed to receive state");
> >> > > + if (ret <= 0) {
> >> > > + SYSERROR("client failed to receive lxc_msg
> >> > > (monitord died?)");
> >> > > return -1;
> >> > > }
> >> > > -
> >> > > return ret;
> >> > > }
> >> > >
> >> > > @@ -156,7 +148,61 @@ int lxc_monitor_read(int fd, struct
> >> > > lxc_msg *msg) return lxc_monitor_read_timeout(fd, msg, -1);
> >> > > }
> >> > >
> >> > > -int lxc_monitor_close(int fd)
> >> > > +
> >> > > +
> >> > > +/* used to spawn a monitord either on startup of a container,
> >> > > or when
> >> > > + * lxc-monitor starts
> >> > > + */
> >> > > +int lxc_monitord_spawn(const char *lxcpath)
> >> > > {
> >> > > - return close(fd);
> >> > > + pid_t pid1,pid2;
> >> > > + int pipefd[2];
> >> > > + char pipefd_str[10];
> >> > > +
> >> > > + char * const args[] = {
> >> > > + "/usr/bin/lxc-monitord",
> >> > > + (char *)lxcpath,
> >> > > + pipefd_str,
> >> > > + NULL,
> >> > > + };
> >> > > +
> >> > > + /* double fork to avoid zombies when monitord exits */
> >> > > + pid1 = fork();
> >> > > + if (pid1 < 0) {
> >> > > + SYSERROR("failed to fork");
> >> > > + return -1;
> >> > > + }
> >> > > +
> >> > > + if (pid1) {
> >> > > + waitpid(pid1, NULL, 0);
> >> > > + return 0;
> >> > > + }
> >> > > +
> >> > > + if (pipe(pipefd) < 0) {
> >> > > + SYSERROR("failed to create pipe");
> >> > > + exit(EXIT_FAILURE);
> >> > > + }
> >> > > +
> >> > > + pid2 = fork();
> >> > > + if (pid2) {
> >> > > + char c;
> >> > > + /* wait for daemon to create socket */
> >> > > + close(pipefd[1]);
> >> > > + read(pipefd[0], &c, 1);
> >> > > + close(pipefd[0]);
> >> > > + exit(EXIT_SUCCESS);
> >> > > + }
> >> > > +
> >> > > + umask(0);
> >> > > + if (setsid() < 0) {
> >> > > + SYSERROR("failed to setsid");
> >> > > + exit(EXIT_FAILURE);
> >> > > + }
> >> > > + freopen("/dev/null", "r", stdin);
> >> > > + freopen("/dev/null", "w", stdout);
> >> > > + freopen("/dev/null", "w", stderr);
> >> > > + close(pipefd[0]);
> >> > > + sprintf(pipefd_str, "%d", pipefd[1]);
> >> > > + execvp(args[0], args);
> >> > > + exit(EXIT_FAILURE);
> >> > > }
> >> > > diff --git a/src/lxc/monitor.h b/src/lxc/monitor.h
> >> > > index 8bef4c7..faa606c 100644
> >> > > --- a/src/lxc/monitor.h
> >> > > +++ b/src/lxc/monitor.h
> >> > > @@ -24,6 +24,10 @@
> >> > > #define __monitor_h
> >> > >
> >> > > #include <sys/param.h>
> >> > > +#include <sys/un.h>
> >> > > +
> >> > > +#include <lxc/conf.h>
> >> > > +#include <lxc/mainloop.h>
> >> > >
> >> > > typedef enum {
> >> > > lxc_msg_state,
> >> > > @@ -32,11 +36,14 @@ typedef enum {
> >> > >
> >> > > struct lxc_msg {
> >> > > lxc_msg_type_t type;
> >> > > - char name[MAXPATHLEN];
> >> > > + char name[NAME_MAX+1];
> >> > > int value;
> >> > > };
> >> > >
> >> > > -void lxc_monitor_send_state(const char *name, lxc_state_t
> >> > > state, +extern int lxc_monitor_open(const char *lxcpath);
> >> > > +extern int lxc_monitor_sock_name(const char *lxcpath, struct
> >> > > sockaddr_un *addr);
> >> > > +extern void lxc_monitor_send_state(const char *name,
> >> > > lxc_state_t state, const char *lxcpath);
> >> > > +extern int lxc_monitord_spawn(const char *lxcpath);
> >> > >
> >> > > #endif
> >> > > diff --git a/src/lxc/start.c b/src/lxc/start.c
> >> > > index aefccd6..9451fb7 100644
> >> > > --- a/src/lxc/start.c
> >> > > +++ b/src/lxc/start.c
> >> > > @@ -429,6 +429,9 @@ struct lxc_handler *lxc_init(const char
> >> > > *name, struct lxc_conf *conf, const char
> >> > > if (lxc_command_init(name, handler, lxcpath))
> >> > > goto out_free_name;
> >> > >
> >> > > + if (lxc_monitord_spawn(lxcpath))
> >> > > + goto out_close_maincmd_fd;
> >> > > +
> >> > > if (lxc_read_seccomp_config(conf) != 0) {
> >> > > ERROR("failed loading seccomp policy");
> >> > > goto out_close_maincmd_fd;
> >> > > @@ -437,7 +440,7 @@ struct lxc_handler *lxc_init(const char
> >> > > *name, struct lxc_conf *conf, const char
> >> > > /* Begin the set the state to STARTING*/
> >> > > if (lxc_set_state(name, handler, STARTING)) {
> >> > > ERROR("failed to set state '%s'",
> >> > > lxc_state2str(STARTING));
> >> > > - goto out_free_name;
> >> > > + goto out_close_maincmd_fd;
> >> > > }
> >> > >
> >> > > /* Start of environment variable setup for hooks */
> >> > > @@ -808,7 +811,7 @@ int lxc_spawn(struct lxc_handler *handler)
> >> > > /* TODO - pass lxc.cgroup.dir (or user's pam cgroup)
> >> > > in for first argument */
> >> > > if ((handler->cgroup = lxc_cgroup_path_create(NULL,
> >> > > name)) == NULL) goto out_delete_net;
> >> > > -
> >> > > +
> >> > > if (lxc_cgroup_enter(handler->cgroup, handler->pid) <
> >> > > 0) goto out_delete_net;
> >> > >
> >> > > --
> >> > > 1.8.1.4
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> ------------------------------------------------------------------------------
> >> > > Precog is a next-generation analytics platform capable of
> >> > > advanced analytics on semi-structured data. The platform
> >> > > includes APIs for building apps and a phenomenal toolset for
> >> > > data science. Developers can use our toolset for easy data
> >> > > analysis & visualization. Get a free account!
> >> > > http://www2.precog.com/precogplatform/slashdotnewsletter
> >> > > _______________________________________________ Lxc-devel
> >> > > mailing list Lxc-devel at lists.sourceforge.net
> >> > > https://lists.sourceforge.net/lists/listinfo/lxc-devel
> >> > >
> >> >
> >> >
> >> >
> >>
> >>
> >
> >
> > --
> > S.Çağlar Onur <caglar at 10ur.org>
> >
>
>
>
More information about the lxc-devel
mailing list