<div dir="ltr">OK reproduced it with C API<div><br></div><div><div>26624 lxc-monitord 1367687488.006 NOTICE lxc_monitord - monitoring lxcpath /var/lib/lxc</div><div>26627 lxc-monitord 1367687488.008 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26630 lxc-monitord 1367687488.009 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div><div>26634 lxc-monitord 1367687488.011 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26635 lxc-monitord 1367687488.011 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div><div>26636 lxc-monitord 1367687488.014 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26639 lxc-monitord 1367687488.014 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div><div>26644 lxc-monitord 1367687488.014 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26650 lxc-monitord 1367687488.018 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div><div>26652 lxc-monitord 1367687488.018 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26624 lxc-monitord 1367687488.020 INFO lxc_monitord - accepted client fd:5 clients:1</div><div>26624 lxc-monitord 1367687488.020 INFO lxc_monitord - accepted client fd:9 clients:2</div><div>26643 lxc-monitord 1367687488.021 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26664 lxc-monitord 1367687488.023 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div><div>26624 lxc-monitord 1367687488.025 INFO lxc_monitord - accepted client fd:10 clients:3</div>
<div>26667 lxc-monitord 1367687488.026 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div><div>26624 lxc-monitord 1367687488.028 INFO lxc_monitord - accepted client fd:11 clients:4</div>
<div>26677 lxc-monitord 1367687488.028 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div><div>26679 lxc-monitord 1367687488.028 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26624 lxc-monitord 1367687488.028 INFO lxc_monitord - accepted client fd:12 clients:5</div><div>26680 lxc-monitord 1367687488.031 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26624 lxc-monitord 1367687488.031 INFO lxc_monitord - accepted client fd:13 clients:6</div><div>26684 lxc-monitord 1367687488.032 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26676 lxc-monitord 1367687488.033 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div><div>26624 lxc-monitord 1367687488.034 INFO lxc_monitord - accepted client fd:14 clients:7</div>
<div>26624 lxc-monitord 1367687488.034 INFO lxc_monitord - accepted client fd:15 clients:8</div><div>26683 lxc-monitord 1367687488.036 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26624 lxc-monitord 1367687488.037 INFO lxc_monitord - accepted client fd:16 clients:9</div><div>26690 lxc-monitord 1367687488.038 INFO lxc_monitord - monitor fifo /var/lib/lxc/monitor-fifo exists, already running?</div>
<div>26624 lxc-monitord 1367687488.039 INFO lxc_monitord - accepted client fd:17 clients:10</div><div>26624 lxc-monitord 1367687488.748 INFO lxc_monitord - removing fd 15</div><div>26624 lxc-monitord 1367687489.239 INFO lxc_monitord - removing fd 17</div>
<div>26624 lxc-monitord 1367687489.395 INFO lxc_monitord - removing fd 16</div><div>26624 lxc-monitord 1367687519.425 NOTICE lxc_monitord - # of clients: 7</div><div><br></div><div style>code sits @ <a href="http://10ur.org/lxc/">http://10ur.org/lxc/</a></div>
</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, May 4, 2013 at 12:22 PM, S.Çağlar Onur <span dir="ltr"><<a href="mailto:caglar@10ur.org" target="_blank">caglar@10ur.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Dwight,<div><br></div><div>I run same commands and this time it ended up with 6 clients (output -> <a href="http://10ur.org/monitor.txt" target="_blank">http://10ur.org/monitor.txt</a>). I'll try to reproduce this with c/python.;.</div>
<div><br></div><div><div class="gmail_extra"><div><div class="h5"><div class="gmail_quote">On Sat, May 4, 2013 at 11:15 AM, S.Çağlar Onur <span dir="ltr"><<a href="mailto:caglar@10ur.org" target="_blank">caglar@10ur.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi Dwight,<div class="gmail_extra"><br><div class="gmail_quote">
<div>On Sat, May 4, 2013 at 9:16 AM, Dwight Engen <span dir="ltr"><<a href="mailto:dwight.engen@oracle.com" target="_blank">dwight.engen@oracle.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Çağlar,<br>
<br>
I'm confused by your output, it certainly looks like something isn't<br>
right. Do you have a theory as to why monitord thinks it still has 9<br>
clients?</blockquote><div><br></div></div><div>Nope, I don't. I'll try to debug more and also try to reproduce this with python or c api to rule out go.</div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div>
On Sat, 4 May 2013 00:01:45 -0400<br>
S.Çağlar Onur <<a href="mailto:caglar@10ur.org" target="_blank">caglar@10ur.org</a>> wrote:<br>
<br>
> Hi all,<br>
><br>
> I think I understand why I'm confused before while chasing another<br>
> bug. This is what I'm seeing right now.<br>
><br>
> * I patched lxc_monitord.c with following<br>
><br>
> diff --git a/src/lxc/lxc_monitord.c b/src/lxc/lxc_monitord.c<br>
> index e76af71..59f1e9d 100644<br>
> --- a/src/lxc/lxc_monitord.c<br>
> +++ b/src/lxc/lxc_monitord.c<br>
> @@ -373,6 +373,7 @@ int main(int argc, char *argv[])<br>
> }<br>
><br>
> if (lxc_monitord_create(&mon)) {<br>
> + NOTICE("create failed");<br>
> goto out;<br>
> }<br>
><br>
> @@ -398,6 +399,7 @@ int main(int argc, char *argv[])<br>
> NOTICE("no clients for 30 seconds, exiting");<br>
> break;<br>
> }<br>
> + NOTICE("clients %d", mon.clientfds_cnt);<br>
> }<br>
><br>
> lxc_mainloop_close(&mon.descr);<br>
><br>
> * I started 10 containers using go bindings<br>
><br>
> [caglar@qgq:~/Project/lxc/examples] sudo ./concurrent_start<br>
> Starting the container (3)...<br>
> Starting the container (2)...<br>
> Starting the container (4)...<br>
> Starting the container (0)...<br>
> Starting the container (1)...<br>
> Starting the container (8)...<br>
> Starting the container (7)...<br>
> Starting the container (6)...<br>
> Starting the container (5)...<br>
> Starting the container (9)...<br>
><br>
> * Then started to stop them 1 by 1 using lxc-stop<br>
><br>
> [caglar@qgq:~/Project/lxc/examples] sudo lxc-stop -n 0<br>
> [caglar@qgq:~/Project/lxc/examples] sudo ./list<br>
> 0 (STOPPED)<br>
> 1 (RUNNING)<br>
> 2 (RUNNING)<br>
> 3 (RUNNING)<br>
> 4 (RUNNING)<br>
> 5 (RUNNING)<br>
> 6 (RUNNING)<br>
> 7 (RUNNING)<br>
> 8 (RUNNING)<br>
> 9 (RUNNING)<br>
<br>
</div></div>I assume you stopped 1-8 here?</blockquote><div><br></div></div></div><div>Yes, sorry for pasting the half of the output.</div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div>
> [caglar@qgq:~/Project/lxc/examples] date && sudo ./list<br>
> Fri May 3 23:57:14 EDT 2013<br>
> 0 (STOPPED)<br>
> 1 (STOPPED)<br>
> 2 (STOPPED)<br>
> 3 (STOPPED)<br>
> 4 (STOPPED)<br>
> 5 (STOPPED)<br>
> 6 (STOPPED)<br>
> 7 (STOPPED)<br>
> 8 (STOPPED)<br>
> 9 (RUNNING)<br>
> bleach (STOPPED)<br>
><br>
> * lxc-monitord is still around after ~10min<br>
<br>
</div>Looks like its not going away because it thinks there are 9 clients<br>
still. My guess is somehow its not getting notified of the client<br>
closes (or they're still around?). The following patch should provide a<br>
bit more info in the log:<br></blockquote><div><br></div></div><div>As long as I see they are not around (I use lxc-stop to stop them). </div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
diff --git a/src/lxc/lxc_monitord.c b/src/lxc/lxc_monitord.c<br>
index e76af71..537a2b3 100644<br>
--- a/src/lxc/lxc_monitord.c<br>
+++ b/src/lxc/lxc_monitord.c<br>
@@ -114,6 +114,7 @@ static int lxc_monitord_fifo_delete(struct lxc_monitor *mon)<br>
static void lxc_monitord_sockfd_remove(struct lxc_monitor *mon, int fd) {<br>
int i;<br>
<br>
+ INFO("removing fd %d\n", fd);<br>
if (lxc_mainloop_del_handler(&mon->descr, fd))<br>
CRIT("fd:%d not found in mainloop", fd);<br>
close(fd);<br>
@@ -343,7 +344,7 @@ int main(int argc, char *argv[])<br>
if (ret < 0 || ret >= sizeof(logpath))<br>
return EXIT_FAILURE;<br>
<br>
- ret = lxc_log_init(NULL, logpath, "NOTICE", "lxc-monitord", 0, lxcpath);<br>
+ ret = lxc_log_init(NULL, logpath, "INFO", "lxc-monitord", 0, lxcpath);<br>
if (ret)<br>
return ret;<br>
<div><div><br>
<br>
<br>
> [caglar@qgq:~/Project/lxc/examples] ps aux |<br>
> grep /usr/bin/lxc-monitord caglar 1170 0.0 0.0 13580 940<br>
> pts/3 S+ 23:57 0:00 grep --color=auto /usr/bin/lxc-monitord<br>
> root 29997 0.0 0.0 15000 744 ? Ss 23:47 0:00<br>
> /usr/bin/lxc-monitord /var/lib/lxc 5<br>
> [caglar@qgq:~/Project/lxc/examples] date<br>
> Fri May 3 23:57:52 EDT 2013<br>
><br>
> * And lastly here is what lxc-monitord.log shows<br>
><br>
> [caglar@qgq:~/Project/lxc(clone)] tail<br>
> -f /var/lib/lxc/lxc-monitord.log lxc-monitord 1367639242.631 NOTICE<br>
> lxc_monitord - monitoring lxcpath /var/lib/lxc<br>
> lxc-monitord 1367639242.633 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.633 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.636 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.639 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.643 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.643 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.651 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.654 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.665 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.678 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.681 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.681 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.682 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.707 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.710 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.710 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.722 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.733 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639242.831 NOTICE lxc_monitord - create failed<br>
> lxc-monitord 1367639274.071 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639323.928 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639372.862 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639444.107 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639474.130 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639504.133 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639534.161 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639564.190 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639594.209 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639624.223 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639654.256 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639684.287 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639714.317 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639744.347 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639774.370 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639804.396 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639834.426 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639864.456 NOTICE lxc_monitord - clients 9<br>
> lxc-monitord 1367639894.486 NOTICE lxc_monitord - clients 9<br>
<br>
</div></div>You might want to consider patching the log stuff to print out pids, I<br>
found that helpful while working on this:<br></blockquote><div><br></div></div></div><div>Will patch both monitord.c and log.c and report back. Thanks!</div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
diff --git a/src/lxc/log.c b/src/lxc/log.c<br>
index d49a544..98581c1 100644<br>
--- a/src/lxc/log.c<br>
+++ b/src/lxc/log.c<br>
@@ -58,7 +58,7 @@ static int log_append_stderr(const struct lxc_log_appender *appender,<br>
if (event->priority < LXC_LOG_PRIORITY_ERROR)<br>
return 0;<br>
<br>
- fprintf(stderr, "%s: ", log_prefix);<br>
+ fprintf(stderr, "%-5d %s: ", getpid(), log_prefix);<br>
vfprintf(stderr, event->fmt, *event->vap);<br>
fprintf(stderr, "\n");<br>
return 0;<br>
@@ -75,7 +75,8 @@ static int log_append_logfile(const struct lxc_log_appender *appender,<br>
return 0;<br>
<br>
n = snprintf(buffer, sizeof(buffer),<br>
- "%15s %10ld.%03ld %-8s %s - ",<br>
+ "%-5d %15s %10ld.%03ld %-8s %s - ",<br>
+ getpid(),<br>
log_prefix,<br>
event->timestamp.tv_sec,<br>
event->timestamp.tv_usec / 1000,<br>
<div><div><br>
<br>
> On Fri, Apr 26, 2013 at 4:52 PM, S.Çağlar Onur <<a href="mailto:caglar@10ur.org" target="_blank">caglar@10ur.org</a>><br>
> wrote:<br>
><br>
> > Yeah, I think you all correct and I'm just confused - probably<br>
> > direct effect of lack of caffeine. And no, it's not complicating<br>
> > something for me, it's working great. I just want to make sure that<br>
> > I'm wrong :)<br>
> ><br>
> ><br>
> > On Fri, Apr 26, 2013 at 4:37 PM, Dwight Engen<br>
> > <<a href="mailto:dwight.engen@oracle.com" target="_blank">dwight.engen@oracle.com</a>>wrote:<br>
> ><br>
> >> On Fri, 26 Apr 2013 22:07:22 +0200<br>
> >> Stéphane Graber <<a href="mailto:stgraber@ubuntu.com" target="_blank">stgraber@ubuntu.com</a>> wrote:<br>
> >><br>
> >> > On 04/26/2013 09:42 PM, S.Çağlar Onur wrote:<br>
> >> > > Hey Dwight,<br>
> >> > ><br>
> >> > > I'm observing following behavior with staging tree and just<br>
> >> > > wanted to make sure that what I'm seeing is the expected;<br>
> >> > ><br>
> >> > > * Initially nothing runs<br>
> >> > ><br>
> >> > > [caglar@qgq:~/Projects/lxc/examples] sudo ./list<br>
> >> > > bankai (STOPPED)<br>
> >> > > bleach (STOPPED)<br>
> >> > > zangetsu (STOPPED)<br>
> >> > ><br>
> >> > > * I start one container using the API<br>
> >> > ><br>
> >> > > [caglar@qgq:~/Projects/lxc/examples] sudo ./start -name<br>
> >> > > zangetsu Starting the container...<br>
> >> > ><br>
> >> > > [caglar@qgq:~/Projects/lxc/examples] sudo ./list<br>
> >> > > bankai (STOPPED)<br>
> >> > > bleach (STOPPED)<br>
> >> > > zangetsu (RUNNING)<br>
> >> > ><br>
> >> > > * monitord starts as expected but exits after 30 seconds later<br>
> >> > > (although container is still running);<br>
> >> > ><br>
> >> > > [caglar@qgq:~/Projects/lxc-upstream(staging)] tail -f<br>
> >> > > /var/lib/lxc/lxc-monitord.log<br>
> >> > > lxc-monitord 1367004858.616 NOTICE lxc_monitord -<br>
> >> > > monitoring lxcpath /var/lib/lxc<br>
> >> > > lxc-monitord 1367004888.677 NOTICE lxc_monitord - no<br>
> >> > > clients for 30 seconds, exiting<br>
> >> > > lxc-monitord 1367004888.677 NOTICE lxc_monitord - monitor<br>
> >> > > exiting<br>
> >> > ><br>
> >> > > [caglar@qgq:~/Projects/lxc/examples] sudo ./list<br>
> >> > > bankai (STOPPED)<br>
> >> > > bleach (STOPPED)<br>
> >> > > zangetsu (RUNNING)<br>
> >> > ><br>
> >> > > [caglar@qgq:~/Projects/lxc/examples] ps aux | grep monitord<br>
> >> > > caglar 28404 0.0 0.0 7240 624 pts/54 S+ 15:34<br>
> >> > > 0:00 tail -f /var/lib/lxc/lxc-monitord.log<br>
> >> > > caglar 29037 0.0 0.0 9436 948 pts/0 S+ 15:38<br>
> >> > > 0:00 grep --color=auto monitord<br>
> >> > > [caglar@qgq:~/Projects/lxc/examples]<br>
> >> > ><br>
> >> > > I'm asking cause I was under the impression that lxc-monitord<br>
> >> > > will keep running as long as there is a container. Am I wrong?<br>
> >> ><br>
> >> > I believe the monitor will get spawned the first time something<br>
> >> > needs it (lxc-monitor/lxc-wait) and exit 30s after the last<br>
> >> > client disconnects. It'll then be respawned the next time<br>
> >> > lxc-monitor or lxc-wait is started again that container.<br>
> >><br>
> >> Yep Stéphane, that is correct. Also note that the monitord is per<br>
> >> lxcpath, not per container.<br>
> >><br>
> >> Çağlar, you may have been slightly confused because if you start a<br>
> >> container in daemon mode through the API, the API does an internal<br>
> >> lxc_wait() and thus a monitord will get spawned when you first<br>
> >> start a container, but will go away ~30 seconds afterwards.<br>
> >><br>
> ><br>
> ><br>
> ><br>
> > --<br>
> > S.Çağlar Onur <<a href="mailto:caglar@10ur.org" target="_blank">caglar@10ur.org</a>><br>
> ><br>
><br>
><br>
><br>
<br>
</div></div></blockquote></div></div></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br>S.Çağlar Onur <<a href="mailto:caglar@10ur.org" target="_blank">caglar@10ur.org</a>>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br>S.Çağlar Onur <<a href="mailto:caglar@10ur.org" target="_blank">caglar@10ur.org</a>>
</font></span></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>S.Çağlar Onur <<a href="mailto:caglar@10ur.org">caglar@10ur.org</a>>
</div>