[lxc-devel] [PATCH] tests: Introduce lxc-test-concurrent for testing basic actions concurrently
S.Çağlar Onur
caglar at 10ur.org
Sat Sep 14 16:52:41 UTC 2013
Hi Serge,
This seems to fix the creates but starts started to fail this time. One
quick note, my concurrent.c still uses ubuntu as it's much more easy to
replicate the issue with it.
diff --git a/src/tests/concurrent.c b/src/tests/concurrent.c
index 7faf34c..ffc025c 100644
--- a/src/tests/concurrent.c
+++ b/src/tests/concurrent.c
@@ -40,7 +40,7 @@ void * concurrent(void *arguments) {
args->return_code = 1;
if (strcmp(args->mode, "create") == 0) {
if (!c->is_defined(c)) {
- if (!c->create(c, "busybox", NULL, NULL, 1, NULL)) {
+ if (!c->create(c, "ubuntu", NULL, NULL, 1, NULL)) {
fprintf(stderr, "Creating the container (%s) failed...\n",
name);
goto out;
}
[caglar at oOo:~/Projects/lxc(staging)] sudo lxc-test-concurrent
Executing (create) for 5 containers...
Executing (start) for 5 containers...
Starting the container (4) failed...
lxc_container: command get_cgroup failed to receive response
Starting the container (3) failed...
Starting the container (2) failed...
Starting the container (1) failed...
Starting the container (0) failed...
[caglar at oOo:~/go/src/github.com/caglar10ur/lxc(devel)] sudo lxc-ls
0 1 2 3 4
[caglar at oOo:~/go/src/github.com/caglar10ur/lxc(devel)] sudo lxc-start -d -n
0
lxc-start: command get_cgroup failed to receive response
[caglar at oOo:~/go/src/github.com/caglar10ur/lxc(devel)]
On Fri, Sep 13, 2013 at 10:20 PM, Serge Hallyn <serge.hallyn at ubuntu.com>wrote:
> Quoting Serge Hallyn (serge.hallyn at ubuntu.com):
> > Quoting Dwight Engen (dwight.engen at oracle.com):
> > > On Fri, 13 Sep 2013 17:29:53 -0400
> > > Dwight Engen <dwight.engen at oracle.com> wrote:
> > >
> > > > On Fri, 13 Sep 2013 12:09:55 -0400
> > > > S.Çağlar Onur <caglar at 10ur.org> wrote:
> > > >
> > > > > Hi Dwight,
> > > > >
> > > > > Yes, I only observed a hang so far but not this assertion (in fact
> I
> > > > > don't remember ever seeing that). What I'm seeing is this;
> > > >
> > > > Okay, something funny is going on, but I don't know what yet. That
> > > > assertion is coming from liblxc.so->libgnutls->libgcrypt and seems to
> > > > be complaining that we're unlocking something that is already
> > > > unlocked. So I compiled lxc without GNUTLS support (by commenting out
> > > > the check for it in configure.ac) and now I get past that and get
> > > > hangs similar to yours.
> > > >
> > > > Interestingly, I modified your program to just do the create and
> > > > destroy and not the start nor stop and I still get the hangs during
> > > > the creation part.
> > >
> > > Sorry to reply to myself: So now I modified your program to do the
> > > create single threaded and everything else threaded (see attached
> > > patch) and that doesn't hang for me.
> > >
> > > It looked like to me that the hung threads were stuck on process_lock()
> > > so I added a backtrace there to debug it a bit more and it looks like
> > > the hang is in the flow:
> > >
> > > lxcapi_create()
> > > c->save_config()
> > > container_disk_lock()
> > > lxclock()
> > > process_lock()
> > >
> > > Not sure why yet though. Serge, do you think this could have to do with
> > > the fork()ing from a thread in that flow?
> >
> > Well, we've got save_config() doing an fopen without the process_lock()
> > for one thing which is bad. That could explain it in a few ways - we
> > could have corruption due to both tasks changing fdtable at the same
> > time, or perhaps because open() is a cancelation point...
> >
> > Does doing a process_lock() around the fopen and fclose in
> > lxcapi_save_config() fix it?
>
> something like this should be a start:
>
> Subject: api_start: make thread-safe (or at least start to)
>
> Signed-off-by: Serge Hallyn <serge.hallyn at ubuntu.com>
> ---
> src/lxc/lxccontainer.c | 6 ++++++
> src/lxc/parse.c | 4 ++++
> 2 files changed, 10 insertions(+)
>
> diff --git a/src/lxc/lxccontainer.c b/src/lxc/lxccontainer.c
> index 79237df..3c51c4a 100644
> --- a/src/lxc/lxccontainer.c
> +++ b/src/lxc/lxccontainer.c
> @@ -665,7 +665,9 @@ static bool create_container_dir(struct lxc_container
> *c)
> free(s);
> return false;
> }
> + process_lock();
> ret = mkdir(s, 0755);
> + process_unlock();
> if (ret) {
> if (errno == EEXIST)
> ret = 0;
> @@ -1362,11 +1364,15 @@ static bool lxcapi_save_config(struct
> lxc_container *c, const char *alt_file)
> if (lret)
> return false;
>
> + process_lock();
> fout = fopen(alt_file, "w");
> + process_unlock();
> if (!fout)
> goto out;
> write_config(fout, c->lxc_conf);
> + process_lock();
> fclose(fout);
> + process_unlock();
> ret = true;
>
> out:
> diff --git a/src/lxc/parse.c b/src/lxc/parse.c
> index 26cbbdd..f154b89 100644
> --- a/src/lxc/parse.c
> +++ b/src/lxc/parse.c
> @@ -90,7 +90,9 @@ int lxc_file_for_each_line(const char *file, lxc_file_cb
> callback, void *data)
> char *line = NULL;
> size_t len = 0;
>
> + process_lock();
> f = fopen(file, "r");
> + process_unlock();
> if (!f) {
> SYSERROR("failed to open %s", file);
> return -1;
> @@ -104,7 +106,9 @@ int lxc_file_for_each_line(const char *file,
> lxc_file_cb callback, void *data)
>
> if (line)
> free(line);
> + process_lock();
> fclose(f);
> + process_unlock();
> return err;
> }
>
> --
> 1.8.3.2
>
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. Consolidate legacy IT systems to a single system of record for IT
> 2. Standardize and globalize service processes across IT
> 3. Implement zero-touch automation to replace manual, redundant tasks
> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
> _______________________________________________
> Lxc-devel mailing list
> Lxc-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/lxc-devel
>
--
S.Çağlar Onur <caglar at 10ur.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130914/c3af8ea7/attachment.html>
More information about the lxc-devel
mailing list