[Lxc-users] Containers are all getting same IP address

[Jay Taylor]

Hi Serge,

I added zfs support to the application and systems creating/hosting the
containers, and I have subsequently been unable to reproduce any issues.

As far as trying to reproduce it with btrfs, I've had some success.

The general system state is something like:
N containers already running happily
Launch N+ more containers in rapid succession (in parallell, not serially).

I've modified your test script to reflect more closely what my application
is actually doing, by slowly launching 10 containers, and then using "&" to
rapidly fork and additional 10 clone/start operations.  I have it doing 2
cycles of this and it eventually triggers the problem (it's taken up to 3
runs for to trigger the problem).

And for reference, here is an exact copy the scripts I used to reproduce
the problem:

test.sh:

#!/usr/bin/env bash

prefix=$1

test -z "${prefix}" && echo 'error: missing required parameter: prefix'
1>&2 && exit 1

path=/mnt

sudo lxc-destroy -n c1 2>/dev/null
sudo lxc-create -t ubuntu -B btrfs -n c1

for i in `seq 1 10`; do
    sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
    sudo lxc-start -d -n $prefix$i
done
for i in `seq 11 20`; do
    echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo
lxc-start -d -n $prefix$i) &
done

sleep 10

# Create even more.
for i in `seq 21 30`; do
    sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
    sudo lxc-start -d -n $prefix$i
done
for i in `seq 31 40`; do
    echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo
lxc-start -d -n $prefix$i) &
done


stop.sh:

#!/usr/bin/env bash

prefix=$1

test -z "${prefix}" && echo 'error: missing required parameter: prefix'
1>&2 && exit 1

sudo lxc-destroy -n c1;

for i in `seq 1 40`; do
    echo $(sudo lxc-stop -k -n $prefix$i; sudo lxc-destroy -n $prefix$i) &
done



bash ./test.sh x
bash ./test.sh y
bash ./test.sh z


If it doesn't manifest at first, try stopping/starting varying quantities
of containers for several cycles.  Eventually I consistently end up not
ever getting ip addresses:

x1                          RUNNING  -     -     NO
x10                         RUNNING  -     -     NO
x11                         RUNNING  -     -     NO
x12                         RUNNING  -     -     NO
x13                         RUNNING  -     -     NO
x14                         RUNNING  -     -     NO
x15                         RUNNING  -     -     NO
x16                         RUNNING  -     -     NO
x17                         RUNNING  -     -     NO
x18                         RUNNING  -     -     NO
x19                         RUNNING  -     -     NO
x2                          RUNNING  -     -     NO
x20                         RUNNING  -     -     NO
x21                         RUNNING  -     -     NO
x22                         RUNNING  -     -     NO
x23                         RUNNING  -     -     NO
x24                         RUNNING  -     -     NO
x25                         RUNNING  -     -     NO
x26                         RUNNING  -     -     NO
x27                         RUNNING  -     -     NO
x28                         RUNNING  -     -     NO
x29                         RUNNING  -     -     NO
x3                          RUNNING  -     -     NO
x30                         RUNNING  -     -     NO
x31                         RUNNING  -     -     NO
x32                         RUNNING  -     -     NO
x33                         RUNNING  -     -     NO
x34                         RUNNING  -     -     NO
x35                         RUNNING  -     -     NO
x36                         RUNNING  -     -     NO
x37                         RUNNING  -     -     NO
x38                         RUNNING  -     -     NO
x39                         RUNNING  -     -     NO
x4                          RUNNING  -     -     NO
x40                         RUNNING  -     -     NO
x5                          RUNNING  -     -     NO
x6                          RUNNING  -     -     NO
x7                          RUNNING  -     -     NO
x8                          RUNNING  -     -     NO
x9                          RUNNING  -     -     NO


On Wed, Aug 14, 2013 at 10:12 AM, Serge Hallyn <serge.hallyn at ubuntu.com>wrote:

> Quoting Serge Hallyn (serge.hallyn at ubuntu.com):
> > Quoting Jay Taylor (jay at jaytaylor.com):
> > > After further investigation yesterday, I am not convinced it is an
> > > IP-address issue.  The affected host machines are unable to start any
> > > existing or newly created containers.  The incident that triggered the
> > > issue was cloning 1 container into 10 new ones, and then launching
> them all
> > > simultaneously.  Are there any known concurrency issues with LXC which
> > > would explain why executing a lot of clone/start LXC commands at the
> same
> >
> > Known, no, but that doesn't mean they're not there :)
> >
> > However, could you try to reproduce this with non-btrfs?
> >
> > I'll try to reproduce with btrfs...
>
> In a fresh raring instance I mounted a btrfs disk on /mnt, and did
>
>         lxc-create -t ubuntu -B btrfs -P /mnt -n c1
>         for i in `seq 1 10`; do
>                 lxc-clone -s -p /mnt -o c1 -n x$i
>         done
>         for i in `seq 1 10`; do
>                 lxc-start -d -P /mnt -n x$i
>         done
>
>         Then connected to two of the containers with lxc-console,
>                 lxc-console -P /mnt -n x2
>                 lxc-console -P /mnt -n x9
>
>         both were up and had unique ip addresses.
>
> Again this was a raring instance with ppa:ubuntu-lxc/daily installed.
>
> -serge
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20130814/36f500a1/attachment.html>