[Lxc-users] Networking fails sporadically in containers on ubuntu server 12.10

David Parks davidparks21 at yahoo.com
Wed Apr 3 06:27:06 UTC 2013


I take it back, I'm still reproducing the issue. I just stopped 5 LXC
containers, cloned 5 from a base image, and re-set the IP to the same as the
original 5 were using (e.g. the MAC changed as I don't have that defined in
the config file). Restarting the first one now and it won't ping the router
or get out to the internet, and there's a ~30 sec delay SSHing to it from
the host.  At this same time I have one other LXC container running that
works (this one starting working after I changed the MAC of the NAT
interface in virtualbox, eth0 on the host OS).

 

 

From: David Parks [mailto:davidparks21 at yahoo.com] 
Sent: Wednesday, April 03, 2013 12:13 PM
To: 'lxc-users at lists.sourceforge.net'
Subject: RE: [Lxc-users] Networking fails sporadically in containers on
ubuntu server 12.10

 

I should also add to this that I just changed the MAC address of the
virtualbox NAT interface to 22:00:22:00:22:00 as Serge had previously
suggested. It's working now, though I can be confident that changing the
virtualbox NAT's MAC was the solution or if it's just intermittently working
now. I'll make another update in a few days if it continues to work, or if
it fails again.

 

Thanks, by the way, for the super help!!

 

Dave

 

 

From: David Parks [mailto:davidparks21 at yahoo.com] 
Sent: Wednesday, April 03, 2013 11:38 AM
To: lxc-users at lists.sourceforge.net
Subject: Re: [Lxc-users] Networking fails sporadically in containers on
ubuntu server 12.10

 

Yesterday I couldn't reproduce the issue with 2 LXC containers running, but
today I've rebooted everything and I have one LXC container running and it
can't access the router or internet (but can access the host IP and other
LXC containers).

 

Right now the br0 MAC is configured to be the same as the eth0 MAC address.

 

Is it odd that, on the host OS, when I check the ARP table the mac address
for the LXC IP (10.1.3.45) is seen as the MAC *inside* the container, not of
vethR1s1ny on the host. But maybe that's expected behavior?

 

Also notable, from the host OS I can ping 10.1.3.45 (the LXC containers IP),
but when I SSH to it there is around a 30 second delay before the SSH login
prompt comes up. When things are working there is no such lag.

 

I turned on debug logging like Serge suggested, but don't see anything
interesting in there (also below).

 

David

 

IFCONFIG / BRCTL SHOW / ARP -A

davidparks21 at ubuntuserver:~$ ifconfig

br0       Link encap:Ethernet  HWaddr 08:00:27:ca:5f:7a

          inet addr:10.1.0.5  Bcast:10.1.255.255  Mask:255.255.0.0

          inet6 addr: fe80::a00:27ff:feca:5f7a/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:983 errors:0 dropped:0 overruns:0 frame:0

          TX packets:949 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:87727 (87.7 KB)  TX bytes:108287 (108.2 KB)

 

eth0      Link encap:Ethernet  HWaddr 08:00:27:ca:5f:7a

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:883 errors:0 dropped:0 overruns:0 frame:0

          TX packets:725 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:97519 (97.5 KB)  TX bytes:79510 (79.5 KB)

 

lo        Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0

          inet6 addr: ::1/128 Scope:Host

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:7 errors:0 dropped:0 overruns:0 frame:0

          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:616 (616.0 B)  TX bytes:616 (616.0 B)

 

vethR1s1ny Link encap:Ethernet  HWaddr fe:d5:4a:f2:07:db

          inet6 addr: fe80::fcd5:4aff:fef2:7db/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:247 errors:0 dropped:0 overruns:0 frame:0

          TX packets:367 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:28718 (28.7 KB)  TX bytes:37972 (37.9 KB)

 

davidparks21 at ubuntuserver:~$ brctl show

bridge name     bridge id               STP enabled     interfaces

br0             8000.080027ca5f7a       no              eth0

                                                        vethR1s1ny

davidparks21 at ubuntuserver:~$ arp -a

? (10.1.0.2) at 52:54:00:12:35:02 [ether] on br0

? (10.1.3.45) at 26:b2:18:b5:fc:e7 [ether] on br0

 

 

LXC DEBUG LOG

davidparks21 at ubuntuserver:~/lxc$ cat lxc.debug

      lxc-start 1364962992.454 INFO     lxc_conf - tty's configured

      lxc-start 1364962992.454 INFO     lxc_start - 'cloudera-manager' is
initialized

      lxc-start 1364962992.560 INFO     lxc_conf - opened
/var/lib/lxc/cloudera-manager/rootfs.hold as fd 18

 

      lxc-start 1364962992.562 INFO     lxc_cgroup - [1] found cgroup
mounted at '/sys/fs/cgroup/cpuset',opts='rw,relatime,cpuset,clone_children'

      lxc-start 1364962992.733 INFO     lxc_cgroup - created cgroup
'/sys/fs/cgroup/cpuset//lxc/cloudera-manager'

      lxc-start 1364962992.733 INFO     lxc_cgroup - [2] found cgroup
mounted at '/sys/fs/cgroup/cpu',opts='rw,relatime,cpu,clone_children'

      lxc-start 1364962992.743 INFO     lxc_cgroup - created cgroup
'/sys/fs/cgroup/cpu//lxc/cloudera-manager'

      lxc-start 1364962992.743 INFO     lxc_cgroup - [3] found cgroup
mounted at
'/sys/fs/cgroup/cpuacct',opts='rw,relatime,cpuacct,clone_children'

      lxc-start 1364962992.749 INFO     lxc_cgroup - created cgroup
'/sys/fs/cgroup/cpuacct//lxc/cloudera-manager'

      lxc-start 1364962992.749 INFO     lxc_cgroup - [4] found cgroup
mounted at '/sys/fs/cgroup/memory',opts='rw,relatime,memory,clone_children'

      lxc-start 1364962992.757 INFO     lxc_cgroup - created cgroup
'/sys/fs/cgroup/memory//lxc/cloudera-manager'

      lxc-start 1364962992.757 INFO     lxc_cgroup - [5] found cgroup
mounted at
'/sys/fs/cgroup/devices',opts='rw,relatime,devices,clone_children'

      lxc-start 1364962992.769 INFO     lxc_cgroup - created cgroup
'/sys/fs/cgroup/devices//lxc/cloudera-manager'

      lxc-start 1364962992.769 INFO     lxc_cgroup - [6] found cgroup
mounted at
'/sys/fs/cgroup/freezer',opts='rw,relatime,freezer,clone_children'

      lxc-start 1364962992.770 INFO     lxc_cgroup - created cgroup
'/sys/fs/cgroup/freezer//lxc/cloudera-manager'

      lxc-start 1364962992.770 INFO     lxc_cgroup - [7] found cgroup
mounted at '/sys/fs/cgroup/blkio',opts='rw,relatime,blkio,clone_children'

      lxc-start 1364962992.771 INFO     lxc_cgroup - created cgroup
'/sys/fs/cgroup/blkio//lxc/cloudera-manager'

      lxc-start 1364962992.771 INFO     lxc_cgroup - [8] found cgroup
mounted at
'/sys/fs/cgroup/perf_event',opts='rw,relatime,perf_event,clone_children'

      lxc-start 1364962992.772 INFO     lxc_cgroup - created cgroup
'/sys/fs/cgroup/perf_event//lxc/cloudera-manager'

      lxc-start 1364962992.774 INFO     lxc_conf - 'cloudera-manager'
hostname has been setup

      lxc-start 1364962992.778 INFO     lxc_conf - network has been setup

      lxc-start 1364962992.778 INFO     lxc_conf - mount points have been
setup

      lxc-start 1364962992.778 INFO     lxc_conf - mount points have been
setup

      lxc-start 1364962992.781 INFO     lxc_conf - cgroup has been setup

      lxc-start 1364962992.781 INFO     lxc_conf - created
/usr/lib/x86_64-linux-gnu/lxc/dev/lxc

 

      lxc-start 1364962992.781 INFO     lxc_conf - console has been setup on
lxc/console

      lxc-start 1364962992.781 INFO     lxc_conf - 4 tty(s) has been setup

      lxc-start 1364962992.781 INFO     lxc_conf - rootfs path is
./var/lib/lxc/cloudera-manager/rootfs., mount is
./usr/lib/x86_64-linux-gnu/lxc.

      lxc-start 1364962992.781 INFO     lxc_conf - I am 1, /proc/self points
to 1

 

      lxc-start 1364962992.782 INFO     lxc_conf - created new pts instance

      lxc-start 1364962992.782 INFO     lxc_conf - set personality to '0x0'

      lxc-start 1364962992.782 NOTICE   lxc_conf - 'cloudera-manager' is
setup.

      lxc-start 1364962992.782 INFO     lxc_start - changed apparmor profile
to lxc-container-default

      lxc-start 1364962992.782 NOTICE   lxc_start - exec'ing '/sbin/init'

      lxc-start 1364962992.783 NOTICE   lxc_start - '/sbin/init' started
with pid '3333'

      lxc-start 1364962992.783 WARN     lxc_console - console input disabled

      lxc-start 1364962992.783 WARN     lxc_start - invalid pid for SIGCHLD

 

 

-----Original Message-----
From: Hans Feldt [mailto:hans.feldt at ericsson.com] 
Sent: Tuesday, April 02, 2013 9:46 PM
To: Serge Hallyn
Cc: David Parks; lxc-users at lists.sourceforge.net
Subject: Re: [Lxc-users] Networking fails sporadically in containers on
ubuntu server 12.10

 

Could it be the issue described in

 
<http://backreference.org/2010/07/28/linux-bridge-mac-addresses-and-dynamic-
ports/>
http://backreference.org/2010/07/28/linux-bridge-mac-addresses-and-dynamic-p
orts/

?

 

I encountered that one and solved it by creating a dummy interface with the
same life cycle as the bridge.

 

thanks,

Hans

 

On 04/01/2013 02:34 PM, Serge Hallyn wrote:

> Quoting David Parks ( <mailto:davidparks21 at yahoo.com>
davidparks21 at yahoo.com):

>> davidparks21 at hostOS:~$ arp -a

>> 

>> ? (10.1.0.4) at c6:63:bb:a7:d8:60 [ether] on br0

>> 

>> ? (10.1.0.2) at 52:54:00:12:35:02 [ether] on br0

>> 

>> ? (10.1.0.45) at 32:8c:fc:c1:7f:e5 [ether] on br0

> 

> Can you set the macaddr on your virtualbox nic to something lower, 

> like something starting with 22: ?

> 

> 32: sounds really low for a veth peer nic, I'm surprised - mine are 

> always fe:.  In fact, looking at 

> src/lxc/conf.c:setup_private_host_hw_addr(), it makes sure to set 0xfe 

> in the first two bytes.  So why are you getting 32: ?  Hmmm.

> 

> Could you also try adding '-l info -o lxc.debug' to your lxc-start 

> command on a failing container, and send the resulting lxc.debug file?  

> We're looking for output from:

> 

>          DEBUG("mac address of host interface '%s' changed to private "

>                           "%02x:%02x:%02x:%02x:%02x:%02x", veth1,

> 

> thanks,

> -serge

> 

> ----------------------------------------------------------------------

> -------- Own the Future-Intel® Level Up Game Demo Contest 2013 

> Rise to greatness in Intel's independent game demo contest.

> Compete for recognition, cash, and the chance to get your game on 

> Steam. $5K grand prize plus 10 genre and skill prizes.

> Submit your demo by 6/6/13.  <http://p.sf.net/sfu/intel_levelupd2d>
http://p.sf.net/sfu/intel_levelupd2d

> _______________________________________________

> Lxc-users mailing list

>  <mailto:Lxc-users at lists.sourceforge.net> Lxc-users at lists.sourceforge.net

>  <https://lists.sourceforge.net/lists/listinfo/lxc-users>
https://lists.sourceforge.net/lists/listinfo/lxc-users

> 

> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20130403/63113485/attachment.html>


More information about the lxc-users mailing list