<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 27, 2016 at 4:38 PM, Guido Jäkel <span dir="ltr"><<a href="mailto:G.Jaekel@dnb.de" target="_blank">G.Jaekel@dnb.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Dear Joshua,<br>
<br>
you wrote, that there's a trunk on eth1 and eth2. But for eth2, i can't see any VLAN (501 ?) detrunking as with eth1 & eth1.500. In the other hand you wrote, that eth2 is working. Are you shure, that you realy receive this trunk of 3 VLANs on your both eth's?<br></blockquote><div><br></div><div>I started to think about this as well and I've found the reason. VMware allows you to tag NICs from the hypervisor level. Eth1 and eth2 were both setup under VLAN 500 so that is why no tagging on the LXC host was required, hence why eth2 worked. So the lesson there is don't mix dot1q, either set it on the hypervisor and leave it completely out of the LXC host and container or vice versa.</div><div><br></div><div>I've completely removed VLAN tagging from my LXC host and making progress, but still running into odd situations:</div><div><br></div><div><div>lxcuser@prvlxc01:~$ sudo ip -d link show</div><div>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default</div><div> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0</div><div>2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000</div><div> link/ether 00:50:56:be:13:94 brd ff:ff:ff:ff:ff:ff promiscuity 0</div><div>3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0-500 state UP mode DEFAULT group default qlen 1000</div><div> link/ether 00:50:56:be:46:c5 brd ff:ff:ff:ff:ff:ff promiscuity 1</div><div> bridge_slave</div><div>4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000</div><div> link/ether 00:50:56:be:26:4f brd ff:ff:ff:ff:ff:ff promiscuity 0</div><div>5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000</div><div> link/ether 00:50:56:be:01:d8 brd ff:ff:ff:ff:ff:ff promiscuity 0</div><div>6: br0-500: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default</div><div> link/ether 00:50:56:be:46:c5 brd ff:ff:ff:ff:ff:ff promiscuity 0</div><div> bridge</div><div>7: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default</div><div> link/ether de:ef:8c:53:01:0b brd ff:ff:ff:ff:ff:ff promiscuity 0</div><div> bridge</div><div>9: vethKAG02C: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0-500 state UP mode DEFAULT group default qlen 1000</div><div> link/ether fe:bf:b5:cf:f0:83 brd ff:ff:ff:ff:ff:ff promiscuity 1</div><div> veth</div><div> bridge_slave</div></div><div><br></div><div><b>Scenario 1</b>: When assigning an IP directly to eth1 on the host, no bridging involved, no containers involved (Success):</div><div><br></div><div>/etc/network/interfaces</div><div><div>auto eth1</div><div>iface eth1 inet static</div><div> address <a href="http://10.240.78.3/24">10.240.78.3/24</a></div></div><div><br></div><div>route -n</div><div>10.240.78.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1<br></div><div><br></div><div><div>PING 10.240.78.1 (10.240.78.1) 56(84) bytes of data.</div><div>64 bytes from <a href="http://10.240.78.1">10.240.78.1</a>: icmp_seq=1 ttl=255 time=8.25 ms</div><div>64 bytes from <a href="http://10.240.78.1">10.240.78.1</a>: icmp_seq=2 ttl=255 time=2.59 ms</div><div>^C</div><div>--- 10.240.78.1 ping statistics ---</div><div>2 packets transmitted, 2 received, 0% packet loss, time 1001ms</div><div>rtt min/avg/max/mdev = 2.597/5.425/8.254/2.829 ms</div></div><div><br></div><div><b>Scenario 2</b>: When assigning an IP to a bridge and making eth1 a slave to the bridge, no containers involved (Success):</div><div><br></div><div>/etc/network/interfaces</div><div><div>auto eth1</div><div>iface eth1 inet manual</div><div><br></div><div>auto br0-500<br></div><div>iface br0-500 inet static</div><div> address <a href="http://10.240.78.3/24">10.240.78.3/24</a></div><div> bridge_ports eth1</div><div> bridge_stp off</div><div> bridge_fd 0</div><div> bridge_maxwait 0</div></div><div><br></div><div><br></div><div>route -n</div><div>10.240.78.0 0.0.0.0 255.255.255.0 U 0 0 0 br0-500<br></div><div><br></div><div><div>PING 10.240.78.1 (10.240.78.1) 56(84) bytes of data.</div><div>64 bytes from <a href="http://10.240.78.1">10.240.78.1</a>: icmp_seq=1 ttl=255 time=3.26 ms</div><div>64 bytes from <a href="http://10.240.78.1">10.240.78.1</a>: icmp_seq=2 ttl=255 time=1.51 ms</div><div>64 bytes from <a href="http://10.240.78.1">10.240.78.1</a>: icmp_seq=3 ttl=255 time=2.30 ms</div><div>^C</div><div>--- 10.240.78.1 ping statistics ---</div><div>3 packets transmitted, 3 received, 0% packet loss, time 2003ms</div><div>rtt min/avg/max/mdev = 1.514/2.360/3.262/0.715 ms</div></div><div><br></div><div><b>Scenario 3</b>: Same scenario as above, expect the bridge is not assigned and IP and a container is created and connects to the same bridge (Failure):</div><div><br></div><div>/etc/network/interfaces</div><div>auto eth1<br></div><div><div>iface eth1 inet manual</div><div><br></div><div>auto br0-500</div><div>iface br0-500 inet static</div><div> bridge_ports eth1</div><div> bridge_stp off</div><div> bridge_fd 0</div><div> bridge_maxwait 0</div></div><div><br></div><div>~/.local/share/lxc/c4/config</div><div><div># Network configuration</div><div>lxc.network.type = veth</div><div>lxc.network.link = br0-500</div><div>lxc.network.ipv4 = <a href="http://10.240.78.3/24">10.240.78.3/24</a></div><div>lxc.network.ipv4.gateway = 10.240.78.1</div><div>lxc.network.flags = up</div><div>lxc.network.hwaddr = 00:16:3e:f7:0a:83</div></div><div><br></div><div>route -n (on host)</div><div>10.240.78.0 0.0.0.0 255.255.255.0 U 0 0 0 br0-500<br></div><div><br></div><div>route -n (inside container)</div><div><div>10.240.78.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0<br></div></div><div><br></div><div>ping (on host)</div><div><div>PING 10.240.78.1 (10.240.78.1) 56(84) bytes of data.</div><div>64 bytes from <a href="http://10.240.78.1">10.240.78.1</a>: icmp_seq=1 ttl=255 time=1.12 ms</div><div>64 bytes from <a href="http://10.240.78.1">10.240.78.1</a>: icmp_seq=2 ttl=255 time=1.17 ms</div><div>64 bytes from <a href="http://10.240.78.1">10.240.78.1</a>: icmp_seq=3 ttl=255 time=6.54 ms</div><div>^C</div><div>--- 10.240.78.1 ping statistics ---</div><div>3 packets transmitted, 3 received, 0% packet loss, time 2002ms</div><div>rtt min/avg/max/mdev = 1.125/2.950/6.548/2.544 ms</div></div><div><br></div><div>ping (inside container)</div><div><div>PING 10.240.78.1 (10.240.78.1) 56(84) bytes of data.</div><div>From 10.240.78.3 icmp_seq=1 Destination Host Unreachable</div><div>From 10.240.78.3 icmp_seq=2 Destination Host Unreachable</div><div>From 10.240.78.3 icmp_seq=3 Destination Host Unreachable</div><div>^C</div><div>--- 10.240.78.1 ping statistics ---</div><div>4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3013ms</div></div><div><br></div><div>Here is the odd part, if I sniff the traffic on the bridge from the host I can see the container arping for the gateway and getting a response. However nothing is being added to my arp table on the container.</div><div><br></div><div><br></div><div><div>lxcuser@prvlxc01:~$ su root -c "tcpdump -i br0-500 -Uw - | tcpdump -en -r - arp" #this is the host</div><div>Password:</div><div>reading from file -, link-type EN10MB (Ethernet)</div><div>tcpdump: listening on br0-500, link-type EN10MB (Ethernet), capture size 262144 bytes</div><div>17:32:10.223168 00:16:3e:f7:0a:83 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.240.78.1 tell 10.240.78.3, length 28</div><div>17:32:10.223337 00:16:3e:f7:0a:83 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.240.78.1 tell 10.240.78.3, length 46</div><div>17:32:10.225821 00:13:c4:f2:64:4d > 00:16:3e:f7:0a:83, ethertype ARP (0x0806), length 60: Reply 10.240.78.1 is-at 00:13:c4:f2:64:4d, length 46</div><div>17:32:11.220216 00:16:3e:f7:0a:83 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.240.78.1 tell 10.240.78.3, length 28</div><div>17:32:11.220418 00:16:3e:f7:0a:83 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 10.240.78.1 tell 10.240.78.3, length 46</div><div>17:32:11.230455 00:13:c4:f2:64:4d > 00:16:3e:f7:0a:83, ethertype ARP (0x0806), length 60: Reply 10.240.78.1 is-at 00:13:c4:f2:64:4d, length 46</div><div><br></div></div><div>arp -n (from container)</div><div><div>Address HWtype HWaddress Flags Mask Iface</div><div>10.240.78.1 (incomplete) eth0</div></div><div><br></div><div>If I manually add the gateway's MAC address to the arp table I can ping it and have full internet access! Anybody know why the container isn't adding the MAC address when a response is being given?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
I'm using a (working) comparable setup: On the host, eth0 is used for host management on a detrunked port. On eth1, there's a trunk with the needed VLANs for different network for a staged environment. On eth1, there is a VLAN decoder for each of the needed VLANs. And this is attached to a seperate software bridge for each VLAN. A container's outside veth is attached for the appropriate bridge - this is done in a start script by a calculated configuration statement based on the container's name.</blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">But the lxc host is located on plain hardware, not in a VM.</blockquote><div><br></div><div>That's basically what I have in my home lab (and everything working successfully there) and what I'm trying to reproduce here, unfortunately I don't have the pull here to get a physical LXC host so gotta work with what I got.</div><div><br></div><div>Thanks,</div><div>Joshua</div></div><br></div></div>