[lxc-users] LXD networking between guest, host, and KVM guest
Sean McNamara
smcnam at gmail.com
Wed Oct 7 21:47:41 UTC 2015
I have the following configuration:
- A single /32 IP address on the public Internet, dedicated to the physical box
- A /27 subnet (separate from the /32) on the public Internet, for
use with containers/VMs
- Several Ubuntu 14.04 LXD containers (host OS is Ubuntu 14.04 also)
and a Windows virtual machine on KVM
- Each container and VM gets one /32 out of the /27
- The default gateway that routes traffic to the public Internet is
within the subnet of the /32 primary IP, but is not within the /27
Right now, every container, the host OS, and the Windows VM can all
reach the public Internet without problem.
However, the containers cannot reach each other nor the host, and the
KVM guest cannot reach any of the containers nor the host.
Here's an example from LXD config, where the following placeholders
are used to mask my specific information:
"1.2.3.4" is the single publicly routable IP of the container
"5.6.7.255" is the bcast address for the publicly routable IP of the
physical box
"DEFAULT_GATEWAY" is the default gateway of the host that routes all
traffic to and from the public Internet (this WORKS for the
containers, KVM guest, and host)
"de:ad:be:ef" is a placeholder for a MAC address, which may be
hard-coded in the configuration for my containers, since the default
gateway uses ebtables (ethernet firewall) to whitelist specific MAC
addresses on specific IPs within the /27
"guid" is a placeholder for an actual GUID
"hash" is a placeholder for an md5 or SHA hash.
"MAIN" refers to the primary /32 IP of the physical box
"TBD" refers to the fact that I'm not using IPv6 so it's just
allocating a default IPv6 address (I'm not concerned with setting up
IPv6 until IPv4 works)
"xxx" is just a placeholder for a number
"MEM" is a placeholder for a memory address
"RAND" is a placeholder for the random veth string following the
prefix "veth" (e.g., vethR92AD)
"PRIV" is a placeholder for a private IP (192.168.x.x, 10.0.x.x, etc.)
-------------
name: foo
profiles:
- default
config:
raw.lxc: "lxc.network.ipv4=1.2.3.4/32
5.6.7.255\nlxc.network.ipv4.gateway=DEFAULT_GATEWAY\nlxc.network.hwaddr=de:ad:be:ef\nlxc.network.flags=up
\ \nlxc.network.mtu=1500\n"
volatile.base_image: hash
volatile.eth0.hwaddr: de:ad:be:ef
volatile.last_state.idmap:
'[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
devices:
eth0:
hwaddr: de:ad:be:ef
nictype: bridged
parent: br0
type: nic
ephemeral: false
-----------------
I also have the following ifconfig -a output (truncated to exclude lo
and an unused NIC port), where the placeholders are as follows:
-----------------
br0 Link encap:Ethernet HWaddr de:ad:be:ef
inet addr: MAIN Bcast: 5.6.7.255 Mask:255.255.255.0
inet6 addr: TBD Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:xxx errors:0 dropped:0 overruns:0 frame:0
TX packets:xxx errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:xxx (xxx MB) TX bytes:xxx (xxx MB)
eth0 Link encap:Ethernet HWaddr de:ad:be:ef
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:xxx errors:0 dropped:xxx overruns:0 frame:0
TX packets:xxx errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes: xxx (xxx GB) TX bytes:xxx (xxx MB)
Memory:MEM-MEM
lxcbr0 Link encap:Ethernet HWaddr de:ad:be:ef
inet addr: PRIV Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: TBD Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:xxx errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:xxx (xxx B)
vethRAND1 Link encap:Ethernet HWaddr de:ad:be:ef
inet6 addr: TBD Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:xxx errors:0 dropped:0 overruns:0 frame:0
TX packets:xxx errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:xxx (xxx MB) TX bytes:xxx (xxx MB)
vethRAND2 Link encap:Ethernet HWaddr de:ad:be:ef
inet6 addr: TBD Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:xxx errors:0 dropped:0 overruns:0 frame:0
TX packets:xxx errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:xxx (xxx MB) TX bytes:xxx (xxx MB)
virbr0 Link encap:Ethernet HWaddr de:ad:be:ef
inet addr:PRIV Bcast:PRIV.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
vnet0 Link encap:Ethernet HWaddr de:ad:be:ef
inet6 addr: TBD Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:xxx errors:0 dropped:0 overruns:0 frame:0
TX packets:xxx errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:xxx (xxx KB) TX bytes:xxx (xxx MB)
----------------
And here's ifconfig -a from the perspective of a container, ignoring loopback:
eth0 Link encap:Ethernet HWaddr de:ad:be:ef
inet addr:1.2.3.4 Bcast:5.6.7.255 Mask:255.255.255.255
inet6 addr: TBD Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:xxx errors:0 dropped:0 overruns:0 frame:0
TX packets:xxx errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:xxx (xxx GB) TX bytes:xxx (xxx MB)
----------------
And finally, route -n, from the perspective of the host:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 DEFAULT_GATEWAY 0.0.0.0 UG 0 0 0 br0
PRIV 0.0.0.0 255.255.255.0 U 0 0 0 lxcbr0
MAIN.0 0.0.0.0 255.255.255.0 U 0 0 0 br0
PRIV 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
iptables is wide-open (ACCEPT everything), IPv4 forwarding is enabled,
etc... so I'm kind of stumped as to why I'm not able to route between
the guest and host, and guests and other guests/containers. Any tips
or things I should try?
Thanks,
Sean
More information about the lxc-users
mailing list