[lxc-users] Route to Containers
Stephen Nelson-Smith
stephen at atalanta-systems.com
Tue Feb 16 18:31:31 UTC 2016
Hi,
I might be overlooking something very obvious here, but I appear to
have a routing problem on my new (CentOS 7) server, which I don't have
on any of my old (CentOS 6) ones.
Desired outcome:
- Containers have their own routable IP addresses which can be reached
from any machine on the same subnet, or with a route to that subnet
- Specifically: on the physical host I should be able to ssh to the IP
of a container, and connect, and from other containers on other
machines on the same subnet, I should be able to connect.
Current state:
- LXC is installed and working
- A bridge is configured on the physical host
- The LXC containers have physical IPs on the same subnet as the
physical host (10.1.1.0/24)
- Networking / routing on the containers works - I can install packages
- Ssh is running on the container
- Other machines on the network can ping the centos 7 containers, but
cannot ssh to them
- I cannot ssh to the containers from the physical host
- I can ssh to the container from my laptop (which has a route to the subnet)
Evidence:
1) LXC installation:
[root at localhost ~]# rpm -q lxc
lxc-1.0.8-1.el7.x86_64
[root at localhost ~]# rpm -q lxc-templates
lxc-templates-1.0.8-1.el7.x86_64
[root at localhost ~]# lxc-checkconfig
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-3.10.0-327.4.5.el7.x86_64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled
--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled
--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
Bridges: enabled
Advanced netfilter: enabled
CONFIG_NF_NAT_IPV4: enabled
CONFIG_NF_NAT_IPV6: enabled
CONFIG_IP_NF_TARGET_MASQUERADE: enabled
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled
--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: enabled
Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig
2) The bridge is configured
[root at localhost ~]# nmcli -p connection show
=========================================================================
NetworkManager connection profiles
=========================================================================
NAME UUID TYPE DEVICE
-------------------------------------------------------------------------
br0-port1 6a970e7b-ce84-4bda-8432-9fc08f50d85b 802-3-ethernet enp1s0
enp1s0 d5c83edf-4ec5-4349-ba5d-ab03b73eee11 802-3-ethernet --
enp2s0 b5caeeaf-2e23-45da-9f43-1cc021832fbf 802-3-ethernet --
br0 d4ae2142-1100-4c54-a725-7cf0b46774a5 bridge br0
br0-port2 22bcb7fd-b0c3-439b-be25-c83b6244d046 802-3-ethernet enp2s0
[root at localhost ~]# nmcli -p connection show br0
===============================================================================
Connection profile details (br0)
===============================================================================
connection.id: br0
connection.uuid: d4ae2142-1100-4c54-a725-7cf0b46774a5
connection.interface-name: br0
connection.type: bridge
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1455650188
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.autoconnect-slaves: -1 (default)
connection.secondaries:
connection.gateway-ping-timeout: 0
connection.metered: unknown
-------------------------------------------------------------------------------
ipv4.method: manual
ipv4.dns:
ipv4.dns-search:
ipv4.addresses: 10.1.1.230/24
ipv4.gateway: 10.1.1.1
ipv4.routes:
ipv4.route-metric: -1
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: no
ipv4.dhcp-client-id: --
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.never-default: no
ipv4.may-fail: yes
-------------------------------------------------------------------------------
ipv6.method: auto
ipv6.dns:
ipv6.dns-search:
ipv6.addresses:
ipv6.gateway: --
ipv6.routes:
ipv6.route-metric: -1
ipv6.ignore-auto-routes: no
ipv6.ignore-auto-dns: no
ipv6.never-default: no
ipv6.may-fail: yes
ipv6.ip6-privacy: -1 (unknown)
ipv6.dhcp-send-hostname: yes
ipv6.dhcp-hostname: --
-------------------------------------------------------------------------------
bridge.mac-address: --
bridge.stp: yes
bridge.priority: 32768
bridge.forward-delay: 15
bridge.hello-time: 2
bridge.max-age: 20
bridge.ageing-time: 300
-------------------------------------------------------------------------------
===============================================================================
Activate connection details (d4ae2142-1100-4c54-a725-7cf0b46774a5)
===============================================================================
GENERAL.NAME: br0
GENERAL.UUID: d4ae2142-1100-4c54-a725-7cf0b46774a5
GENERAL.DEVICES: br0
GENERAL.STATE: activated
GENERAL.DEFAULT: yes
GENERAL.DEFAULT6: no
GENERAL.VPN: no
GENERAL.ZONE: --
GENERAL.DBUS-PATH:
/org/freedesktop/NetworkManager/ActiveConnection/0
GENERAL.CON-PATH:
/org/freedesktop/NetworkManager/Settings/4
GENERAL.SPEC-OBJECT: /
GENERAL.MASTER-PATH: --
-------------------------------------------------------------------------------
IP4.ADDRESS[1]: 10.1.1.230/24
IP4.GATEWAY: 10.1.1.1
-------------------------------------------------------------------------------
IP6.ADDRESS[1]: fe80::21e:c9ff:fe4f:ba7b/64
IP6.GATEWAY:
-------------------------------------------------------------------------------
[root at localhost ~]# ip r l
default via 10.1.1.1 dev br0 proto static metric 425
10.1.1.0/24 dev br0 proto kernel scope link src 10.1.1.230 metric 425
[root at localhost ~]# ping 10.1.1.1
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_seq=1 ttl=255 time=0.520 ms
64 bytes from 10.1.1.1: icmp_seq=2 ttl=255 time=0.426 ms
^C
--- 10.1.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.426/0.473/0.520/0.047 ms
[root at localhost ~]# ping 10.1.1.103
PING 10.1.1.103 (10.1.1.103) 56(84) bytes of data.
64 bytes from 10.1.1.103: icmp_seq=1 ttl=64 time=1.40 ms
64 bytes from 10.1.1.103: icmp_seq=2 ttl=64 time=0.345 ms
^C
--- 10.1.1.103 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.345/0.875/1.406/0.531 ms
3) Containers have IP addresses:
[root at localhost ~]# grep -v ^# /var/lib/lxc/example/config
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = fe:2d:e2:8b:83:1e
lxc.rootfs = /var/lib/lxc/example/rootfs
lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.arch = x86_64
lxc.utsname = example
lxc.autodev = 1
lxc.network.ipv4 = 10.1.1.233
lxc.network.name = eth0
lxc.network.ipv4.gateway = 10.1.1.1
lxc.kmsg = 0
[root at localhost ~]# lxc-start -d -n example
[root at localhost ~]# lxc-attach -n example
[root at example ~]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
5: eth0 at if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP qlen 1000
link/ether fe:2d:e2:8b:83:1e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.1.1.233/0 brd 255.255.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::fc2d:e2ff:fe8b:831e/64 scope link
valid_lft forever preferred_lft forever
[root at example ~]# ip r l
default via 10.1.1.1 dev eth0
10.1.1.1 dev eth0 scope link
169.254.0.0/16 dev eth0 scope link metric 1005
[root at example ~]# ping 10.1.1.1
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_seq=1 ttl=255 time=0.808 ms
^C
--- 10.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.808/0.808/0.808/0.000 ms
[root at example ~]# ping 10.1.1.103
PING 10.1.1.103 (10.1.1.103) 56(84) bytes of data.
64 bytes from 10.1.1.103: icmp_seq=1 ttl=64 time=1.59 ms
^C
--- 10.1.1.103 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.596/1.596/1.596/0.000 ms
4) Networking on containers works
[root at example ~]# yum install zsh
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.coreix.net
* extras: mirrors.coreix.net
* updates: mirror.mhd.uk.as44574.net
Resolving Dependencies
--> Running transaction check
---> Package zsh.x86_64 0:5.0.2-14.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===============================================================================================================================================================================================
Package Arch
Version
Repository Size
===============================================================================================================================================================================================
Installing:
zsh x86_64
5.0.2-14.el7
base 2.4 M
Transaction Summary
===============================================================================================================================================================================================
Install 1 Package
Total download size: 2.4 M
Installed size: 5.6 M
Is this ok [y/d/N]: y
Downloading packages:
zsh-5.0.2-14.el7.x86_64.rpm
| 2.4 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : zsh-5.0.2-14.el7.x86_64
1/1
Verifying : zsh-5.0.2-14.el7.x86_64
1/1
Installed:
zsh.x86_64 0:5.0.2-14.el7
Complete!
5) Ssh is running on the container
[root at example ~]# systemctl status sshd
● sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled;
vendor preset: enabled)
Active: active (running) since Tue 2016-02-16 19:25:30 UTC; 46min ago
Docs: man:sshd(8)
man:sshd_config(5)
Main PID: 28 (sshd)
CGroup: /user.slice/user-0.slice/session-287.scope/system.slice/sshd.service
└─28 /usr/sbin/sshd -D
Feb 16 19:25:30 example systemd[1]: Started OpenSSH server daemon.
Feb 16 19:25:30 example systemd[1]: Starting OpenSSH server daemon...
Feb 16 19:25:30 example sshd[28]: Server listening on 0.0.0.0 port 22.
Feb 16 19:25:30 example sshd[28]: Server listening on :: port 22.
Feb 16 19:39:27 example sshd[312]: Invalid user sns from 192.168.168.67
Feb 16 19:39:27 example sshd[312]: input_userauth_request: invalid
user sns [preauth]
Feb 16 19:39:29 example sshd[312]: Connection closed by 192.168.168.67 [preauth]
Feb 16 19:49:25 example sshd[314]: Connection closed by
192.168.168.231 [preauth]
6) Other machines on the network can ping but not ssh
[root at sns00 ~]# ping 10.1.1.233
PING 10.1.1.233 (10.1.1.233) 56(84) bytes of data.
64 bytes from 10.1.1.233: icmp_seq=1 ttl=64 time=1.29 ms
64 bytes from 10.1.1.233: icmp_seq=2 ttl=64 time=0.364 ms
^C
--- 10.1.1.233 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1214ms
rtt min/avg/max/mdev = 0.364/0.828/1.293/0.465 ms
[root at sns00 ~]# ssh 10.1.1.233
^C
[root at sns00 ~]# ssh 10.1.1.233 -v
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 10.1.1.233 [10.1.1.233] port 22.
^C
7) I can connect from my laptop
sns at Stephens-MacBook-Pro ~> ssh -l root 10.1.1.233
root at 10.1.1.233's password:
Last login: Tue Feb 16 20:15:10 2016 from 192.168.168.67
[root at example ~]#
I'm rather puzzled.
Any thoughts?
S.
More information about the lxc-users
mailing list