[lxc-users] Passing infiniband network interface as phys to the LXC
Kean Sum Ooi
keansum at gmail.com
Fri Mar 11 07:25:08 UTC 2016
Hello fellow LXC users!
I have hit a brick wall with a problem of trying to pass an infiniband
network interface inside a container.
The host is Ubuntu 14.04.1 LTS (Trusty Tahr)
$ uname -a
Linux MYHOST 3.19.0-49-generic #55~14.04.1-Ubuntu SMP Fri Jan 22 11:24:31
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
these are the infiniband network interfaces:
$ ifconfig
ib0 Link encap:UNSPEC HWaddr
80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.0.254 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::202:c902:2a:7c31/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:7456285 errors:0 dropped:0 overruns:0 frame:0
TX packets:9528644 errors:0 dropped:130 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:276184857127 (276.1 GB) TX bytes:16792176768 (16.7 GB)
ib1 Link encap:UNSPEC HWaddr
80-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.1.254 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::202:c902:2a:7c32/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:4212 errors:0 dropped:0 overruns:0 frame:0
TX packets:2234 errors:0 dropped:127 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:671192 (671.1 KB) TX bytes:481417 (481.4 KB)
I create a Ubuntu Trust container called "mycontainer" with an alternate
container path set to /home/sum/sumLXC
$ sudo lxc-create -t download -n mycontainer -P /home/sum/sumLXC -- -d
ubuntu -r trusty -a amd64
I add ib0 / ib1 as a phys network type to the mycontainer's config file
/home/sum/sumLXC/mycontainer/config
# ib0 configuration
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = ib0
lxc.network.ipv4 = 192.168.0.63/24
# ib1 configuration
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = ib1
lxc.network.ipv4 = 192.168.1.63/24
I start up the container
$ sudo lxc-start -n mycontainer -P /home/sum/sumLXC
Now ib0 / iib1 disappears from the host. I attach to the container
$ sudo lxc-attach -n mycontainer -P /home/sum/sumLXC
Now inside the container I see ib0 / ib1
root at mycontainer:/# ifconfig
ib0 Link encap:UNSPEC HWaddr
80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.0.63 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::202:c902:2a:7c31/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:7456338 errors:0 dropped:0 overruns:0 frame:0
TX packets:9528739 errors:0 dropped:144 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:276184863867 (276.1 GB) TX bytes:16792190084 (16.7 GB)
ib1 Link encap:UNSPEC HWaddr
80-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.1.63 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::202:c902:2a:7c32/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:4288 errors:0 dropped:0 overruns:0 frame:0
TX packets:2279 errors:0 dropped:141 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:682135 (682.1 KB) TX bytes:489120 (489.1 KB)
The kernel version for the container matches that of the host
root at mycontainer:/# uname -a
Linux mycontainer 3.19.0-49-generic #55~14.04.1-Ubuntu SMP Fri Jan 22
11:24:31 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
I install the infiniband software inside the container
root at mycontainer:/# apt-get install ibverbs-utils opensm infiniband-diags
libopensm5 libibnetdisc5 libibverbs1 libibumad3 libibmad5 libibcommon1
libmlx4-1 libipathver bs1 libmthca1
along with the kernel headers
root at mycontainer:/# apt-get install libmthca-dev linux-headers-3.19.0-49
linux-image-extra-3.19.0-49-generic linux-headers-3.19.0-49-generic
along with infiniband testing software
root at mycontainer:/# apt-get install perftest
next I add the /dev/infiniband special files to the container (done on host)
$ sudo lxc-device -n mycontainer -P /home/sum/sumLXC add
/dev/infiniband/uverbs0
$ sudo lxc-device -n mycontainer -P /home/sum/sumLXC add
/dev/infiniband/issm0
$ sudo lxc-device -n mycontainer -P /home/sum/sumLXC add
/dev/infiniband/issm1
$ sudo lxc-device -n mycontainer -P /home/sum/sumLXC add
/dev/infiniband/rdma_cm
$ sudo lxc-device -n mycontainer -P /home/sum/sumLXC add
/dev/infiniband/umad0
$ sudo lxc-device -n mycontainer -P /home/sum/sumLXC add
/dev/infiniband/umad1
Inside the container, give same permissions as the host
root at mycontainer:/# chmod a+w /dev/infiniband/rdma_cm
root at mycontainer:/# chmod a+w /dev/infiniband/uverbs0
So now inside the container /dev/infiniband exists like this:
root at mycontainer:/# ls -lh /dev/infiniband/
total 0
crw------- 1 root root 231, 64 Mar 10 06:55 issm0
crw------- 1 root root 231, 65 Mar 10 06:56 issm1
crw-rw-rw- 1 root root 10, 56 Mar 10 06:56 rdma_cm
crw------- 1 root root 231, 0 Mar 10 06:56 umad0
crw------- 1 root root 231, 1 Mar 10 06:56 umad1
crw-rw-rw- 1 root root 231, 192 Mar 4 08:01 uverbs0
root at mycontainer:/# ib_read_bw
------------------------------------------------------------------
RDMA_Read BW Test
Connection type : RC
Couldn't get context for mthca0
Now ib_read_bw works fine on the host, but not inside the container. I've
searched for this error but could not solve it myself. Anyone had this same
problem?
PS: The recommended way to use infiniband inside containers is to use
SR-IOV. However our hardware network interface card is Realtek and it
doesn't support SR-IOV so that is why I was hoping adding ib0 / ib1 as phys
could do the trick.
https://git.framasoft.org/AJNOURI/pipework
"Support Infiniband IPoIB
Passing an IPoIB interface to a container is supported. However, the entire
device is moved into the network namespace of the container. It therefore
becomes hidden from the host.
To provide infiniband to multiple containers, use SR-IOV and pass the
virtual function devices to the containers."
Many thanks
Sum
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20160311/538c0775/attachment.html>
More information about the lxc-users
mailing list