[lxc-devel] Network interface - netns migration bug

[Bogdan Purcareata]

Hello,

I've encountered this bug with lxc-0.9.0, using the lxc-phys scenario - assigning a physical interface to a container.

In my understanding, this is what happens when a container starts (well, the relevant parts):
- the configuration file is parsed into a config structure - the kernel interface index is found using if_nametoindex(). This ifindex will be further used to move the interface from the host netns to the container's.
- the child process is cloned in a new netns;
- the parent issues a RTM_NEWLINK message to create a new interface in the netns associated with the pid of the child, having the known ifindex;
- it is implied that the interface will be removed from the host namespace;
- the child process will further use the same ifindex in order to rename the interface as needed; it is assumed that the interface will keep the same ifindex in the container network namespaces.

Here's my scenario - I'm using a small binary [1] that lists all the known interfaces and their respective ifindexes.

- the following interfaces are available on host:
Interface 1 : lo
Interface 2 : fm1-mac5
Interface 3 : fm2-mac5
Interface 4 : eth2
Interface 5 : tunl0
Interface 6 : sit0

- the following virtual interfaces are created when lxc-unsharing a process in a new netns:
Interface 1 : lo
Interface 2 : tunl0
Interface 3 : sit0

At this point, I want to create a container that has fm1-mac5 physically assigned. lxc-start fails with this message:
lxc-start: failed to rename tunl0->fm1-mac5 : File exists
lxc-start: failed to setup netdev
lxc-start: failed to setup the network for 'foo'
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'foo'

By adding a few printf()s in the code, I was able to find the following facts:
- fm1-mac5 will have "2" as associated ifindex
- the new child process will be cloned with a new netns, with interfaces lo, tunl0 and sit0
- when issuing the RTM_NEWLINK netlink message, ifindex 2 (fm2-mac5) will be moved to the new netns, _however_ it will receive ifindex 4, since all previous ones are occupied
- when trying to rename the interface represented by ifindex 2 (tunl0) to "fm1-mac5", it will fail, because the interface is already there and it has ifindex 4

After container start fails, here are the interfaces on the host again:
Interface 1 : lo
Interface 3 : fm2-mac5
Interface 4 : eth2
Interface 5 : tunl0
Interface 6 : sit0
Interface 7 : fm1-mac5

fm1-mac5 now has ifindex 7 - when moving fm1-mac5 (having ifindex 4 in the container netns) back to the host netns, it will receive a new ifindex, since 4 is occupied (eth2).

Running lxc-start again will succeed, and the following interfaces will be present in the container:
Interface 1 : lo
Interface 2 : tunl0
Interface 3 : sit0
Interface 7 : fm1-mac5

There was no problem assigning ifindex 7 to fm1-mac5 in the new netns, since it was used by no other interface.

I'm assuming that this doesn't manifest itself with virtual interfaces, since they receive high ifindexes (after all other interfaces) at creation time, with no possibility to clash with a new netns' default interfaces. This should only appear in the LXC_NET_PHYS case.

In this case, one thing that can be done is to force re-reading the ifindex, based on netdev->link:
---
 src/lxc/conf.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/src/lxc/conf.c b/src/lxc/conf.c
index 6b3f318..ff6b30b 100644
--- a/src/lxc/conf.c
+++ b/src/lxc/conf.c
@@ -1846,6 +1846,14 @@ static int setup_netdev(struct lxc_netdev *netdev)
        return 0;
      }

+     /* if LXC_NET_PHYS, update the ifindex */
+     if (netdev->type == LXC_NET_PHYS)
+       if (!(netdev->ifindex = if_nametoindex(netdev->link))) {
+         ERROR("unable to retrieve index for interface %s",
+               netdev->link);
+         return -1;
+       }
+
      /* retrieve the name of the interface */
      if (!if_indextoname(netdev->ifindex, current_ifname)) {
        ERROR("no interface corresponding to index '%d'",
--
1.7.11.7

However, this does not solve a main issue: assuming that the ifindex will remain the same, in all situations, after an interface has been moved to a new netns. It _should_ not appear with virtual interfaces, but that depends on the ifindexes that are allocated to new interfaces.

How do you think this should be handled?

Thank you,
Bogdan P.

[1] http://iijean.blogspot.com/2010/03/howto-get-list-of-network-interfaces-in.html