[Lxc-users] mac addresses
Brian K. White
brian at aljex.com
Fri Feb 12 16:37:32 UTC 2010
I have a host set up with 0.6.5 + force-umount-rootfs.patch and 9
containers.
Bridge/veth networking with all ip's on real internet ip's on a Verizion
FiOS static ip account. (Verizon in this case, supplies essentially a
connection to a switch, there is no customer-side router to provide the
IP's, you just put the single network connection into a switch to get
more physical ports. The IP's already work on it and the gateway ip is
off somewhere on the Verizon side of the connection. My netmask is /24
although my own ip's are just a chunk of 30 or so in the middle, but the
netmask is correct since the gateway is .1 of that subnet so I have to
be able to reach .1.)
Originally I had hand-crafted MAC addresses for each virtual nic that
looked like "00:16:3e:00:01:hh" where "hh" was the hex rendition of the
last octet of the IP address.
This worked perfectly, survived reboots etc, it "just worked".
Then I thought to simplify my config & setup recipe by removing the hand
made MAC's and let lxc assign them automatically since in a basic config
there is nothing that really cares about the MAC.
I removed the hwaddr lines from all the configs and restarted all the
containers.
Networking didn't work so good. If I ping from a container out to the
world, then for a few seconds, the world can reach the container. If I
stop causing traffic from the container, the world loses sight of the
container.
So, back to static assigned MAC's but, I still wanted a simpler yet
always working rule for creating the MAC, so I did this:
IP=192.168.0.50
HA=`printf "02:00:%x:%x:%x:%x" ${IP//./ }`
and
lxc.network.hwaddr = $HA
Updated all config files with that and restarted all containers.
I'm still seeing the same result.
The container can reach out to any address it usually could, including
the host, neighboring containers on the same host, neighboring physical
servers on the same public yet physically local network, and anywhere
else on the internet.
But only neighboring physical machines on the same network (on the same
local switch but using public ip's) , the host, and neighboring
containers can ping IN to my containers.
Unless, if I log into a container via the screen/console session and
from there I ping out to some remote box on some non-verizon network,
then, only right while those pings are running and just a few seconds
after, can that remote box ping (or anything else) in to the container.
I let it sit this way over night to make sure arp tables must have
expired everywhere. I tried pinging non-existent ip's on the subnet to
trigger the switch to re-discover arp. The local switch (Foundry
EdgeIron 24g with stock settings) Seems to have updated fine since all
local real & virtual boxes can see each other at any time in any direction.
I have not yet tried rebooting the host, or doing anything else on the
host like maybe down/up it's eth0 or it's br0.
I have not yet tried putting back all the original 00:16:3e:x:x:x macs.
The reason I'm trying to use the new mac numbering scheme in case it's
not apparent is, an OUI consumes 3 of the available 6 bytes in a mac,
leaving only 3 to make a unique number out of. Yet IP addresses have 4
bytes and I want a _simple_ rule or bit of script that will always
result in a safely unique, statically assigned mac, without having to
maintain a db of them. Something I could put into documentation and it
would always work, not just usually work in the simple cases.
So I figured, 02:00 for the first two bytes, and directly convert the IP
into the remaining 4 bytes. Every admin already has to make his IP's
unique at least within the same network, so those macs should always
work. My initial rule of using the OUI that Novell uses for Xen guests
(I'm using openSUSE) and then convering just the last byte of the IP
into the last byte of the mac, and using the second-to-last byte of the
mac to seperate different networks if a host or container has multiple
nics on multiple networks, that rule falls down in all but the simplest
cases.
So my question is, is "02:x:x:x:x:x" in some way non-routable just
because it sets the locally-administered bit?
Is there some sort of packet I can send that will trigger Verizons
switches & routers to update to the new mac for more than 5 seconds?
If I traceroute from a remote box in to one of these containers, it just
gets to a particular Verizon router that I have no control over and goes
no further.
> pa2:~ # traceroute nj12
> traceroute to nj12 (71.187.206.76), 30 hops max, 40 byte packets
> 1 gw-238-225.quonix.net (208.72.238.225) 0.349 ms 0.224 ms 0.198 ms
> 2 ge-11-1-2.mpr3.phl2.us.above.net (209.249.122.165) 0.280 ms 0.300 ms 0.199 ms
> 3 xe-4-0-0.mpr1.lga5.us.above.net (64.125.31.34) 2.533 ms 2.437 ms 2.431 ms
> 4 xe-0-1-0.er1.lga5.us.above.net (64.125.27.61) 2.264 ms 2.312 ms 2.325 ms
> 5 0.ge-3-2-0.BR3.NYC4.ALTER.NET (204.255.168.25) 2.291 ms 2.294 ms 2.321 ms
> 6 0.ge-4-2-0.NY5030-BB-RTR2.verizon-gni.net (152.63.10.54) 2.652 ms 2.683 ms 6.821 ms
> 7 so-6-3-0-0.NWRK-BB-RTR2.verizon-gni.net (130.81.19.97) 4.155 ms 4.303 ms 3.696 ms
> 8 P15-0-0.NWRKNJ-LCR-04.verizon-gni.net (130.81.29.195) 4.904 ms 4.806 ms 4.697 ms
> 9 P12-0-0.NWRKNJ-LCR-06.verizon-gni.net (130.81.27.7) 5.657 ms 5.434 ms 5.447 ms
> 10 P14-0.NWRKNJ-LCR-08.verizon-gni.net (130.81.30.95) 5.658 ms 5.551 ms 5.573 ms
If I traceroute from the same remote box to the host that the containers
is on, it take a very different-looking path and reaches the host.
Though possibly the differences are just load-balancing hardware?
> pa2:~ # traceroute nj10
> traceroute to nj10 (71.187.206.74), 30 hops max, 40 byte packets
> 1 gw-238-225.quonix.net (208.72.238.225) 0.325 ms 0.187 ms 0.196 ms
> 2 ge-11-1-2.mpr3.phl2.us.above.net (209.249.122.165) 0.282 ms 0.179 ms 0.201 ms
> 3 xe-4-0-0.mpr1.lga5.us.above.net (64.125.31.34) 2.535 ms 2.439 ms 2.430 ms
> 4 xe-0-1-0.er1.lga5.us.above.net (64.125.27.61) 2.259 ms 2.288 ms 2.317 ms
> 5 0.ge-3-2-0.BR3.NYC4.ALTER.NET (204.255.168.25) 2.327 ms 2.314 ms 2.307 ms
> 6 0.ge-8-1-0.NY325-BB-RTR1.verizon-gni.net (152.63.18.38) 2.690 ms 2.756 ms 2.646 ms
> 7 so-4-0-0-0.NWRK-BB-RTR1.verizon-gni.net (130.81.17.7) 7.708 ms 20.080 ms 7.678 ms
> 8 P15-0-0.NWRKNJ-LCR-03.verizon-gni.net (130.81.29.193) 8.419 ms 8.098 ms 8.031 ms
> 9 P12-0-0.NWRKNJ-LCR-05.verizon-gni.net (130.81.27.2) 9.065 ms 9.111 ms 9.059 ms
> 10 P14-0.NWRKNJ-LCR-07.verizon-gni.net (130.81.30.9) 7.729 ms 7.671 ms 7.595 ms
> 11 * * *
> 12 static-71-187-206-74.nwrknj.fios.verizon.net (71.187.206.74) 17.883 ms 12.380 ms 14.285 ms
> pa2:~ #
--
bkw
More information about the lxc-users
mailing list