[lxc-devel] [PATCH 1/6] Move lxcbr0 setup logic into lxc.net script
Michael H. Warfield
mhw at WittsEnd.com
Mon Aug 11 17:17:03 UTC 2014
On Mon, 2014-08-11 at 15:37 +0000, Serge Hallyn wrote:
> Quoting Michael H. Warfield (mhw at WittsEnd.com):
> > On Thu, 2014-07-31 at 08:53 +0200, Martin Pitt wrote:
> > > Factor this out of the lxc-net.conf upstart job, so that it can be used by
> > > init.d scripts and systemd units, too.
> >
> > Crap. Never fails. I was in Europe when this came out.
> >
> > Looking at lxc.net, I would say it's going to break some existing setups
> > (notably mine) where lxcbr0 is already setup. Yes, we can set
> > LXC_BRIDGE to no but we should also include some autodetect logic such
> > that, if lxcbr0 already exists, this doesn't commit random acts of
> > terrorism.
> That logic should already be there. If /sys/class/net/lxcbr0 already
> exists, then start will do nothing; if /run/lxc/network_up does not
> exist then stop will do nothing.
Yeah, I traced through that logic and finally realized that. I'm a
little nudgey about even calling "stop" from "start" in the case where
the bridge already exists or when USE_LXC_BRIDGE is not true and then
relying on "stop" to rely on the non-existence of network_up to exit
without doing anything to the bridge. If that's the case, why even call
"stop" in the "start" function at all? That had me confused.
I don't see the need for this logic at all in "start":
--
if [ -d /sys/class/net/${LXC_BRIDGE} ]; then
if [ ! -f ${varrun}/network_up ]; then
# bridge exists, but we didn't start it
stop;
fi
exit 0;
fi
--
If this bridge exists and we didn't start it, why even attempt to stop
it? Yeah, that will hit the same check in "stop" and then exit but it
seems like an overly convoluted logic (which includes
loading /etc/default/lxc twice) where both paths logically end up in an
exit. Why have that inner "if" stanza at all? Just "exit 0"? Just
this:
--
if [ -d /sys/class/net/${LXC_BRIDGE} ]; then
exit 0;
fi
--
I'm still unsure about the default configs and handling discrepancies
between what's in /etc/lxc/default.conf that the sysvinit references for
lxc.network.link (virbr0 in the case of Oracle, CentOS, and Fedora) and
what's in /etc/default/lxc for LXC_BRIDGE. What's the corner cases
where those two definitions disagree?
That file raises another question. That file should
be /etc/sysconfig/lxc on the RH derived distributions as well (not so
sure about Suse). That's going to need to be fixed as well. That will
probably need to be autoconfigured somehow.
The config/init/sysvinit/lxc.in file references @SYSCONFDIR@/sysconfig
for that, which expands to /etc/sysconfig for us as the parent directory
containing the lxc init configuration file. That makes it a bit
problematical on Ubuntu (and that script is used by the lxc.service
systemd unit file as well as the sysvinit lxc script). So we've got the
reverse situation in there and I don't think we want two lxc init
configuration files.
I don't, right off the top of my head, see a clean way to handle that
unless we check both locations, since both conventions are based off of
@SYSCONFDIR@ in autoconf. If we don't, we're going to be prone to
inconsistencies there as well.
I'm also curious why that particular helper ended up in /usr/share/lxc
instead of /usr/libexec/lxc where we have the other pieces of the
autostart helpers like the sysvinit script Dwight stuffed over there.
This seems to be a parallel init setup (lxc-net). Not a critical point
but another inconsistency.
Looking at those firewall rules, I do still have some heartburn there.
They don't look very configurable the way they're set up. That would
screw me up by blocking ssh from a container into the host, wouldn't it?
Only allow dhcp and dns with no provision for user defined rules? When
I was doing similar things with IPSec and Openswan, we did all of "our"
changes in a specific chain we set up and then only added a single -j
jump into that chain. That way, if someone modified the chain, we could
just arbitrarily flush our chain and drop the link into the chain, all
the while preserving an atomic add and removal of the chain
functionality.
I'll send in a patch for the lxc.spec.in problem shortly. I have that
one pinned at least. Minor patch, I just had to find it.
Regards,
Mike
> > For example, my lxcbr0 on Fedora 20 is a bridge bridge (I happen to have
> > LOTS of IPv4 address space so eth0 or whatever is bridged to the bridge
> > in static networking), not a nat bridge. I'm also not real sure how
> > this use of iptables is going to play with firewalld on Fedora or CentOS
> > 7 either (or maybe Oracle 7 if they're using firewalld). I have to
> > examine that (and I'm not a big fan of firewalld).
> >
> > This also potentially impacts the default lxc.confg for Fedora, CentOS,
> > Oracle, and possibly others, that had been depending on libvirt for
> > setting up virbr0 as a natted bridge. That's in configure with this:
> >
> > redhat|centos|fedora|oracle|oracleserver)
> > distroconf=default.conf.libvirt
> >
> > So we either have to change those defaults and change all preexisting
> > systems and containers, or this has a high probability of doing the
> > wrong thing even on new setups. At the very least, if "lxc.network.link
> > = " is not "lxcbr0" it won't work properly for new containers using that
> > default. It will set up an unnecessary bridge and firewall rules where
> > containers are using virbr0. At worst, it'll break existing setups
> > where lxcbr0 is already set up in static networking in a conflicting
> > manner.
> >
> > I'm already looking at the "make rpm" breakage. I'll look at this as
> > well. At the very least, there has to be a "do no harm" check early in
> > on the lxc.net script and exit in start() if lxcbr0 already exists.
> >
> > I'd like to hear from Dwight and the Oracle side since he did most of
> > the sysvinit stuff for Oracle (most particularly the bridge wait code
> > based on the lxc.conf default) and now we both have to deal with the
> > systemd side of things for Oracle, Fedora, and CentOS (and possibly
> > Suse).
> >
> > Regards,
> > Mike
> >
> > > Part of https://launchpad.net/bugs/1312532
> > > ---
> > > config/init/upstart/lxc-net.conf | 88 +----------------------------------
> > > src/lxc/Makefile.am | 1 +
> > > src/lxc/lxc.net | 99 ++++++++++++++++++++++++++++++++++++++++
> > > 3 files changed, 102 insertions(+), 86 deletions(-)
> > > create mode 100755 src/lxc/lxc.net
> > >
> > > diff --git a/config/init/upstart/lxc-net.conf b/config/init/upstart/lxc-net.conf
> > > index 279cd1e..38f6ea3 100644
> > > --- a/config/init/upstart/lxc-net.conf
> > > +++ b/config/init/upstart/lxc-net.conf
> > > @@ -4,89 +4,5 @@ author "Serge Hallyn <serge.hallyn at canonical.com>"
> > > start on starting lxc
> > > stop on stopped lxc
> > >
> > > -env USE_LXC_BRIDGE="true"
> > > -env LXC_BRIDGE="lxcbr0"
> > > -env LXC_ADDR="10.0.3.1"
> > > -env LXC_NETMASK="255.255.255.0"
> > > -env LXC_NETWORK="10.0.3.0/24"
> > > -env LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
> > > -env LXC_DHCP_MAX="253"
> > > -env LXC_DHCP_CONFILE=""
> > > -env varrun="/run/lxc"
> > > -env LXC_DOMAIN=""
> > > -
> > > -pre-start script
> > > - [ -f /etc/default/lxc ] && . /etc/default/lxc
> > > -
> > > - [ "x$USE_LXC_BRIDGE" = "xtrue" ] || { stop; exit 0; }
> > > -
> > > - use_iptables_lock="-w"
> > > - iptables -w -L -n > /dev/null 2>&1 || use_iptables_lock=""
> > > - cleanup() {
> > > - # dnsmasq failed to start, clean up the bridge
> > > - iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p udp --dport 67 -j ACCEPT
> > > - iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p tcp --dport 67 -j ACCEPT
> > > - iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p udp --dport 53 -j ACCEPT
> > > - iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p tcp --dport 53 -j ACCEPT
> > > - iptables $use_iptables_lock -D FORWARD -i ${LXC_BRIDGE} -j ACCEPT
> > > - iptables $use_iptables_lock -D FORWARD -o ${LXC_BRIDGE} -j ACCEPT
> > > - iptables $use_iptables_lock -t nat -D POSTROUTING -s ${LXC_NETWORK} ! -d ${LXC_NETWORK} -j MASQUERADE || true
> > > - iptables $use_iptables_lock -t mangle -D POSTROUTING -o ${LXC_BRIDGE} -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
> > > - ifconfig ${LXC_BRIDGE} down || true
> > > - brctl delbr ${LXC_BRIDGE} || true
> > > - }
> > > -
> > > - if [ -d /sys/class/net/${LXC_BRIDGE} ]; then
> > > - if [ ! -f ${varrun}/network_up ]; then
> > > - # bridge exists, but we didn't start it
> > > - stop;
> > > - fi
> > > - exit 0;
> > > - fi
> > > -
> > > - # set up the lxc network
> > > - brctl addbr ${LXC_BRIDGE} || { echo "Missing bridge support in kernel"; stop; exit 0; }
> > > - echo 1 > /proc/sys/net/ipv4/ip_forward
> > > - mkdir -p ${varrun}
> > > - ifconfig ${LXC_BRIDGE} ${LXC_ADDR} netmask ${LXC_NETMASK} up
> > > - iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p udp --dport 67 -j ACCEPT
> > > - iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p tcp --dport 67 -j ACCEPT
> > > - iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p udp --dport 53 -j ACCEPT
> > > - iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p tcp --dport 53 -j ACCEPT
> > > - iptables $use_iptables_lock -I FORWARD -i ${LXC_BRIDGE} -j ACCEPT
> > > - iptables $use_iptables_lock -I FORWARD -o ${LXC_BRIDGE} -j ACCEPT
> > > - iptables $use_iptables_lock -t nat -A POSTROUTING -s ${LXC_NETWORK} ! -d ${LXC_NETWORK} -j MASQUERADE
> > > - iptables $use_iptables_lock -t mangle -A POSTROUTING -o ${LXC_BRIDGE} -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
> > > -
> > > - LXC_DOMAIN_ARG=""
> > > - if [ -n "$LXC_DOMAIN" ]; then
> > > - LXC_DOMAIN_ARG="-s $LXC_DOMAIN -S /$LXC_DOMAIN/"
> > > - fi
> > > - dnsmasq $LXC_DOMAIN_ARG -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=${varrun}/dnsmasq.pid --conf-file=${LXC_DHCP_CONFILE} --listen-address ${LXC_ADDR} --dhcp-range ${LXC_DHCP_RANGE} --dhcp-lease-max=${LXC_DHCP_MAX} --dhcp-no-override --except-interface=lo --interface=${LXC_BRIDGE} --dhcp-leasefile=/var/lib/misc/dnsmasq.${LXC_BRIDGE}.leases --dhcp-authoritative || cleanup
> > > - touch ${varrun}/network_up
> > > -end script
> > > -
> > > -post-stop script
> > > - [ -f /etc/default/lxc ] && . /etc/default/lxc
> > > - [ -f "${varrun}/network_up" ] || exit 0;
> > > - # if $LXC_BRIDGE has attached interfaces, don't shut it down
> > > - ls /sys/class/net/${LXC_BRIDGE}/brif/* > /dev/null 2>&1 && exit 0;
> > > -
> > > - if [ -d /sys/class/net/${LXC_BRIDGE} ]; then
> > > - use_iptables_lock="-w"
> > > - iptables -w -L -n > /dev/null 2>&1 || use_iptables_lock=""
> > > - ifconfig ${LXC_BRIDGE} down
> > > - iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p udp --dport 67 -j ACCEPT
> > > - iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p tcp --dport 67 -j ACCEPT
> > > - iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p udp --dport 53 -j ACCEPT
> > > - iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p tcp --dport 53 -j ACCEPT
> > > - iptables $use_iptables_lock -D FORWARD -i ${LXC_BRIDGE} -j ACCEPT
> > > - iptables $use_iptables_lock -D FORWARD -o ${LXC_BRIDGE} -j ACCEPT
> > > - iptables $use_iptables_lock -t nat -D POSTROUTING -s ${LXC_NETWORK} ! -d ${LXC_NETWORK} -j MASQUERADE || true
> > > - iptables $use_iptables_lock -t mangle -D POSTROUTING -o ${LXC_BRIDGE} -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
> > > - pid=`cat ${varrun}/dnsmasq.pid 2>/dev/null` && kill -9 $pid || true
> > > - rm -f ${varrun}/dnsmasq.pid
> > > - brctl delbr ${LXC_BRIDGE}
> > > - fi
> > > - rm -f ${varrun}/network_up
> > > -end script
> > > +pre-start exec /usr/share/lxc/lxc.net start
> > > +post-stop exec /usr/share/lxc/lxc.net stop
> > > diff --git a/src/lxc/Makefile.am b/src/lxc/Makefile.am
> > > index cdc6833..ee74e3c 100644
> > > --- a/src/lxc/Makefile.am
> > > +++ b/src/lxc/Makefile.am
> > > @@ -255,6 +255,7 @@ endif
> > > install-exec-local: install-soPROGRAMS
> > > mkdir -p $(DESTDIR)$(datadir)/lxc
> > > install -c -m 644 lxc.functions $(DESTDIR)$(datadir)/lxc
> > > + install -c -m 755 lxc.net $(DESTDIR)$(datadir)/lxc
> > > mv $(DESTDIR)$(libdir)/liblxc.so $(DESTDIR)$(libdir)/liblxc.so.$(VERSION)
> > > cd $(DESTDIR)$(libdir); \
> > > ln -sf liblxc.so.$(VERSION) liblxc.so.$(firstword $(subst ., ,$(VERSION))); \
> > > diff --git a/src/lxc/lxc.net b/src/lxc/lxc.net
> > > new file mode 100755
> > > index 0000000..5ea4f1d
> > > --- /dev/null
> > > +++ b/src/lxc/lxc.net
> > > @@ -0,0 +1,99 @@
> > > +#!/bin/sh
> > > +set -eu
> > > +
> > > +USE_LXC_BRIDGE="true"
> > > +LXC_BRIDGE="lxcbr0"
> > > +LXC_ADDR="10.0.3.1"
> > > +LXC_NETMASK="255.255.255.0"
> > > +LXC_NETWORK="10.0.3.0/24"
> > > +LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
> > > +LXC_DHCP_MAX="253"
> > > +LXC_DHCP_CONFILE=""
> > > +varrun="/run/lxc"
> > > +LXC_DOMAIN=""
> > > +
> > > +start() {
> > > + [ -f /etc/default/lxc ] && . /etc/default/lxc
> > > +
> > > + [ "x$USE_LXC_BRIDGE" = "xtrue" ] || { stop; exit 0; }
> > > +
> > > + use_iptables_lock="-w"
> > > + iptables -w -L -n > /dev/null 2>&1 || use_iptables_lock=""
> > > + cleanup() {
> > > + # dnsmasq failed to start, clean up the bridge
> > > + iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p udp --dport 67 -j ACCEPT
> > > + iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p tcp --dport 67 -j ACCEPT
> > > + iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p udp --dport 53 -j ACCEPT
> > > + iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p tcp --dport 53 -j ACCEPT
> > > + iptables $use_iptables_lock -D FORWARD -i ${LXC_BRIDGE} -j ACCEPT
> > > + iptables $use_iptables_lock -D FORWARD -o ${LXC_BRIDGE} -j ACCEPT
> > > + iptables $use_iptables_lock -t nat -D POSTROUTING -s ${LXC_NETWORK} ! -d ${LXC_NETWORK} -j MASQUERADE || true
> > > + iptables $use_iptables_lock -t mangle -D POSTROUTING -o ${LXC_BRIDGE} -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
> > > + ifconfig ${LXC_BRIDGE} down || true
> > > + brctl delbr ${LXC_BRIDGE} || true
> > > + }
> > > +
> > > + if [ -d /sys/class/net/${LXC_BRIDGE} ]; then
> > > + if [ ! -f ${varrun}/network_up ]; then
> > > + # bridge exists, but we didn't start it
> > > + stop;
> > > + fi
> > > + exit 0;
> > > + fi
> > > +
> > > + # set up the lxc network
> > > + brctl addbr ${LXC_BRIDGE} || { echo "Missing bridge support in kernel"; stop; exit 0; }
> > > + echo 1 > /proc/sys/net/ipv4/ip_forward
> > > + mkdir -p ${varrun}
> > > + ifconfig ${LXC_BRIDGE} ${LXC_ADDR} netmask ${LXC_NETMASK} up
> > > + iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p udp --dport 67 -j ACCEPT
> > > + iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p tcp --dport 67 -j ACCEPT
> > > + iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p udp --dport 53 -j ACCEPT
> > > + iptables $use_iptables_lock -I INPUT -i ${LXC_BRIDGE} -p tcp --dport 53 -j ACCEPT
> > > + iptables $use_iptables_lock -I FORWARD -i ${LXC_BRIDGE} -j ACCEPT
> > > + iptables $use_iptables_lock -I FORWARD -o ${LXC_BRIDGE} -j ACCEPT
> > > + iptables $use_iptables_lock -t nat -A POSTROUTING -s ${LXC_NETWORK} ! -d ${LXC_NETWORK} -j MASQUERADE
> > > + iptables $use_iptables_lock -t mangle -A POSTROUTING -o ${LXC_BRIDGE} -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
> > > +
> > > + LXC_DOMAIN_ARG=""
> > > + if [ -n "$LXC_DOMAIN" ]; then
> > > + LXC_DOMAIN_ARG="-s $LXC_DOMAIN -S /$LXC_DOMAIN/"
> > > + fi
> > > + dnsmasq $LXC_DOMAIN_ARG -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=${varrun}/dnsmasq.pid --conf-file=${LXC_DHCP_CONFILE} --listen-address ${LXC_ADDR} --dhcp-range ${LXC_DHCP_RANGE} --dhcp-lease-max=${LXC_DHCP_MAX} --dhcp-no-override --except-interface=lo --interface=${LXC_BRIDGE} --dhcp-leasefile=/var/lib/misc/dnsmasq.${LXC_BRIDGE}.leases --dhcp-authoritative || cleanup
> > > + touch ${varrun}/network_up
> > > +}
> > > +
> > > +stop() {
> > > + [ -f /etc/default/lxc ] && . /etc/default/lxc
> > > + [ -f "${varrun}/network_up" ] || exit 0;
> > > + # if $LXC_BRIDGE has attached interfaces, don't shut it down
> > > + ls /sys/class/net/${LXC_BRIDGE}/brif/* > /dev/null 2>&1 && exit 0;
> > > +
> > > + if [ -d /sys/class/net/${LXC_BRIDGE} ]; then
> > > + use_iptables_lock="-w"
> > > + iptables -w -L -n > /dev/null 2>&1 || use_iptables_lock=""
> > > + ifconfig ${LXC_BRIDGE} down
> > > + iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p udp --dport 67 -j ACCEPT
> > > + iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p tcp --dport 67 -j ACCEPT
> > > + iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p udp --dport 53 -j ACCEPT
> > > + iptables $use_iptables_lock -D INPUT -i ${LXC_BRIDGE} -p tcp --dport 53 -j ACCEPT
> > > + iptables $use_iptables_lock -D FORWARD -i ${LXC_BRIDGE} -j ACCEPT
> > > + iptables $use_iptables_lock -D FORWARD -o ${LXC_BRIDGE} -j ACCEPT
> > > + iptables $use_iptables_lock -t nat -D POSTROUTING -s ${LXC_NETWORK} ! -d ${LXC_NETWORK} -j MASQUERADE || true
> > > + iptables $use_iptables_lock -t mangle -D POSTROUTING -o ${LXC_BRIDGE} -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
> > > + pid=`cat ${varrun}/dnsmasq.pid 2>/dev/null` && kill -9 $pid || true
> > > + rm -f ${varrun}/dnsmasq.pid
> > > + brctl delbr ${LXC_BRIDGE}
> > > + fi
> > > + rm -f ${varrun}/network_up
> > > +}
> > > +
> > > +if [ "$1" = start ]; then
> > > + start
> > > +elif [ "$1" = stop ]; then
> > > + stop
> > > +else
> > > + echo "Usage: $0 start|stop" >&2
> > > + exit 1
> > > +fi
> > > +
> > > --
> > > 2.0.1
> > >
> > > _______________________________________________
> > > lxc-devel mailing list
> > > lxc-devel at lists.linuxcontainers.org
> > > http://lists.linuxcontainers.org/listinfo/lxc-devel
> > >
> >
> > --
> > Michael H. Warfield (AI4NB) | (770) 978-7061 | mhw at WittsEnd.com
> > /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
> > NIC whois: MHW9 | An optimist believes we live in the best of all
> > PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
> >
>
>
>
> > _______________________________________________
> > lxc-devel mailing list
> > lxc-devel at lists.linuxcontainers.org
> > http://lists.linuxcontainers.org/listinfo/lxc-devel
>
> _______________________________________________
> lxc-devel mailing list
> lxc-devel at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-devel
>
--
Michael H. Warfield (AI4NB) | (770) 978-7061 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 465 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20140811/f6f31277/attachment-0001.sig>
More information about the lxc-devel
mailing list