[lxc-devel] Building NST under LXC

[Michael H. Warfield]

Hey Paul, Ron...

Been a while, no chat...  Last time was almost a year ago over problems
building Network Security Tookit 18 over root/sudo/user.  I've recently
started building some custom NST 20 (Fedora 20) builds.  Couple of minor
gotcha's but nothing I couldn't work out for myself.  Good work as usual
on the NST front.

You do need to update the web pages on "Building NST" to reflect the
update to Fedora 20.  It still says Fedora 18.  :-)=)  Just saying...

I've lately been involved in the LXC project (Linux Containers) and been
contributing heavily to the Fedora and CentOS templates there.  The LXC
devel group is cc'ed on this.  I wouldn't mind creating an NST template
if things work out right.  Incorporating LXC into NST on an NST host
would also be a desirable target.

It's been a standing goal for me to be able to build NST iso images from
within LXC light weight containers.  To date, I have had a dedicated
i686 tower and an x86_64 tower just for building custom NST builds.
Being able to build NST from within a container would allow me to retire
two entire tower systems and consolidate them onto a big rack server I
have.  I have a beefy 2U x86_64 system with 16 processors, 64G of RAM
and several TB of disk that can take over those tasks without even
flinching the load average.  So, that's been a goal for a while now...

I've largely achieved that goal now (and this is for some of the LXC
gang)...  The LXC container must have loop devices available to it and
created (and I'm sure Serge and Stéphane over at the LXC group are
immediately going "well duh") and must also have the "setfcap"
capability (which is NOT in the default for containers).  But, that's
just LXC stuff.  The NST side of the house also has some gotcha's.

For the ARCH like-on-like case (x86_64 on x86_64) this proved to be
almost a no-brainer.  It has a few gotchas like the container has to
have the kernel, kernel-devel, and kernel-header packages installed,
which would not be required for a "container" but that's understandable.
I also found the build (configure) failed initially in two spots for
other failed dependencies.  One was on "yumdownloader" (yum-utils) and
the other was on "ed".  You might want to add those as early
dependencies to the build install/configure.

Also, NST seems to be rather picky about matching the running kernel and
the kernel modules, which can get complicated when running in a
container.  In the like-on-like case, I was able to avoid this simply by
insuring that I had the correct kernel packages installed in the
container as the running kernel in the host.  A bit ugly but OK.  With
those, I have now been able to build NST x86_64 images in a container on
my big beast server without surrendering the entire machine to NST or
resorting to the performance overhead of a heavy weight hypervisor for
full HW virtualization like LKVM or VirtualBox.

I'm still working on the cross-arch case though (i386/i686 built on
x86_64).  I have the LXC Fedora template where it will create an i686
container on an x86_64 host and that works pretty decent.  Stéphane
Graber and I were recently working out details where we build an i686
Fedora container on an x86_64 host which was not running Fedora
(Ubuntu), but we sorted those details and can now even do the
cross-distro/cross-arch container builds.  :-)=)

Now, here comes the fun with NST.  NST does NOT like cross builds like
that, at all.  There are two significant problems in the configure
script.

One is that both "ARCH" and "BASEARCH" are based on the running kernel
with no way to override.  A simple environment override would help
there, but I'm not sure that's the correct course of action.  This is
around line 218 in the NST "configure" where you have this:

ARCH="$(/bin/arch)";
BASEARCH="$(/bin/uname -i)";
HWARCH="${BASEARCH}";

There needs to be an override or something there.  Not sure if this
would work but something like...

: ${ARCH="$(/bin/arch)"};
: ${BASEARCH="$(/bin/uname -i)"};
HWARCH="${BASEARCH}";

Problem there is that you then have to export and set both ARCH and
BASEARCH before doing the builds.  Ugly, but not all that outside of the
norm.  It should also incorporate some sanity checking for nonsensical
cross builds.  :-P

Haven't tested that, though, and I'm not sure the order of precedence
allows those expansions correctly.  On my TODO list to test.

The other problem is in the "set_kerver" function.  That's the home of
the pickieness of NST with regards to the running kernel vs what's
installed.  Rather than just bailing with an error if there's
no /lib/modules/${KERVER} directory, a quick hack to choose the latest
version in /lib/modules seems to help there and makes NST less picky in
that region of code.  But, again, I'm not so sure that's the best answer
from your standpoint.

I'm still working through some of the i686/x86_64 gotcha's (core-utils
is kicking my a** at the moment) but this case is very close.

Getting that last case to work allows me to retire the second tower
system and subsume it into the big server using light weight containers
as well.  I'll send you some suggested patches later but I would also
like to see your thoughts on the matter as well, which is why I'm
pinging you before I actually have something fully working w/ patches.
I don't think LXC has any ability to "fake out" $(arch) or $(uname -i)
so I'm assuming we need this needs to be more generalized in NST.

Once past this, I'm looking at the Raspberry Pi and ARM arch both with
and without LXC.  :-)=)

Regards,
Mike
-- 
Michael H. Warfield (AI4NB) | (770) 978-7061 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20140331/95cdcf86/attachment.pgp>