[lxc-devel] [Lxc-users] Working LXC templates? EUREAKA! I think I've got it!

[Michael H. Warfield]

All - Especially Tony Su,

Couple of people where I work thought you couldn't do what I was trying
to do, that it was "impossible".  Oh well.  Looks like they were
wrong.  :-P  It may not be "efficient" but it can be made to work.

Way down below, in-line...

On Mon, 2013-09-09 at 07:28 -0400, Michael H. Warfield wrote: 
> On Mon, 2013-09-09 at 08:58 +0200, Natanael Copa wrote: 
> > On Sun, 08 Sep 2013 20:33:16 -0400
> > "Michael H. Warfield" <mhw at WittsEnd.com> wrote:
> > 
> > > With all due respect...
> > > 
> > > On Sun, 2013-09-08 at 16:08 -0700, Tony Su wrote: 
> > > > After putting some thought into this,
> > > > IMO LXC badly needs a universal tool with the following features
> > > > 
> > > > - A single script should be used to run on <any> HostOS, creating
> > > > <any> supported Container OS. Although this would make the script
> > > > extremely large, IMO it would actually be easier to maintain in the
> > > > long run.
> > > 
> > > Actually, no.  From my experience (30+ years in software development),
> > > it would turn into a morass.
> > > 
> > > The problem here is that the maintainer(s) would then need to understand
> > > how each and every distribution is installed and how it would be
> > > installed on each and every distribution.  It would distill the worse of
> > > all the problems we have now in the templates into one great big dung
> > > pile.  It would rapidly become unmaintainable.  The "extremely large" is
> > > the red letter warning that it will become unmaintainable as fewer and
> > > fewer people understand what this great big huge blob does.
> 
> > I tend to agree with this.

> > What I do think could be done is to use template APIs. Either by having
> > a script "library" with helper functions (for things like:
> > generate_mac_address etc) or let the template scripts be "plugins" that
> > must provide a set of pre-defined functions (eg. install_distro,
> > configure_distro, copy_configuration etc) or maybe a combination of
> > those two.

> I like this idea, it's just that, in the short term, we have to get past
> this one gotcha.

> In the git-stage current Fedora template, the entire problem is embodied
> in the "download_fedora" function starting around line 201...  The
> gotcha's are three commands around line 272 after we've identified and
> downloaded the initial release rpm.  We have this:
> 
>     mkdir -p $INSTALL_ROOT/var/lib/rpm
>     rpm --root $INSTALL_ROOT  --initdb
>     rpm --root $INSTALL_ROOT -ivh ${INSTALL_ROOT}/${RELEASE_RPM}
>     $YUM install $PKG_LIST

> Ok...  Those are running in the host local run time environment.
> Obviously, if the host does not have a (compatible) version of rpm
> and/or yum in the host local environment, you're screwed.  That's the
> catch-22 situation and it's the same situation with zypper in the
> OpenSuse template.  That short space of code has to be recreated in a
> totally distro-agnostic manner so it runs on any distro to create our
> initial rootfs.  After that, we can do what ever distro (Fedora)
> specific commands we want by chrooting into the target container first
> (including rebuilding the rpm database to the target version).  That's
> only even needed if you don't already have a cached rootfs for that
> distro and version.

> I was SOOOO close last week working on this while on vacation.  Recent
> revs of Fedora have this downloadable LiveOS core that runs on the
> netinst CD and others.  That's the 200MB - 300MB blob I was referring
> to.  You just download it, mount it (requires squashfs) to a directory
> and then mount the ext3 image it contains on another directory.  Then
> create and bind mount a few rw directories (etc var run), mount proc in
> the image, and bind mount your rootfs to run/install in the image.  Then
> chroot into the image and voila, instant RTE.  Yum and rpm are capable
> of installing other versions of Fedora, so I wouldn't (shouldn't) even
> need a version specific instantiation of the RTE, just one that can
> install every version we might be interested in.

> Except...  It didn't work.  Sigh...  $#@$#@#@

> Fedora came so close and then face planted for what I wanted to do.
> Sure rpm is in there.  But rpmdb is not.  So, no --initdb and no
> --rebuilddb.  They've got yum in there, but no yummain.py so yum hurls
> chunks immediately upon execution trying to import yummain.  What the
> hell good does it do to have yum in there but no yummain?!?!  But,
> something, fixed up, like that from all the distros would be a perfect
> answer (albeit somewhat less than high performance thanks to that
> download, but it's a single download that could be cached).  The only
> potential gotcha I see in there is requiring squashfs available for
> mounting the image.  Anyone have heartburn over that?

> That squashfs image has to be able to do an install or it would be
> useless on the netinst CD.  I'm not sure if they still have anaconda on
> there or not but it has to be able to do a kickstart install, so all I
> need is a minimal.ks kickstart file to perform essentially an unattended
> install of a minimal build into the target rootfs.  That's where I'm at
> now, trying to get that to work.

I got this to work with just one (err, minor, gag, couch) butt ugly
hack / dirty trick.  I seems, in this case, the problem with the Fedora
LiveOS squashfs image is that it has rpm (but no rpmdb) and yum (but no
yummain.py) partially installed and damn near everything else
(dependeices) are but it's not build with an rpm database in place.

The answer was to stage this with the original LiveOS fs being the stage
0 and copying that to a read-write directory and then downloading and
force (--nodeps) an install of rpm and yum installing the full packages
in place for a stage 1 run time environment.  That enabled me to create
a stage 2 uber-minimal run time environment with rpm and yum installed
along with all their dependencies.  This I could then cache and is
capable of installing arbitrary versions of Fedora (so it only needs to
be done once for any given arch).

I've integrated it into the lxc-fedora template but it has undergone
minimal testing (fedora on fedora).  It has a "white listing" facility
in it to run a native install on distros (and also checks if the distro
has yum and rpm available) that will support it but that's disabled on
this, attached, template so it can get more testing.  White listing is
based on the CPE OS identifier.  What will go into the patch to the
project will have fedora, centos, and maybe a few others whitelisted
that are known to work.

Attached to this message is my trial balloon lxc-fedora template.  This
is NOT final.  This is just a test for comments and feedback.  Please
try it out.  Other template owners, PLEASE look at the code and see if
something similar makes sense for your templates!  I would love to
install your distros on my Fedora hosts!

The code is commented moderately heavy (and somewhat sarcasticly) but
should explain what I was trying to do (if not - tell me!).  The bulk of
the changes are a block of code and comments starting around line 201
plus a few modifications just to the "download_fedora()" routine, now at
line 543 (yes it was a big insert in front of it).

I can not emphasize this enough...  READ THE COMMENTS!  If I'm full of
shit doing it this way, then tell me so!

Once it builds that RTE the first time, it will use it for all future
Fedora container builds.

Tony:  You are on OpenSuse.  You, more than anyone else, I really need
to test this code by.  I'll answer your other questions about WHY
systemd is incompatible with 0.8.0 and earlier in another message.  But
this template should run and build a container.  If it doesn't, I want
to fix that before I submit a patch.  I could use some people on
AltLinux and ArchLinux to also test this out on a host and give me
feedback.

Maybe I'll see a few of you at the LinuxPlumbers conference in a few
days in New Orleans.

Regards,
Mike

> But it's all that work just to replace those few lines of code above
> where you can not perform a host native install of the rootfs.  That's
> all that's standing in the way and it's frustratingly close for the
> Fedora template.
> 
> We've got the following distro templates in the project, currently:
> 
> lxc-alpine
> lxc-alt
> lxc-arch
> lxc-busybox (is this really a distro template - I'm not sure)
> lxc-debian
> lxc-fedora
> lxc-opensuse
> lxc-oracle (really?)
> lxc-ubuntu / lxc-ubuntu-cloud
> 
> Who are the owners?  I don't "own" the Fedora template but I work on it.
> Each template owner has to solve this problem in their own way and it
> all boils down to the same problem.  Building that initial rootfs in the
> host native environment.
> 
> After that, the idea of generalizing the functions into an API form
> sounds like a great idea.  It could be that this catch-22 could be
> buried in a "create_rootfs" function which would create a cacheable
> rootfs.
> 
> I would love to see templates for CentOS, Scientific (SL), Slackware and
> maybe even RedHat EL.  NST (Network Security Toolkit) and Kali (fka
> BackTrack) would also be nice and I might work on them (which can I say,
> I'm a security fanatic).  But I gotta solve this problem first in a way
> I can use for the ones I might want to write down the road.
> 
> But, in the meant time, we have "the recipe for rabbit stew"...  First
> you catch your rabbit.  First we have to have a way to build the initial
> rootfs.
> 
> <aside>
> 
> This entire process reminds me of when we (I) had to bootstrap gcc into
> environments (HPUX and AIX years ago) which had no C compiler installed.
> First you cross compiled a compiler (on Solaris in my case) for your
> target arch and transported it there.  Then you used that "stage 1"
> compiler to compile the gcc compiler on the native platform (stage 2).
> Then you used the stage 2 compiler to compile a full stage 3 compiler
> where you could then confirm the outputs from the stage 2 and stage 3
> compilers to confirm they matched (which never seems to happen the first
> time you try it).  When you were done and finally confirmed working, you
> really needed a beer.  This is the same sort of bootstrap process.
> 
> </aside>
> 
> > -nc
> 
> Regards,
> Mike

-- 
Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lxc-fedora
Type: application/x-shellscript
Size: 27544 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130912/81b26597/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130912/81b26597/attachment.pgp>