[Lxc-users] [lxc-devel] Working LXC templates? EUREAKA! I think I've got it!

[Michael H. Warfield]

Tony:

On Fri, 2013-09-13 at 00:20 -0400, Michael H. Warfield wrote: 
> On Thu, 2013-09-12 at 15:23 -0400, Michael H. Warfield wrote: 
> > All - Especially Tony Su,
> 
> > Couple of people where I work thought you couldn't do what I was trying
> > to do, that it was "impossible".  Oh well.  Looks like they were
> > wrong.  :-P  It may not be "efficient" but it can be made to work.

> Never fails, I swear, it never fails.  Not minutes after I posted this
> original message, one of my test runs blew an error that pointed out a
> typo that pointed out six other spots where I had environment variables
> out of sync and, after fixing them, I realized I had missed several
> update paths.  And then there were the initialization checks I missed.
> Would work fine on fedora on fedora but not what my target was.

> Ok...  So...  I fixed all that.  The attached template should actually
> work if you're not on a Fedora host (I hope).

Ok...  This morning, I wrapped up some testing of an OpenClient Fedora
19 (IBM internal distribution layer) on a spare Dell GX 620 which I
could then burn that puppy down and retask it.  I built it back up as an
OpenSUSE-1203 build and installed lxc on it.  Generic (no mods) tried
running lxc-create -t fedora on that.  Yeah, that caught fire and burned
real quick.  How did you get as far as you did???  No matter.  It was
destined to fail anyways.

IAC...  I copied my new template in place and it happily built Fedora
19, 18, 17, and 16 containers (I have run the F19 container and I
haven't gotten that far with playing with LXC on OpenSUSE).  So this
template should work for you, if you have the other pieces of the puzzle
(generic LXC setup) in place.  There IS a problem with versions prior to
F16 and I strongly encourage that NOBODY build containers of these
unsupported version (and F16 is guaranteed to NOT RUN - DON'T TRY)!  I'd
like this script to be as generic as possible but there's going to have
to be some limits.  F15 has a STUPID dependency on fedora-rawhide but I
can work around that for testing.  WTH???

Regards,
Mike

> It does dawn on me that all this effort really is a "one trick pony"
> and, once it has built the very first container, is no longer needed
> after the first run.  Reason?  Each container built can be a run time
> environment to build new containers.
> 
> So...  Before I submit an actual patch back to the upstream project,
> I'll probably scope out the logic to scan the cached containers for the
> most appropriate container and use it for the RTE and never use the
> cached install RTE again.
> 
> This may also be appropriate for other templates.  Once you've
> bootstrapped the template process, you may not need the bootstrap again.
> Just build and cache your best shot and use it in the future.
> 
> I have tested this template and built containers from F14 through F19.
> F15 failed but, I already know it would not have run as a container
> anyways because of the horribly broken and incompatible systemd in that
> distro.  Prior to F14, some versions will build and some won't.
> Shouldn't be a problem with Fedora on Fedora and anything prior to F18
> is past EOL anyways, so why should I care?
> 
> IAC...  Please discard the previous template.  Here is my current
> template.  And please test it.
> 
> Thanks!
> 
> Regards,
> Mike
> 
> > Way down below, in-line...
> > 
> > On Mon, 2013-09-09 at 07:28 -0400, Michael H. Warfield wrote: 
> > > On Mon, 2013-09-09 at 08:58 +0200, Natanael Copa wrote: 
> > > > On Sun, 08 Sep 2013 20:33:16 -0400
> > > > "Michael H. Warfield" <mhw at WittsEnd.com> wrote:
> > > > 
> > > > > With all due respect...
> > > > > 
> > > > > On Sun, 2013-09-08 at 16:08 -0700, Tony Su wrote: 
> > > > > > After putting some thought into this,
> > > > > > IMO LXC badly needs a universal tool with the following features
> > > > > > 
> > > > > > - A single script should be used to run on <any> HostOS, creating
> > > > > > <any> supported Container OS. Although this would make the script
> > > > > > extremely large, IMO it would actually be easier to maintain in the
> > > > > > long run.
> > > > > 
> > > > > Actually, no.  From my experience (30+ years in software development),
> > > > > it would turn into a morass.
> > > > > 
> > > > > The problem here is that the maintainer(s) would then need to understand
> > > > > how each and every distribution is installed and how it would be
> > > > > installed on each and every distribution.  It would distill the worse of
> > > > > all the problems we have now in the templates into one great big dung
> > > > > pile.  It would rapidly become unmaintainable.  The "extremely large" is
> > > > > the red letter warning that it will become unmaintainable as fewer and
> > > > > fewer people understand what this great big huge blob does.
> > > 
> > > > I tend to agree with this.
> > 
> > > > What I do think could be done is to use template APIs. Either by having
> > > > a script "library" with helper functions (for things like:
> > > > generate_mac_address etc) or let the template scripts be "plugins" that
> > > > must provide a set of pre-defined functions (eg. install_distro,
> > > > configure_distro, copy_configuration etc) or maybe a combination of
> > > > those two.
> > 
> > > I like this idea, it's just that, in the short term, we have to get past
> > > this one gotcha.
> > 
> > > In the git-stage current Fedora template, the entire problem is embodied
> > > in the "download_fedora" function starting around line 201...  The
> > > gotcha's are three commands around line 272 after we've identified and
> > > downloaded the initial release rpm.  We have this:
> > > 
> > >     mkdir -p $INSTALL_ROOT/var/lib/rpm
> > >     rpm --root $INSTALL_ROOT  --initdb
> > >     rpm --root $INSTALL_ROOT -ivh ${INSTALL_ROOT}/${RELEASE_RPM}
> > >     $YUM install $PKG_LIST
> > 
> > > Ok...  Those are running in the host local run time environment.
> > > Obviously, if the host does not have a (compatible) version of rpm
> > > and/or yum in the host local environment, you're screwed.  That's the
> > > catch-22 situation and it's the same situation with zypper in the
> > > OpenSuse template.  That short space of code has to be recreated in a
> > > totally distro-agnostic manner so it runs on any distro to create our
> > > initial rootfs.  After that, we can do what ever distro (Fedora)
> > > specific commands we want by chrooting into the target container first
> > > (including rebuilding the rpm database to the target version).  That's
> > > only even needed if you don't already have a cached rootfs for that
> > > distro and version.
> > 
> > > I was SOOOO close last week working on this while on vacation.  Recent
> > > revs of Fedora have this downloadable LiveOS core that runs on the
> > > netinst CD and others.  That's the 200MB - 300MB blob I was referring
> > > to.  You just download it, mount it (requires squashfs) to a directory
> > > and then mount the ext3 image it contains on another directory.  Then
> > > create and bind mount a few rw directories (etc var run), mount proc in
> > > the image, and bind mount your rootfs to run/install in the image.  Then
> > > chroot into the image and voila, instant RTE.  Yum and rpm are capable
> > > of installing other versions of Fedora, so I wouldn't (shouldn't) even
> > > need a version specific instantiation of the RTE, just one that can
> > > install every version we might be interested in.
> > 
> > > Except...  It didn't work.  Sigh...  $#@$#@#@
> > 
> > > Fedora came so close and then face planted for what I wanted to do.
> > > Sure rpm is in there.  But rpmdb is not.  So, no --initdb and no
> > > --rebuilddb.  They've got yum in there, but no yummain.py so yum hurls
> > > chunks immediately upon execution trying to import yummain.  What the
> > > hell good does it do to have yum in there but no yummain?!?!  But,
> > > something, fixed up, like that from all the distros would be a perfect
> > > answer (albeit somewhat less than high performance thanks to that
> > > download, but it's a single download that could be cached).  The only
> > > potential gotcha I see in there is requiring squashfs available for
> > > mounting the image.  Anyone have heartburn over that?
> > 
> > > That squashfs image has to be able to do an install or it would be
> > > useless on the netinst CD.  I'm not sure if they still have anaconda on
> > > there or not but it has to be able to do a kickstart install, so all I
> > > need is a minimal.ks kickstart file to perform essentially an unattended
> > > install of a minimal build into the target rootfs.  That's where I'm at
> > > now, trying to get that to work.
> > 
> > I got this to work with just one (err, minor, gag, couch) butt ugly
> > hack / dirty trick.  I seems, in this case, the problem with the Fedora
> > LiveOS squashfs image is that it has rpm (but no rpmdb) and yum (but no
> > yummain.py) partially installed and damn near everything else
> > (dependeices) are but it's not build with an rpm database in place.
> > 
> > The answer was to stage this with the original LiveOS fs being the stage
> > 0 and copying that to a read-write directory and then downloading and
> > force (--nodeps) an install of rpm and yum installing the full packages
> > in place for a stage 1 run time environment.  That enabled me to create
> > a stage 2 uber-minimal run time environment with rpm and yum installed
> > along with all their dependencies.  This I could then cache and is
> > capable of installing arbitrary versions of Fedora (so it only needs to
> > be done once for any given arch).
> > 
> > I've integrated it into the lxc-fedora template but it has undergone
> > minimal testing (fedora on fedora).  It has a "white listing" facility
> > in it to run a native install on distros (and also checks if the distro
> > has yum and rpm available) that will support it but that's disabled on
> > this, attached, template so it can get more testing.  White listing is
> > based on the CPE OS identifier.  What will go into the patch to the
> > project will have fedora, centos, and maybe a few others whitelisted
> > that are known to work.
> > 
> > Attached to this message is my trial balloon lxc-fedora template.  This
> > is NOT final.  This is just a test for comments and feedback.  Please
> > try it out.  Other template owners, PLEASE look at the code and see if
> > something similar makes sense for your templates!  I would love to
> > install your distros on my Fedora hosts!
> > 
> > The code is commented moderately heavy (and somewhat sarcasticly) but
> > should explain what I was trying to do (if not - tell me!).  The bulk of
> > the changes are a block of code and comments starting around line 201
> > plus a few modifications just to the "download_fedora()" routine, now at
> > line 543 (yes it was a big insert in front of it).
> > 
> > I can not emphasize this enough...  READ THE COMMENTS!  If I'm full of
> > shit doing it this way, then tell me so!
> > 
> > Once it builds that RTE the first time, it will use it for all future
> > Fedora container builds.
> > 
> > Tony:  You are on OpenSuse.  You, more than anyone else, I really need
> > to test this code by.  I'll answer your other questions about WHY
> > systemd is incompatible with 0.8.0 and earlier in another message.  But
> > this template should run and build a container.  If it doesn't, I want
> > to fix that before I submit a patch.  I could use some people on
> > AltLinux and ArchLinux to also test this out on a host and give me
> > feedback.
> > 
> > Maybe I'll see a few of you at the LinuxPlumbers conference in a few
> > days in New Orleans.
> > 
> > Regards,
> > Mike
> > 
> > > But it's all that work just to replace those few lines of code above
> > > where you can not perform a host native install of the rootfs.  That's
> > > all that's standing in the way and it's frustratingly close for the
> > > Fedora template.
> > > 
> > > We've got the following distro templates in the project, currently:
> > > 
> > > lxc-alpine
> > > lxc-alt
> > > lxc-arch
> > > lxc-busybox (is this really a distro template - I'm not sure)
> > > lxc-debian
> > > lxc-fedora
> > > lxc-opensuse
> > > lxc-oracle (really?)
> > > lxc-ubuntu / lxc-ubuntu-cloud
> > > 
> > > Who are the owners?  I don't "own" the Fedora template but I work on it.
> > > Each template owner has to solve this problem in their own way and it
> > > all boils down to the same problem.  Building that initial rootfs in the
> > > host native environment.
> > > 
> > > After that, the idea of generalizing the functions into an API form
> > > sounds like a great idea.  It could be that this catch-22 could be
> > > buried in a "create_rootfs" function which would create a cacheable
> > > rootfs.
> > > 
> > > I would love to see templates for CentOS, Scientific (SL), Slackware and
> > > maybe even RedHat EL.  NST (Network Security Toolkit) and Kali (fka
> > > BackTrack) would also be nice and I might work on them (which can I say,
> > > I'm a security fanatic).  But I gotta solve this problem first in a way
> > > I can use for the ones I might want to write down the road.
> > > 
> > > But, in the meant time, we have "the recipe for rabbit stew"...  First
> > > you catch your rabbit.  First we have to have a way to build the initial
> > > rootfs.
> > > 
> > > <aside>
> > > 
> > > This entire process reminds me of when we (I) had to bootstrap gcc into
> > > environments (HPUX and AIX years ago) which had no C compiler installed.
> > > First you cross compiled a compiler (on Solaris in my case) for your
> > > target arch and transported it there.  Then you used that "stage 1"
> > > compiler to compile the gcc compiler on the native platform (stage 2).
> > > Then you used the stage 2 compiler to compile a full stage 3 compiler
> > > where you could then confirm the outputs from the stage 2 and stage 3
> > > compilers to confirm they matched (which never seems to happen the first
> > > time you try it).  When you were done and finally confirmed working, you
> > > really needed a beer.  This is the same sort of bootstrap process.
> > > 
> > > </aside>
> > > 
> > > > -nc
> > > 
> > > Regards,
> > > Mike
> > 
> 

-- 
Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20130913/83972cb0/attachment.pgp>