[lxc-devel] Cross arch container thoughts - Call for Discussion...

[Dwight Engen]

On Tue, 01 Apr 2014 20:33:04 -0400
"Michael H. Warfield" <mhw at WittsEnd.com> wrote:

> On Tue, 2014-04-01 at 20:18 -0400, Stéphane Graber wrote:
> > On Tue, Apr 01, 2014 at 08:09:30PM -0400, Michael H. Warfield wrote:
> > > On Tue, 2014-04-01 at 19:58 -0400, Stéphane Graber wrote:
> > > > On Tue, Apr 01, 2014 at 07:43:32PM -0400, Michael H. Warfield
> > > > wrote:
> > > > > So...
> > > > > 
> > > > > Some may have read an earlier message I wrote primarily to
> > > > > the NST (Network Security Toolkit) crowd but cc'ed to this
> > > > > list about building their ISO images in an LXC container.
> > > > > That seems to work pretty well for the like-on-like case
> > > > > (x86_64 container on x86_64 host).
> > > > > 
> > > > > The cross arch case (i686 container on x86_64 host) has been
> > > > > beating the crap out of me for days when I seem to stumble
> > > > > onto the simple answer. An i686 container running on an
> > > > > x86_64 host still reports x86_64 from the arch (uname -m)
> > > > > command and related commands.  This causes all sorts of
> > > > > problems when you're running cross-arch.  The answer would
> > > > > seem to be in the "setarch" command.  Starting an i686
> > > > > container by just running "lxc-start" and arch returns
> > > > > "x86_64".  But...  Starting it by running "setarch i686
> > > > > lxc-start -n ${Container}" and arch returns "i686" and my
> > > > > tasks seem to be working properly now.
> > > > > 
> > > > > My question is this...  If we are going to support "cross
> > > > > arch" containers, should we not be incorporating "setarch"
> > > > > into the startup of those containers?  Maybe define an arch
> > > > > type on startup?  If it's set and is not equally to the
> > > > > basearch, then run setarch?  That way, LXC could "fake" the
> > > > > container arch.
> > > > > 
> > > > > That isn't quite as simple as it sounds.  For lxc-autostart,
> > > > > it could be just an arch parameter in the config file and
> > > > > incorporate the command before firing up the container when
> > > > > arch doesn't match our basearch. It's a little more
> > > > > complicated for just lxc-start and I have no idea how that
> > > > > would be incorporated into lxc-execute.
> > > > > 
> > > > > I seem to have solved my "problem" with building NST i686
> > > > > images in an LXC i686 container on an x86_64 host but seem to
> > > > > have stumbled onto a bit of a can of worms here...  Yes, LXC
> > > > > can fake out the arch for a container (where reasonable).
> > > > > Should we?  How should we?
> > > > > 
> > > > > Thoughts?
> > > > 
> > > > Hmm, aren't we already doing that?
> > > > 
> > > > stgraber at castiana:~$ uname -m
> > > > x86_64
> > > > 
> > > > stgraber at castiana:~$ lxc-create -t download -n i386 -- -d
> > > > ubuntu -r trusty -a i386 Using image from local cache
> > > > Unpacking the rootfs
> > > 
> > > > ---
> > > > You just created an Ubuntu container (release=trusty,
> > > > arch=i386, variant=default) The default username/password is:
> > > > ubuntu / ubuntu To gain root privileges, please use sudo.
> > > 
> > > > stgraber at castiana:~$ lxc-start -n i386 -d
> > > 
> > > > stgraber at castiana:~$ lxc-attach -n i386 -- uname -m
> > > > i686
> > > 
> > > > stgraber at castiana:~$ lxc-stop -n i386 -k
> > > > stgraber at castiana:~$ grep arch .local/share/lxc/i386/config 
> > > > lxc.arch = x86
> > > 
> > > > lxc.arch lets you override the personality, we currently only
> > > > support x86 personalities (we probably should look into the
> > > > armhf/arm64 ones at some point).
> > > 
> > > Ok...  Maybe I missed that and maybe that's something that needs
> > > to be added to the CentOS and Fedora templates (Dwight?
> > > Oracle?).  It's definitely not working that way right now with
> > > the Fedora based containers.
> 
> > Works fine with Centos and Oracle if you use them through the
> > download template as the config generated by the image builder
> > always includes the appropriate lxc.arch :)
> 
> Hmmm...  Something you've done with the download templates then.  It's
> not that way in the native templates but I'll fix that.

The native Oracle template does set lxc.arch. Now if
only /proc/sys/kernel/osrelease was writable in a non-root uts
namespace... :)

> > stgraber at castiana:~$ lxc-attach -n i386 -- uname -m
> > i686
> > stgraber at castiana:~$ lxc-attach -n i386 -- cat /etc/redhat-release
> > CentOS release 6.5 (Final)
> 
> Confirmed that setting lxc.arch does the right thing.  It may be in
> the download templates but we need it in the native templates as
> well.  At least that simplifies things greatly.
> 
> Thanks!
> 
> Mike
> 
> > > 
> > > > The download template and I believe a few others will set
> > > > lxc.arch properly by default, so you only need to use setarch
> > > > for commands running during container creation.
> > > 
> > > Well, that's exactly what I'm looking for and maybe that is the
> > > answer that I just didn't know the answer that was there all
> > > along.
> > > 
> > > If that's the case, I'll be testing it out and will probably
> > > patch my templates for it in the next day or two.
> > > 
> > > Thanks!
> 
>