[Lxc-users] Container with different architecture like arm on x86 [How-to]

[Brian K. White]

On 4/6/2011 3:26 PM, lxc at zitta.fr wrote:
> Hi,
>
> I tried to run an arm container under a x86_64 host and it works !!!!
>
> Little how-to :
>
> build a static compiled qemu-arm
>           take qemu sources and build it with :
>               ./configure --static --target-list=arm-linux-user; make
>           U will find static qemu for arm at ./arm-linux-user/qemu-arm
> use the binfmt_misc kernel module
>           mount the pseudofs :
>               mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
> have an arm container
>           let's say it is at /lxc/armcontainer
> copy qemu in the container :
>           cp ./arm-linux-user/qemu-arm /lxc/armcontainer/
> enable binfmt :
> echo
> ':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/local/bin/qemu-arm:'>/proc/sys/fs/binfmt_misc/register
> launch your container normaly.
>
> I found this cool, I hope it be useful someone else.
>
> I have made this how-to from bash history, I could have made some mistakes.
> feel free to ask if you're in troubles.
>
> regards,
>
> Guillaume ZITTA


A few questions,

The echo command references /usr/local/bin/qemu-arm, but I don't see 
that anywhere else in the recipe. Is that a x86_64 binary on the host or 
is that supposed to be a arm binary in the container, or is it simply 
ignored in this case and doesn't matter that it doesn't exist?

It sort of looks like you are telling the host x86_64 kernel to run a 
x86_64 qemu-arm executable any time it encounters an arm elf executable, 
and then since you are placing an arm qemu-arm executable in the 
container fs I guess you are implying that the arm executable you will 
be trying to run will be that arm qemu executable? Why would you do that?

foo -> qemu -> qemu -> kernel ??

ie: arm-executable foo -> arm executable qemu-arm -> x86_64 executable 
qemu-arm -> x86_64 host kernel ??

Assuming that even works. Doesn't there have to be an arm kernel in 
there somewhere? Like:

arm-foo -> arm-kernel -> x86_64-qemu-arm -> x86_64-host-kernel

I don't see the point in this. As long as you have qemu in there 
anywhere it means you are doing full cpu virtualization, avoiding which 
is pretty much the sole purpose of containers.

If it's really true that you can have qemu provide _only_ cpu 
virtualization yet somehow have the host kernel support the arm 
executables through that I guess that's a win since you have a single 
kernel doling out resources directly to all processes instead of kernels 
within kernels. Then again wouldn't that result in every single arm 
executable running inside it's own instance of qemu, auto launched by 
the binfmt? That might be ok for application containers that only run 
one process but that would be terrible for a full system container 
unless that container really only ran one process directly, an arm 
kernel. And in that case I don't see the point of doing that inside a 
container. It's already even more isolated inside qemu than what the 
container provides and the container layer just becomes pointless overhead.

But doesn't the arm kernel have rather a lot more differences than 
merely understanding the arm binary format and cpu? I would have thought 
the container would have to run an x86_64 (or i386) binary, which would 
be qemu, and that qemu would have to run an arm kernel, and all other 
arm processes would have to run in that arm kernel.

I think I need an example to illustrate a use case for this.

-- 
bkw