[Lxc-users] Container with different architecture like arm on x86 [How-to]

[lxc at zitta.fr]

Le 07/04/2011 00:49, Brian K. White a écrit :
> On 4/6/2011 5:30 PM, Justin Cormack wrote:
>> On Wed, 2011-04-06 at 16:45 -0400, Brian K. White wrote:
>>
>>> A few questions,
>>>
>>> The echo command references /usr/local/bin/qemu-arm, but I don't see
>>> that anywhere else in the recipe. Is that a x86_64 binary on the host or
>>> is that supposed to be a arm binary in the container, or is it simply
>>> ignored in this case and doesn't matter that it doesn't exist?
>>>
>>> It sort of looks like you are telling the host x86_64 kernel to run a
>>> x86_64 qemu-arm executable any time it encounters an arm elf executable,
>>> and then since you are placing an arm qemu-arm executable in the
>>> container fs I guess you are implying that the arm executable you will
>>> be trying to run will be that arm qemu executable? Why would you do that?
>>>
>>> foo ->   qemu ->   qemu ->   kernel ??
>>>
>>> ie: arm-executable foo ->   arm executable qemu-arm ->   x86_64 executable
>>> qemu-arm ->   x86_64 host kernel ??
>>>
>>> Assuming that even works. Doesn't there have to be an arm kernel in
>>> there somewhere? Like:
>>>
>>> arm-foo ->   arm-kernel ->   x86_64-qemu-arm ->   x86_64-host-kernel
>>>
>>> I don't see the point in this. As long as you have qemu in there
>>> anywhere it means you are doing full cpu virtualization, avoiding which
>>> is pretty much the sole purpose of containers.
>>>
>>> If it's really true that you can have qemu provide _only_ cpu
>>> virtualization yet somehow have the host kernel support the arm
>>> executables through that I guess that's a win since you have a single
>>> kernel doling out resources directly to all processes instead of kernels
>>> within kernels. Then again wouldn't that result in every single arm
>>> executable running inside it's own instance of qemu, auto launched by
>>> the binfmt? That might be ok for application containers that only run
>>> one process but that would be terrible for a full system container
>>> unless that container really only ran one process directly, an arm
>>> kernel. And in that case I don't see the point of doing that inside a
>>> container. It's already even more isolated inside qemu than what the
>>> container provides and the container layer just becomes pointless overhead.
>>>
>>> But doesn't the arm kernel have rather a lot more differences than
>>> merely understanding the arm binary format and cpu? I would have thought
>>> the container would have to run an x86_64 (or i386) binary, which would
>>> be qemu, and that qemu would have to run an arm kernel, and all other
>>> arm processes would have to run in that arm kernel.
>>>
>>> I think I need an example to illustrate a use case for this.
>>>
>> Qemu is just being used as an arm instruction set interpreter, making
>> x86 system calls to the native kernel. binfmt_misc lets you run other
>> architecture binaries via emulation just by executing the binary.
>> Obviously its slow, but if you want to build an arm distro say it gives
>> another option other than cross compiling or a native compile on a slow
>> machine.
>>
>> Justin
> Back to the first question, this actually works for binaries other than
> hello.c ? How many binaries live entirely within the high level calls
> that are really fully abstracted by the kernel?
>
Yes I was able to boot and login in a crosscompiled gentoo.
> I guess I have to try this because I don't believe it.
> qemu just emulates hardware, as in the cpu and some of the supporting
> system. You can't run user executable on hardware. Only specially
> crafted free-standing things which is pretty much just bios/efi,
> bootloaders, kernels, and memtest86. Not "ls" for instance.
qemu has 2 modes "system" and "user"
You described "system" mode and I used "user" mode
> I'm familiar with binfmt since I used to use iBCS and then linux-abi
> ages ago to run SCO binaries on linux, and similarly to run linux
> binaries within lxrun on SCO, and similarly osrcompat to run unixware
> binaries on open server and vice-versa, and linux on freebsd, etc ...)
>
> But in all those cases, the following always is true:
>
> * The executables have the same cpu instruction set as the host kernel.
yes, but it's qemu that is executed and himself has native instruction set
> * The executables have the same endianness as the host kernel and
> filesystem and utilities.
>
> And at least one or more of the following is also always true:
>
> * The emulation layer explicitly goes to lengths to handle various
> unavoidable differences and conflicts. Like remapping syscalls that take
> different numbers or types of arguments, and exhibit different behavior,
> even though they are named the same and do nominally the same job.
> Faking various system level environmental things like fake device nodes,
> /proc entries, etc... maybe cpu registers or memory/io addresses too I
> don't know everything but I know it's not exactly trivial or ignorable.
>
But here, it is all Linux 2.6
> * The emulation layer provides fail-as-graceful-as-possible stubs for
> things that can't or haven't yet been really translated.
>
> * Users of the emulation layer simply know that only a small subset of
> things will actually work. Anything might have any sort of problem, and
> if it breaks you get to keep the pieces. It's useful in a few very
> special cases and requires significant hacks and workarounds and
> compromises, but isn't generally useful.
>
> I mean it's not just a few things, it's things everywhere you turn,
> filesystems that return values outside the possible range, memory
> addresses outside the possible range, signed vs unsigned assumptions,
> size of int or word assumptions, ...
>
> I would think the primary use for something like this would be to
> develop portable systems and software on fast hardware and in a more
> convenient full system environment. Isn't 99% of that very highly tied
> to the actual system and hardware? What Android apps would you want to
> run on a desktop, and how could you without something that emulates the
> android display at the very least?
>
> I guess you should be able to run a compiler. It doesn't directly access
> any hardware. It just reads files, processes data, and writes files, and
> stdout/stderr. As long as "processes data" doesn't involve cpu-specific
> hacks that qemu doesn't emulate then that should be fine.
> You could maybe compile the system, but what of it could you run? Even
> if you could run any of the results, would doing so be a valid test?
>
> It sounds pretty cool if there is actually enough things that can
> possibly work this way.
>
BTW, you're right : the path was uncorrect, correct one should be

echo
':arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/qemu-arm:'>/proc/sys/fs/binfmt_misc/register

because qemu was droped at the root of the container by :
cp ./arm-linux-user/qemu-arm /lxc/armcontainer/

il'll publish a correction of my howto, but after retry it to be sure 
there is not another error.

I will push forward the concept to see if there is any limitation...

Thank you for the error catching.

Guillaume ZITTA