[Lxc-users] Container with different architecture like arm on x86 [How-to]

[Brian K. White]

On 4/6/2011 5:30 PM, Justin Cormack wrote:
> On Wed, 2011-04-06 at 16:45 -0400, Brian K. White wrote:
>
>> A few questions,
>>
>> The echo command references /usr/local/bin/qemu-arm, but I don't see
>> that anywhere else in the recipe. Is that a x86_64 binary on the host or
>> is that supposed to be a arm binary in the container, or is it simply
>> ignored in this case and doesn't matter that it doesn't exist?
>>
>> It sort of looks like you are telling the host x86_64 kernel to run a
>> x86_64 qemu-arm executable any time it encounters an arm elf executable,
>> and then since you are placing an arm qemu-arm executable in the
>> container fs I guess you are implying that the arm executable you will
>> be trying to run will be that arm qemu executable? Why would you do that?
>>
>> foo ->  qemu ->  qemu ->  kernel ??
>>
>> ie: arm-executable foo ->  arm executable qemu-arm ->  x86_64 executable
>> qemu-arm ->  x86_64 host kernel ??
>>
>> Assuming that even works. Doesn't there have to be an arm kernel in
>> there somewhere? Like:
>>
>> arm-foo ->  arm-kernel ->  x86_64-qemu-arm ->  x86_64-host-kernel
>>
>> I don't see the point in this. As long as you have qemu in there
>> anywhere it means you are doing full cpu virtualization, avoiding which
>> is pretty much the sole purpose of containers.
>>
>> If it's really true that you can have qemu provide _only_ cpu
>> virtualization yet somehow have the host kernel support the arm
>> executables through that I guess that's a win since you have a single
>> kernel doling out resources directly to all processes instead of kernels
>> within kernels. Then again wouldn't that result in every single arm
>> executable running inside it's own instance of qemu, auto launched by
>> the binfmt? That might be ok for application containers that only run
>> one process but that would be terrible for a full system container
>> unless that container really only ran one process directly, an arm
>> kernel. And in that case I don't see the point of doing that inside a
>> container. It's already even more isolated inside qemu than what the
>> container provides and the container layer just becomes pointless overhead.
>>
>> But doesn't the arm kernel have rather a lot more differences than
>> merely understanding the arm binary format and cpu? I would have thought
>> the container would have to run an x86_64 (or i386) binary, which would
>> be qemu, and that qemu would have to run an arm kernel, and all other
>> arm processes would have to run in that arm kernel.
>>
>> I think I need an example to illustrate a use case for this.
>>
>
> Qemu is just being used as an arm instruction set interpreter, making
> x86 system calls to the native kernel. binfmt_misc lets you run other
> architecture binaries via emulation just by executing the binary.
> Obviously its slow, but if you want to build an arm distro say it gives
> another option other than cross compiling or a native compile on a slow
> machine.
>
> Justin

Back to the first question, this actually works for binaries other than 
hello.c ? How many binaries live entirely within the high level calls 
that are really fully abstracted by the kernel?

I guess I have to try this because I don't believe it.
qemu just emulates hardware, as in the cpu and some of the supporting 
system. You can't run user executable on hardware. Only specially 
crafted free-standing things which is pretty much just bios/efi, 
bootloaders, kernels, and memtest86. Not "ls" for instance.

I'm familiar with binfmt since I used to use iBCS and then linux-abi 
ages ago to run SCO binaries on linux, and similarly to run linux 
binaries within lxrun on SCO, and similarly osrcompat to run unixware 
binaries on open server and vice-versa, and linux on freebsd, etc ...)

But in all those cases, the following always is true:

* The executables have the same cpu instruction set as the host kernel.

* The executables have the same endianness as the host kernel and 
filesystem and utilities.

And at least one or more of the following is also always true:

* The emulation layer explicitly goes to lengths to handle various 
unavoidable differences and conflicts. Like remapping syscalls that take 
different numbers or types of arguments, and exhibit different behavior, 
even though they are named the same and do nominally the same job. 
Faking various system level environmental things like fake device nodes, 
/proc entries, etc... maybe cpu registers or memory/io addresses too I 
don't know everything but I know it's not exactly trivial or ignorable.

* The emulation layer provides fail-as-graceful-as-possible stubs for 
things that can't or haven't yet been really translated.

* Users of the emulation layer simply know that only a small subset of 
things will actually work. Anything might have any sort of problem, and 
if it breaks you get to keep the pieces. It's useful in a few very 
special cases and requires significant hacks and workarounds and 
compromises, but isn't generally useful.

I mean it's not just a few things, it's things everywhere you turn, 
filesystems that return values outside the possible range, memory 
addresses outside the possible range, signed vs unsigned assumptions, 
size of int or word assumptions, ...

I would think the primary use for something like this would be to 
develop portable systems and software on fast hardware and in a more 
convenient full system environment. Isn't 99% of that very highly tied 
to the actual system and hardware? What Android apps would you want to 
run on a desktop, and how could you without something that emulates the 
android display at the very least?

I guess you should be able to run a compiler. It doesn't directly access 
any hardware. It just reads files, processes data, and writes files, and 
stdout/stderr. As long as "processes data" doesn't involve cpu-specific 
hacks that qemu doesn't emulate then that should be fine.
You could maybe compile the system, but what of it could you run? Even 
if you could run any of the results, would doing so be a valid test?

It sounds pretty cool if there is actually enough things that can 
possibly work this way.

-- 
bkw