[lxc-users] Running LXD on those new ARM64 cloud servers by Scaleway

[Simos Xenitellis]

On Sat, Apr 29, 2017 at 10:41 AM, Fajar A. Nugraha <list at fajar.net> wrote:
> On Fri, Apr 28, 2017 at 8:20 PM, Simos Xenitellis
> <simos.lists at googlemail.com> wrote:
>>
>> Hi All,
>>
>> I've put together my notes about running LXD on those new ARM64 cloud
>> servers by Scaleway.
>>
>
> Thanks for the info.
>
>>
>> 1. These are inexpensive cloud servers. Not baremetal, but KVM. On
>> ARM64 (Cavium ThunderX) hardware.
>> 2. The Linux kernel lacks ZFS support, thus requires to compile it by
>> hand. Takes time and effort (-1). Have script (+1).
>
>
>
> How is their kernel (and ubuntu distro) differ from upstream ubuntu?
> Looking at https://www.ubuntu.com/download/server/arm ,
> http://packages.ubuntu.com/search?keywords=zfsutils-linux , and
> http://packages.ubuntu.com/search?keywords=linux-generic-hwe , I thought you
> can have ubuntu 16.04.2 as-is on arm with kernel 4.10 (from
> linux-generic-hwe-16.04-edge). Yet your blog shows kernel 4.9. Is it
> scaleway-scpecifc? Does ubuntu's kernel work?
>

Scaleway does not currently use Linux kernels from distributions.
They appear to stick to a stock kernel version and in case it was 4.9.23.
They are able to do so with this ARM64 SoC
because Cavium ThunderX is supported in mainline Linux.

You can see this at https://github.com/scaleway/kernel-tools
which they grab a specific kernel version from kernel.org
and use that to compile and create custom modules.
That process is used to compile spl and zfs (for ZFS support) in my blog post.

The compiled kernel can be found at
http://mirror.scaleway.com/kernel/aarch64/4.9.23-std-1/
(including the .config file).

A long-requested feature for Scaleway has been the ability to run
custom kernels.
It is not available yet and the closer you can have, is to exploit the "kexec"
kernel feature that some of the Scaleway kernel happened to leave enabled.
Unfortunately, the ARM64 kernel does not have "kexec" enabled.

>
>>
>> 3. LXD complains that AppArmor is not complete, thus requires config
>> option to continue. Not sure what is missing.
>
>
> My hunch is that their kernel is custom, based on upstream, and don't have
> latest apparmor kernel patches. Which is why it'd be interesting if ubuntu's
> kernel can work as-is (especially since you say it's KVM, so there should be
> minimal hardware complications).
>

There is currently a single Linux kernel for the ARM64, and it used
for both the Ubuntu and Debian images.
They have a feature to allow alternative Linux kernels, through what
they call "bootscripts".
Here is how to looks like,
https://www.scaleway.com/docs/bootscript-and-how-to-use-it/
What we can do, is ask them to produce an Ubuntu kernel and make it
available through a new bootscript.
>From our side, we need to pick the appropriate Linux kernel repository
(from http://kernel.ubuntu.com ?)
so that we can propose to them.

What Linux kernel repos shall we propose from http://kernel.ubuntu.com ?
I do not know if there is a repo with ZFS applied. If not, then a repo
with the AppArmor support that LXD wants,
would be a good start. ZFS would be able to be compiled separately, or
even be provided in a PPA.

Simos

p.s.
They even retweeted my blog post, https://twitter.com/scaleway