[lxc-devel] Integration Kubernetes and LXD/LXC

[Oliver Schad]

On Mon, 1 Oct 2018 18:38:00 +0200
Christian Brauner <christian at brauner.io> wrote:

> On Mon, Oct 01, 2018 at 06:20:30PM +0200, Oliver Schad wrote:
> > Hi everybody,
> > 
> > first I want to thank the community around LXC and Canonical for
> > this great peace of software.  
> 
> Well, thank you I guess. :)

You're welcome. :-)
 
> > So in fact, we've done it. Kubernetes works together with LXC/LXD
> > and you can do "kubectl get pods", "kubectl create" and so on and
> > you have exactly one API for all your platform services and
> > applications. Configmaps with LXC? Works.  
> 
> That all sounds very interesting. One thing I'm unclear about is what
> exactly you integrated: LXC or LXD, i.e. are you using the LXC C api
> directly or are you using the LXD go client api?

We used the LXC client library (Kubelet -> LXE (with LXC client lib) ->
LXD -> LXC)

> > Do you like it?
> > 
> > For us it really makes sense. If there is a clear demand in the
> > community to have a kubernetes integration, we try to bring that
> > project to the public.  
> 
> We've had multiple requests by people. So I'm sure this is something
> people would be very interested in.
> We also had some people telling us that they were working on this but
> so far I haven't seen anyone actually doing the implementation.
> Is the code for this accessible already? I'd be very interested to
> take look even just to see what this looks like. :)

I can say, that we thought in the first analysis: we write a small
wrapper, 2 weeks, it should work.

We was WRONG. To understand the semantics of Kubelet was very special.
What is the sandbox (pod), how to do networking? That was the hard part.

Most of us heard about CRI and that it abstracts the container
runtime and Kubernetes would be wonderful flexible: it's just wrong.
Docker and semantics are really burned into Kubernetes. And of course,
CRI is an imperative interface. No freedom for decisions. It's like "do
this", "do that". And what is, if this is just wrong?

I.e. if you plan to use a LXC-Container where you host a Kubernetes
inside: Kubelet will show you how abstraction works. It finds a ZFS
mount point (cause the host uses ZFS), no problem. It just activates the
ZFS lib inside Kubelet and let find it out, how much free space is
there (you remember this tiny abstraction layer CRI?). And because there
is no /dev/zfs device, kubelet just crashes.

I don't fool you, they really request storage implementation details of
the container engine directly, without asking the CRI. That abstraction
layer is really thin and so concrete.

Next good story: LXC needs a host name, right? Ok, kubelet doesn't like
it - almost. Kubelet is the intelligent part or better - it should.
Because Kubelet makes decisions and the lower level has just to execute
them (imperative interface).

But Kubelet has assumption, i.e. that a Pod Name could be changed,
after updating Kubelet to a newer version. And why the name should
change? Because the name is created from the pod data structure which
is an obvious moving target, and that is somehow hashed. So if new
attributes appear, you get a new pod name.

What is wrong with new pod names?

Think about a production database, MySQL. You're so proud, it runs
since 3 monthes, you tuned it, you have a great monitoring, you have
great backup/restore procedure, you've tested it, you're the hero of
this MySQL server and your Webshop earns a lot of money with that thing.

But, oh bad, you've updated Kubelet and that creates a new pod name. So
that means that your old pod will just die (die as in forever, deleted,
erased, terminated including your data).

Ok, in a docker world you're stateless and you just respawn an
instance. In the real world, you're fucked, because your boss and/or
your customer is really angry.

So in fact: Kubernetes needs a wider range of requirements and on the
other side people helping to support that direction.

For code access: you can take a personal look, if you agree on keeping
the code confidential. Would it be ok?

Best Regards
Oli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20181001/05c3076c/attachment.sig>