[lxc-users] Recent LXC / LXD and shared file systen infrastructures

[Serge Hallyn]

Quoting Jäkel, Guido (G.Jaekel at dnb.de):
> Dear experts,
> 
> I wonder if the current versions of LXD (and LXC) are aware of a shared file infrastructure like NFS. I'm using LXC 0.8 since a couple of years on a setup based on a bunch of diskless bladeservers (Cisco UCS) and a central NFS filer (Netapp). All the root filesystem (the containers and the host) are a formed by individual directory trees on the NFS. All setup of resources, network and filesystem (root and shared data) is externalized from the Containers - all the configuration information is on a shared resource, too. From this, I'm able to start each container (at a time) on any host. Now it's time to upgrade (or better say rebuild) all of it to get the promises of recent cgroup handling, lxc-fs, uid/gid-shifting and so on.
> 
> I want to ask, if this design is covered by the infrastructure assumptions of current LXD. In special, if I "transfer" (copy or move) a container to another host, is it possible to configure the lxc daemon environment in such a way, that the container root file system will not be copied from host A to host B because it's "already there"? Is it save to share caching directories for images?

I'm not quite sure how your site is put together, I'd assume you have one
large NFS server and the nodes simply mount it locally?  (Nothing fancier
going on?)

Lxd stores its configuration information in a sqlite3 db, so the basic
premise of your current setup doesn't work.  In lxd you'd want to share
the 'images' with all the nodes, and let each local lxd start new local
containers based on the shared images.  You could then, if you wanted,
'publish' the complete containers as new images in the central repository.

Creating a container from an image can be done the following ways:

1. plain directory copy
2. btrfs subvolume clone
3. zfs clone
4. lvm snapshot

The images are stored with no uid shifting, so every container creation
requires uid-shifting the rootfs.  This is pretty fast, though, i.e. on
my laptop with cheap 16Gb ssd the container creation takes .5-1.5s.
The more interesting question is what we can do to speed up the transfers
for you.  One interesting thing might be to use LVs on pvs which sit on
RBD devices shared with all hosts.  I've never done done this, would
be interested to hear how it performs.

> And by the way: Is there any know way to marry NFS with an overlay filesystem facility?

According to Documentation/filesystems/overlayfs.txt you can use NFS
as a lower filesystem but not upper.  So all nodes could share base
images as NFS, start a container with local dir as upper.  When the
container is shut down they could if desired copy the changes back
onto the NFS or stash it as a delta on NFS.

lxd doesn't work with overlayfs however.

-serge