[lxc-devel] devices and containers

[Michael H. Warfield]

On Thu, 2014-11-13 at 13:40 -0500, Michael J Coss wrote:
> Sorry for the long delay in responding but we had a demo of our systems 
> with outside analysts and it's been a long couple of weeks in preparing 
> for the demo.  I'm looking forward to finding a set of common ground 
> changes that could be pushed upstream to better support device use in 
> containers.

> So to clarify our approach, we are running unmodified Linux distro's 
> with a full desktop environment in a container.  There are modifications 
> to the host to be sure, but the container's themselves are straight out 
> of the box distros.  Similar to Michael Warfield, we have created a 
> private dev directory for each container in:

> /etc/lxc/<container name>/dev

> and use a local fstab to mount it on the containers /dev during lxc-start.

One significant gotcha there.  If /etc and the container rootfs are on
the same block dev major:minor and the container is running systemd as
the init system manager, systemd will mount devtmpfs on /dev/ in the
container.  This was pointed out to us by the systemd people which they
have documented.  Bind mounting does not help since systemd checks the
underlying major:minor dev of the block device.  If they are not on the
same block device, you'll be ok.  But, if they are, you'll get the
documented device collisions and ill behavior known from systemd.

> The key trick here was to force uevents to be processed only by the host 
> first, and then forwarded to the appropriate container(s) based on 
> system policy.  Our user space daemon, udevns, in addition to forwarding 
> uevent messages, like udevd in older versions of Unix/Linux, actually 
> manages the creation/deletion of device nodes in the container's /dev as 
> it receives the uevents. The nodes are created, and the messages are 
> forwarded to the specific container(s).  In addition, we specify what 
> access is allowed to what devices in a given container via lxc.conf. 
> Obviously udevns can create any device but we allow (and expect) for a 
> container to run udevd and process uevents as it sees fit within the 
> confines of the restrictions imposed in lxc.conf. Ideally, we should be 
> adjusting the device restrictions on the fly to remove/add access in the 
> container as these hotplug events occur but we don't do that at the 
> moment.  So in our scenario, we start a container with a headless X 
> session.  Adding the display and input devices (keyboard, and mouse) 
> after startup.
> 
> Obviously sysfs is yet another can of worms.  Our solution is to run a 
> FUSE and filter sysfs.  Right now the filter is a bit simplistic, in 
> that it is only filtering 'uevent' files from view in the container.  
> More filter/modifying needs be added to make it more of a container's 
> view of the system, and not the host's.
> 
> I have finally gotten approval to open source the code that implements 
> our udevns, and the kernel changes to support that.  I will be preparing 
> that over the next few weeks.  I'm still working on getting the sysfs 
> FUSE opened.
> 
> BR,
> Michael J Coss
> 

-- 
Michael H. Warfield (AI4NB) | (770) 978-7061 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 465 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20141113/e5fdbc3a/attachment.sig>