[lxc-devel] [PATCH] pass lxcpath to lxc_command

[Serge Hallyn]

Quoting Dwight Engen (dwight.engen at oracle.com):
> On Mon, 11 Feb 2013 16:40:39 -0500
> Stéphane Graber <stgraber at ubuntu.com> wrote:
> 
> > On 02/11/2013 03:43 PM, Serge Hallyn wrote:
> > > 
> > > The previous lxcpath patches added support for a custom LXCPATH set
> > > through a system-wide configuration file.
> > > 
> > > This was also exposed through the C api, so that a custom lxcpath
> > > could be set at the container object instanciation time, or set at
> > > runtime.
> > > 
> > > However the command sock filename was always located under the
> > > global lxcpath, which could be confusing, and would be a problem
> > > for users with insufficient perms to the system-wide lxc path (i.e.
> > > if setting lxcpath to $HOME/lxcbase).  This patch changes that by
> > > passing the lxcpath to all callers of lxc_command().
> > > 
> > > It remains to add an lxcpath command line argument to most of the
> > > command line tools (which are not using the C api) - lxc-start,
> > > lxc-info, lxc-stop, etc.
> > > 
> > > At this point it becomes tempting to do something like
> > > 
> > > 	c = lxc.Container("r1", "/var/lib/lxc")
> > > 	c2 = lxc.Container("r1", "$HOME/lxcbase")
> > > 
> > > However, that's problematic - those two will use the same directory
> > > names for cgroup directories.
> > > 
> > > What would be the best way to handle this?  One way (which I kind
> > > of like) is to give up on naming the cgroups after the container.
> > > use mkstemp for the cgroup name, let lxc keep track of the cgroup
> > > name based on the command socket, and make users use lxc-cgroup to
> > > get and change settings.
> > 
> > As we discussed on IRC, I think it'd be best to do two things:
> > 1) Add a new lxc.cgroup.directory option which would default to "lxc"
> > (our current directory) and let people easily override to another name
> > to avoid the conflict.
> 
> I like this, and I think it might also allow for groups of containers I
> was refering to in irc with a little more support in the config file
> for setting limits at the directory level (at least to a grouping depth
> of 1). For example lets say a config file looks like:
> 
>  lxc.cgroup.directory = builds
>  lxc.cgroup.directory.memory.limit_in_bytes = 1073741824
> 
> This would set /sys/fs/cgroup/memory/builds/memory.limit_in_bytes which
> would limit all containers in the "group" builds together to 1G. Any

I was thinking, rather, that

lxc.cgroup.directory = builds

with a container name of r1 would result in

/sys/fs/cgroup/memory/builds/r1/memory.limit_in_bytes.

And, if I simultaneously run another r1 from a different lxcpath,

/sys/fs/cgroup/memory/builds/r1-1/memory.limit_in_bytes.

This does conflict a bit with the idea of libcgroup-pam putting the user
into his own cgroups, which then confine the containers he creates.

For the general case, I far prefer the libcgroup-pam-based solution,
because we can set up intelligent defaults (from a distro level),
easily overrideable by an interested admin.  And it lets pam (with
privilege) create and set up the cgroups, chown them to the user so
he can create new cgroups under that.

I'm not 100% sure yet how to mix this with the lxc.cgroup.directory.
Unfortunately hierarchy in cgroups has a (reputedly - I haven't tested
myself) steep performance cost, so we don't just want to default to
/sys/fs/cgroup/memory/hallyn/builds/r1/...

I think it makes sense to say that lxc.cgroup.directory requires
privilege, and always groups at the top level.  Each distro can have
an init/upstart/systemd script which sets up configured groups, so
that then on my system I might have under /sys/fs/cgroup/memory/:

	builds/r1
	builds/r1-2
	builds/r1-3
	hallyn/r1
	hallyn/r1-1

This also gives us clear places to configure limits for all the groups -
libcgroup-pam for hallyn, and the lxc init script for builds.

thoughts?

-serge