[lxc-devel] [PATCH] pass lxcpath to lxc_command

[Serge Hallyn]

Quoting Dwight Engen (dwight.engen at oracle.com):
> On Tue, 12 Feb 2013 12:10:15 -0600
> Serge Hallyn <serge.hallyn at ubuntu.com> wrote:
> 
> > Quoting Dwight Engen (dwight.engen at oracle.com):
> > > On Mon, 11 Feb 2013 16:40:39 -0500
> > > Stéphane Graber <stgraber at ubuntu.com> wrote:
> > > 
> > > > On 02/11/2013 03:43 PM, Serge Hallyn wrote:
> > > > > 
> > > > > The previous lxcpath patches added support for a custom LXCPATH
> > > > > set through a system-wide configuration file.
> > > > > 
> > > > > This was also exposed through the C api, so that a custom
> > > > > lxcpath could be set at the container object instanciation
> > > > > time, or set at runtime.
> > > > > 
> > > > > However the command sock filename was always located under the
> > > > > global lxcpath, which could be confusing, and would be a problem
> > > > > for users with insufficient perms to the system-wide lxc path
> > > > > (i.e. if setting lxcpath to $HOME/lxcbase).  This patch changes
> > > > > that by passing the lxcpath to all callers of lxc_command().
> > > > > 
> > > > > It remains to add an lxcpath command line argument to most of
> > > > > the command line tools (which are not using the C api) -
> > > > > lxc-start, lxc-info, lxc-stop, etc.
> > > > > 
> > > > > At this point it becomes tempting to do something like
> > > > > 
> > > > > 	c = lxc.Container("r1", "/var/lib/lxc")
> > > > > 	c2 = lxc.Container("r1", "$HOME/lxcbase")
> > > > > 
> > > > > However, that's problematic - those two will use the same
> > > > > directory names for cgroup directories.
> > > > > 
> > > > > What would be the best way to handle this?  One way (which I
> > > > > kind of like) is to give up on naming the cgroups after the
> > > > > container. use mkstemp for the cgroup name, let lxc keep track
> > > > > of the cgroup name based on the command socket, and make users
> > > > > use lxc-cgroup to get and change settings.
> > > > 
> > > > As we discussed on IRC, I think it'd be best to do two things:
> > > > 1) Add a new lxc.cgroup.directory option which would default to
> > > > "lxc" (our current directory) and let people easily override to
> > > > another name to avoid the conflict.
> > > 
> > > I like this, and I think it might also allow for groups of
> > > containers I was refering to in irc with a little more support in
> > > the config file for setting limits at the directory level (at least
> > > to a grouping depth of 1). For example lets say a config file looks
> > > like:
> > > 
> > >  lxc.cgroup.directory = builds
> > >  lxc.cgroup.directory.memory.limit_in_bytes = 1073741824
> > > 
> > > This would set /sys/fs/cgroup/memory/builds/memory.limit_in_bytes
> > > which would limit all containers in the "group" builds together to
> > > 1G. Any
> > 
> > I was thinking, rather, that
> > 
> > lxc.cgroup.directory = builds
> > 
> > with a container name of r1 would result in
> > 
> > /sys/fs/cgroup/memory/builds/r1/memory.limit_in_bytes.
> > 
> > And, if I simultaneously run another r1 from a different lxcpath,
> > 
> > /sys/fs/cgroup/memory/builds/r1-1/memory.limit_in_bytes.
> 
> I agree! I'm assuming the second r1 (r1-1) also had
> lxc.cgroup.directory = builds.

Right, otherwise it wouldn't need to be r1-1 :)

> I think I wasn't clear, but what I was
> wondering was how to allow setting the upper level:
> 
> /sys/fs/cgroup/memory/builds/memory.limit_in_bytes

I think that should be done by the host's lxc init script
(/etc/init/lxc.conf for us in Ubuntu).

> in order to limit 'r1' + 'r1-1' memory usage. It would be nice to be
> able to set this from lxc config, but I guess it can be done

Yeah, that might be a good place for it.  (But then parsed and set by
the init script)

> afterwards so we don't need it right away. The use case is something
> like "lxc-start N number of build containers each with a different
> version of $distro and only let them use XX amount of memory in total
> since this host machine is also used for [something]...". So r1 and r1-1
> would not have a memory limit, but builds would, limiting the aggregate
> of all containers under builds.
> 
> > This does conflict a bit with the idea of libcgroup-pam putting the
> > user into his own cgroups, which then confine the containers he
> > creates.
> > 
> > For the general case, I far prefer the libcgroup-pam-based solution,
> > because we can set up intelligent defaults (from a distro level),
> > easily overrideable by an interested admin.  And it lets pam (with
> > privilege) create and set up the cgroups, chown them to the user so
> > he can create new cgroups under that.
> 
> I think pam based for users makes sense, but I like the "system" ones
> pretty much the way they are now :)

Exactly - sounds like we agree :)

> > I'm not 100% sure yet how to mix this with the lxc.cgroup.directory.
> > Unfortunately hierarchy in cgroups has a (reputedly - I haven't tested
> > myself) steep performance cost, so we don't just want to default to
> > /sys/fs/cgroup/memory/hallyn/builds/r1/...
> > 
> > I think it makes sense to say that lxc.cgroup.directory requires
> > privilege, and always groups at the top level.  Each distro can have
> > an init/upstart/systemd script which sets up configured groups, so
> > that then on my system I might have under /sys/fs/cgroup/memory/:
> > 
> > 	builds/r1
> > 	builds/r1-2
> > 	builds/r1-3
> > 	hallyn/r1
> > 	hallyn/r1-1
> 
> I agree about lxc.cgroup.directory needing privilege and being at the
> top level (may help alleviate performance hit too). So just to make
> sure, I think we're saying it would be:
> 
> With privilege, lxc.cgroup.directory not specified (just like things
> are now, but with the possibility of r1-1 when using multiple config
> paths):
> 
>   /sys/fs/cgroup/memory/lxc/r1/memory.limit_in_bytes
>   /sys/fs/cgroup/memory/lxc/r1-1/memory.limit_in_bytes

Yup

> With privilege, lxc.cgroup.directory = builds:
> 
>   /sys/fs/cgroup/memory/builds/r1/memory.limit_in_bytes
>   /sys/fs/cgroup/memory/builds/r1-1/memory.limit_in_bytes
> 
> Without privilege, user hallyn:
> 
>   /sys/fs/cgroup/memory/hallyn/r1/memory.limit_in_bytes
>   /sys/fs/cgroup/memory/hallyn/r1-1/memory.limit_in_bytes
>  
> > This also gives us clear places to configure limits for all the
> > groups - libcgroup-pam for hallyn, and the lxc init script for builds.
> 
> Sounds good to me.

Excellent :)

-serge