[lxc-devel] RfC: Script hooks

[Ciprian Dorin, Craciun]

On Sun, Jan 10, 2010 at 6:42 AM, Daniel Lezcano <daniel.lezcano at free.fr> wrote:
> Michael Tokarev wrote:
>> Michael Holzt wrote:
>>
>>> I believe it would be nice it it was able to hook scripts in the process
>>> of container starting and stopping at some points, e.g. before starting
>>> or after pivot_root but before giving over to the container etc.
>>>
>>> Why? For example i would like to run a script which setups ebtables rules
>>> restricting the network access of the container. I think there are other
>>> uses for such hooks as well.
>>>
>>> Comments?
>>>
>>
>> It's _definitely_ useful, and in fact most of the ad-hoc things
>> already implemented in lxc utils can be easily done in hooks
>> instead of the utilities directly.  Even the network setup as
>> currently done is quite limited.  But people says it's slow --
>> starting 10000 containers with hooks in use takes about 0.01s
>> longer than without hooks....
>>
>> (Additional) network setup, namespace preparation, variable
>> expansion (mount paths for example), and alot of other things -
>> that's many different possibilities.  For now all that stuff
>> gets implemented directly in lxc utilities, using poor substitute
>> of tools available thanks to shell and other Unix utilities.
>>
> You can write a shell script doing all the setup,  execing of /sbin/init
> and run this script with lxc-start as well.
> The configuration done by lxc is the minimal needed to run a small
> embedded system, for example an apache server, a HPC job or sshd.
> When you spawn thousand of small applications, it's much more faster to
> have a single lxc process doing the setup than execing shell script code.
> For a system container, all the scripts are located in the rootfs of the
> container and ran by the init.d services.
>
> But, I agree that providing a mechanism to invoke shell script is a must
> have.

    Even though I'm a very new user of LXC (about 4 days :) ), may I
add my opinion about the subject?
    (I'll describe my context, and what I've tried, but if you want
just my conclusion please skip at the end :) )

=== Context and experience ===

    Let me first describe my usage context: at my university I have my
desktop (cheap hardware :( ) act as server, hosting multiple services.
For this I'm currently using Xen with a couple of domU's, but this is
not very flexible / efficient. So I want to move to a more lightweight
solution.
    My target would be not to drop Xen, but to keep only one domU for
all the normal services (mail, www, etc.) but in separate containers
(inside the domU). (I need Xen, because I also want to play with other
OS's like *BSD...) The services domU will be a lightweight homebrew
Linux with only the minimal tools needed to bootstrap the containers
(mainly busybox + lxc).

    For this I had to choose between Vserver and LXC (chroot is not an
option). At first I've tried Vserver (the documentation is not very
good, but a little better (as in more comprehensive) than for LXC :(
). Unfortunately Vserver is highly centered around running a
full-blown operating system, and the initialization is based (almost
entirely?) on shell scripts. So it took a while to reverse-engineer
the scripts to weed-out the actual binary call sequence to bootstrap
the container. (It didn't went into production yet.)

    Then I've tried LXC, and I was surprised on how little you need to
make it bootstrap a container: busybox for bootstraping the host
system, lxc for the containers, and that's all. (No grep, sed, awk,
find, file, etc.) Even more: I've looked into the source code and it's
so plain simple, that I could just understand what happens and I could
write my own initialization tools.
    (I'll give this a try.)

=== Conclusion ===

    So as Daniel said:

> The configuration done by lxc is the minimal needed to run a small
> embedded system, for example an apache server, a HPC job or sshd.

    Please keep the tools as simple (in terms of complexity not in
terms of features), and without too many dependencies... The hooks
might be welcomed, but please don't make them the core of the tools,
let them be optional only for special cases. (As the name suggests,
their "hooks" (as in little pieces of code that should mainly
observe), not "drivers" or "config-lets" :), etc.)

    And Daniel also hit a **very** important issue, that should not be
overlooked: <<HPC jobs, or batch scheduling>>: at our university we
have a brand new cluster (50 * 8 core blades), and we plan to use
slurm as a scheduler. Now we took into consideration an option to jail
somehow the jobs so that it's impossible to interfere / observe other
jobs: the solution might just be LXC (I've initially thought about
Vserver, but see above why it's not so feasible). (Also the same user
jailing could / should be done also on the front-end nodes...)

    Thanks and keep up the good work, (sorry for writing so much :)
and maybe wasting your time :) )
    Ciprian.

    P.S.: A little bit of criticism: you really need to update the
web-page, because my first impression was that the project is
dead-and-buried... Also the documentation (the man-pages are great but
their only reference points) is a little thin, the main doc (lxc.sgml)
is mostly based on examples: and for example it doesn't say that
lxc-start uses /etc/mtab to find out where the cgroup file system is
located, etc.