[lxc-devel] "pre-start" hooks and avahi

[Chris Glass]

Further investigation (with trace):

It seems the failure I'm seeing is caused by not being able to talk to
dbus's system bus in the pre-start hook [1].
Looking at the code (src/lxc/start.c:355 ish), it seems pre-start
hooks are called after lxc_cmd_start and seccomp have been
initialised.
I am expecting the pre-start hooks to be executed in a pristine host
environment with some LXC variables. Is that wrong, an is it actually
run in a (more) permissive guest jail?

Is that a bug, or do I have wrong expectations about pre-start hooks?

I would be glad to help either way - I'd love to follow that rabbit
hole since it's been a great learning experience for me so far, or I
can add to the docs to prevent other people to fall into the same trap
:)

- Chris


[1]: Minimal pre-start hook: http://pastebin.ubuntu.com/6737406/
Resulting strace: http://pastebin.ubuntu.com/6737363/

On Sat, Jan 11, 2014 at 4:41 PM, Chris Glass <chris.glass at canonical.com> wrote:
> Hi,
>
> Unfortunately I don't think that's the cause of this particular issue.
> The error happens in *pre-start*, so the container is not yet started
> as far as I can tell (the documentation stipulates that it's executed
> in the host's environment).
> My changes do not need avahi to be installed on the guests at all. I
> want to query avahi from the host's environment for the info I need in
> pre-start, and put that in a file in the container's rootfs, then boot
> it normally.
>
> For the record I tried disabling rlimit_nproc in the host's avahi
> daemon (as well as setting it to a ridiculous number like 200), but
> that doesn't change anything.
>
> I'm not sure how to investigate this further... pointers welcome!
>
> - Chris
>
>
> On Fri, Jan 10, 2014 at 7:47 PM, Serge Hallyn <serge.hallyn at ubuntu.com> wrote:
>> Avahi sets its own rlimit_nproc to be exactly the # tasks it will spawn
>> as its userid.  If the avahi user in the container is the same uid #
>> as something in use on the host, then there may already be running
>> tasks and so rlimit_nproc is enforced and avahi can't run.
>>
>> If you use a user namespace this shouldn't be an issue (so long as
>> you don't share uid ranges between any containers).
>>
>> Otherwise, you have to change the avahi code to check whether it is in a
>> container before setting rlimit_nproc.  There is a patch to work around
>> this in the ubuntu trusty package, at least, if you wnat an example of
>> how to do it.
>>
>> -serge
>>
>> Quoting Chris Glass (chris.glass at canonical.com):
>>> Hi all,
>>>
>>> I'm currently hacking on a pre-start hook for the ubuntu and
>>> ubuntu-cloud templates that automatically makes ubuntu containers
>>> aware of squid-deb-proxy servers the host knows about.
>>>
>>> For this, I assume the squid-deb-proxy-client package is installed on
>>> the host. If you're not familiar with this package: it's basically a
>>> custom squid config for deb files along with an avahi config to expose
>>> an _apt_proxy._tcp service pointing to the squid.
>>> I currently run a squid-deb-proxy in a container.
>>>
>>> While running the avahi command on the host works fine (both as my
>>> user and as root), it fails when run from a pre-start script, and my
>>> knowledge of lxc is too limited to understand what could interfere
>>> with it.
>>>
>>> The failure:
>>> Running "avahi-browse -kprt _apt_proxy._tcp" in a lxc.hook.pre-start
>>> hook fails with "Failed to create client object: Access denied". It
>>> succeeds when ran at a normal shell on the host.
>>>
>>> A quick "whoami" in the same context yields "root", as expected, and
>>> switching to an unconfined apparmor profile does not change anything,
>>> so I suspect something more subtle is going on. How is the environment
>>> on "pre-start" hooks different?
>>>
>>> Could somebody shed some light here?
>>>
>>> Background:
>>> The reason for this is that I am a little frustrated to have to update
>>> my LXC container's apt proxy settings every time my squid-deb-proxy
>>> environment changes, and so this is an attempt at making this
>>> automatic. If you are curious and want to see it for yourself, have a
>>> look at [1], but please be aware that it's still work in progress.
>>>
>>> Thanks a lot for your help,
>>>
>>> - Chris
>>>
>>> links:
>>> --------
>>> [1]: https://github.com/chrisglass/lxc/blob/make-lxc-squid-deb-proxy-aware/hooks/squid-deb-proxy-client
>>> _______________________________________________
>>> lxc-devel mailing list
>>> lxc-devel at lists.linuxcontainers.org
>>> http://lists.linuxcontainers.org/listinfo/lxc-devel
>> _______________________________________________
>> lxc-devel mailing list
>> lxc-devel at lists.linuxcontainers.org
>> http://lists.linuxcontainers.org/listinfo/lxc-devel