[Lxc-users] Debugging a containers firewall.. (syslog)

[Gordon Henderson]

On Thu, 1 Jul 2010, Michael H. Warfield wrote:

> On Thu, 2010-07-01 at 10:20 +0200, Daniel Lezcano wrote:
>> On 07/01/2010 09:54 AM, Gordon Henderson wrote:
>>> On Thu, 1 Jul 2010, Gordon Henderson wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm experimenting with some iptables inside a container - no real issues
>>>> there, it just works, but I'm a little confused by the logging messages..
>>>>
>>>> I'm running rsyslogd and the firewall log messages are going where they'd
>>>> normally go (ie. I've not changed any settings there), so normally I see
>>>> them in the output of dmesg and they're stored in /var/log/kern.log (this
>>>> is Debian and the rsyslogd.conf file has:
>>>>
>>>>    kern.*                          -/var/log/kern.log
>>>>
>>>> However the file kern.log seems to be missing a lot of entires that are
>>>> appearing in the output of dmesg.
>>>>
>>>> I don't currently have kernel timestamps turned on, so I can't properly
>>>> correlate dmesg output with the log-files, but I'm just wondering if there
>>>> is anything significant here - anything obvious I'm missing?
>>>
>>> Hm. Following up my own post.. I've just realised the messages are getting
>>> stored in the hosts kern.log file too, so I'm now confused. Is it actually
>>> possible to have per-container syslogs, or should it all be done on the
>>> host? I've no issues with the latter, but there doesn't seem to be a way
>>> to tag them if the host is doing all the logging... (Although since this
>>> is firewall, there are DST=i.p.address entries in the hosts kern.log file,
>>> but that's OK for iptables logging, but not for individual container
>>> sendmail, etc. logging...
>>>
>>> Using Debian stable, kernel 2.6.33.3 and LXC 0.6.5
>
>> Syslog per containers is not (yet?) supported. At the first glance,
>> after looked at the kernel code, it is not obvious to make these changes
>> because it impacts /dev/kmsg, /proc/kmsg, syslog syscall and printk
>> which need to be cleanup and modified in a consistent way to assign a
>> ring buffer per container. The netfilter logging, which use printk,
>> occurs in a interrupt context making difficult to identify the right
>> container without using the network namespace (that does not make sense
>> to create a relationship between a network namespace and the ring buffer
>> IMHO).
>
> IIRC...  This is not a "clean" problem.  That is, it's not fully
> deterministic.  I remember some discussions over this some time ago but
> I don't remember when or what forum.
>
> Basically, I see it as there are 4 categories of messages which we have
> to figure out how to dispatch.
>
> Category 1: Belongs to host.  This one is simple enough.
>
> Category 2: Belongs to container.  This one is simple too.
>
> Category 3: Relevant to both or all.  Some messages have global
> significance and should appear in both the host syslog and one or more
> of the container syslogs.  This gets ugly but doable.
>
> Category 4: Unable to determine.  This is where things really get ugly.
> This would be things like kernel messages from interrupt space where you
> can't determine the context at that point in time.  I suppose they could
> be handled like category 3 but that could clutter logs with messages
> which are not relevant to that container.  I don't know if there is a
> good solution to this one.
>
>> Tagging the logs to differentiate where they are coming from is a good
>> idea and this is probably what should be done for the host (log in the
>> container and log in the host). But the syslog per namespace framework
>> will be needed before I think.
>
> Concur there.

Thanks for the replys and insight...

However, I think I've also managed to confuse myself here too... then 
remembered that there's 2 logging systems in there - syslog and klog - 
syslog being userland, klog being the kernel (which then appears to use 
syslog to handle kernel messages!)

The (new?) rsyslogd seems to cover both logging systems whereas before 
(Debian) Lenny, there was syslog and klogd.

So from my point of view - if I revert back to syslog in a container, then 
it's fine - local/userland stuff inside a container like sendmail, etc. 
use syslog and it's logged to local files inside the container, but kernel 
stuff is picked up by the host - and the kernel stuff I'm interested in 
here is the logging produced by iptables. At this point, I can 
differentiate which container by the IP address (and I can also prefix the 
log entries with --log-prefix too.

So (for now), syslogd in a container and syslogd and klogd (or rsyslogd) 
on the host and that'll probably stop any sort of clashes happening with 
containers and the host, all trying to pick up klog messages.

Assuming I've got this right - please correct me if I'm barking!

Anyway keep up the good work - LXC is working well for me for 2 aspects of 
my work - providing me with a management tool to "containerise" servers 
(Even if it's just one container per physical server!) it gives me a well 
defined "chunk of thing" to place onto alternative hardware for whatever 
reason, and I'm running a small number of virtual asterisk servers under 
it too.

Thanks,

Gordon