[lxc-users] I/O error for logrotate in LXC

[Richard Hector]

On 18/06/20 1:11 am, Harald Dunkel wrote:
> Hi folks,
> 
> on some LXC containers I get I/O errors for the logrotate
> service (systemd):
> 
> root at il04:~# systemctl status logrotate
> * logrotate.service - Rotate log files
>    Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor
> preset: enabled)
>    Active: failed (Result: exit-code) since Wed 2020-06-17 00:00:11
> CEST; 15h ago
>      Docs: man:logrotate(8)
>            man:logrotate.conf(5)
>   Process: 359762 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf
> (code=exited, status=1/FAILURE)
>  Main PID: 359762 (code=exited, status=1/FAILURE)
> 
> Jun 17 00:00:00 il04.ac.aixigo.de systemd[1]: Starting Rotate log files...
> Jun 17 00:00:11 il04.ac.aixigo.de logrotate[359762]: Failed to kill unit
> rsyslog.service: Input/output error
> Jun 17 00:00:11 il04.ac.aixigo.de logrotate[359762]: error: error
> running non-shared postrotate script for /var/log/syslog of
> '/var/log/syslog
> Jun 17 00:00:11 il04.ac.aixigo.de logrotate[359762]: '
> Jun 17 00:00:11 il04.ac.aixigo.de systemd[1]: logrotate.service: Main
> process exited, code=exited, status=1/FAILURE
> Jun 17 00:00:11 il04.ac.aixigo.de systemd[1]: logrotate.service: Failed
> with result 'exit-code'.
> Jun 17 00:00:11 il04.ac.aixigo.de systemd[1]: Failed to start Rotate log
> files.
> 
> root at il04:~# ps -ef | grep rsyslog
> root      146014       1  0 Jun08 ?        00:00:06 /usr/sbin/rsyslogd
> -n -iNONE
> root      377934  377898  0 15:09 pts/7    00:00:00 grep rsyslog
> root at il04:~# kill -HUP 146014
> root at il04:~# echo $?
> 0
> 
> 
> Host is Debian 10, LXC is version 4.0.2.
> The client runs Debian 10 as well.
> 
> This doesn't happen on all LXC clients and hosts, but I haven't
> detected a scheme yet. On my "real" hosts there are no I/O errors,
> AFAICT.
> 
> 
> Regards
> Harri

Hi Harri,

I've seen this as well, but haven't had a good resolution. I haven't
seen it with buster on buster though. My hosts are Xen VMs (from a VPS
provider) with LXC inside, and I put it down to some oddity of the
provided image. I installed a new system with buster, and haven't seen
it since.

On the containers that were still stuck in that environment, I used a
workaround ... but I can't find any of those still in use. It involved
editing /etc/logrotate.d/rsyslog, and changing the postrotate hook.
Normally it calls /usr/lib/rsyslog/rsyslog-rotate, which calls
"systemctl kill -s HUP rsyslog.service", which fails. Switching to use
the non-systemd alternative in that file "invoke-rc.d rsyslog rotate >
/dev/null" might be the valid workaround.

Hope that helps.

If as you say it can still happen on a Debian 10 host, then I'm still
interested in finding out more about what kind of permissions might be
missing, and how to reinstate them.

Richard