[Lxc-users] Fun kernel problem: garbled /var/log/kern.log, can't create hard link to particular file in /usr/bin

[Dan Kegel]

There I was, minding my own business, running stock lxc on ubuntu 12.04,
when suddenly I couldn't install a perl update inside the container.
Turned out to be it couldn't create a hard link to /usr/bin/s2p
(though it could to /usr/bin/yes):

buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ sudo ln /usr/bin/yes
/usr/bin/yes.foo
buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ ls -l /usr/bin/yes
-rwxr-xr-x 2 root root 22912 Nov 19 22:25 /usr/bin/yes
buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ ls -l /usr/bin/s2p
-rwxr-xr-x 1 root root 53327 Nov 27 00:50 /usr/bin/s2p
buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ sudo ln /usr/bin/s2p
/usr/bin/s2p.foo
ln: failed to create hard link `/usr/bin/s2p.foo' => `/usr/bin/s2p':
No such file or directory
buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ cp /usr/bin/s2p /tmp
buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ sudo ln -s
/usr/bin/s2p /usr/bin/s2p.foo
buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ sudo rm !$
sudo rm /usr/bin/s2p.foo
buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ sudo ln
/usr/bin/s2p /usr/bin/s2p.foo
ln: failed to create hard link `/usr/bin/s2p.foo' => `/usr/bin/s2p':
No such file or directory

On the host,
uname -a says
Linux testbot01-ubu1204-temp-g-speak-unique 3.2.0-38-generic
#61-Ubuntu SMP Tue Feb 19 12:18:21 UTC 2013 x86_64 x86_64 x86_64
GNU/Linux
and /var/log/kern.log was amusingly garbled:
...
Mar 19 11:27:46 testbot01 kernel: <>[89162.011]ii:bidatrgsairseenig w
Mar 19 11:27:47 testbot01 kernel: psn,sn<7>[89166.955260] eth0: no
IPv6 routers present
Mar 19 11:27:47 testbot01 kernel: nen ag<7>[89167.130907] veth6aA6qs:
no IPv6 routers presnt
Mar 19 11:27:48 testbot01 kernel: <>[817567 t0nI uset>706 tbls-p nosd
pi7[96.965 ehvou oIv otr rsn
Mar 19 11:27:49 testbot01 kernel: >81728e: 6usetbe-gsekmi rcs ne,epng
mt-sam osee swn<7>[89168.776691] eth0: no IPv6 routers present
Mar 19 11:27:50 testbot01 kernel: dpn<4>[89169.599320] init:
buildmaster-g-speak main process ended, respawning
Mar 19 11:27:51 testbot01 kernel: [89170.594697] init:
buildmaster-g-speak main process ended, respawning

Rebooting made both problems go away.

Uptime was one day.

Whee!
- Dan