[Lxc-users] Fun kernel problem: garbled /var/log/kern.log, can't create hard link to particular file in /usr/bin

[Dan Kegel]

And it's repeatable, on two different machines, with different kinds
of root filesystems.
This time, the bad file was /usr/bin/psed.

strace shows

link("/usr/bin/psed", "/usr/bin/psedxyz") = -1 ENOENT (No such file or
directory)

Same thing if I'm in /usr/bin and try it with relative paths:
link("psed", "psedxyz")                 = -1 ENOENT (No such file or directory)

I should mention: this is an ephemeral container, and I'm using aufs
and overlayfs.
cat /proc/mounts inside the container shows

rootfs / rootfs rw 0 0
none / aufs rw,relatime,si=8a0cee4266a2f3b8,noplink 0 0

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/944386 doesn't
look quite related, since psed is mode 755, but...

On the overlay filesystem, du -h says 497M

I guess for now I'll try to bypass the problem by creating the
master container after doing rm -rf /var/cache/lxc
so not so much needs to be re-installed inside the ephemeral container.
- Dan

On Tue, Mar 19, 2013 at 3:18 PM, Dan Kegel <dank at kegel.com> wrote:
> There I was, minding my own business, running stock lxc on ubuntu 12.04,
> when suddenly I couldn't install a perl update inside the container.
> Turned out to be it couldn't create a hard link to /usr/bin/s2p
> (though it could to /usr/bin/yes):
>
> buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ sudo ln /usr/bin/yes
> /usr/bin/yes.foo
> buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ ls -l /usr/bin/yes
> -rwxr-xr-x 2 root root 22912 Nov 19 22:25 /usr/bin/yes
> buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ ls -l /usr/bin/s2p
> -rwxr-xr-x 1 root root 53327 Nov 27 00:50 /usr/bin/s2p
> buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ sudo ln /usr/bin/s2p
> /usr/bin/s2p.foo
> ln: failed to create hard link `/usr/bin/s2p.foo' => `/usr/bin/s2p':
> No such file or directory
> buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ cp /usr/bin/s2p /tmp
> buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ sudo ln -s
> /usr/bin/s2p /usr/bin/s2p.foo
> buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ sudo rm !$
> sudo rm /usr/bin/s2p.foo
> buildbot at testbot01-ubu1204-temp-g-speak-unique:~$ sudo ln
> /usr/bin/s2p /usr/bin/s2p.foo
> ln: failed to create hard link `/usr/bin/s2p.foo' => `/usr/bin/s2p':
> No such file or directory
>
> On the host,
> uname -a says
> Linux testbot01-ubu1204-temp-g-speak-unique 3.2.0-38-generic
> #61-Ubuntu SMP Tue Feb 19 12:18:21 UTC 2013 x86_64 x86_64 x86_64
> GNU/Linux
> and /var/log/kern.log was amusingly garbled:
> ...
> Mar 19 11:27:46 testbot01 kernel: <>[89162.011]ii:bidatrgsairseenig w
> Mar 19 11:27:47 testbot01 kernel: psn,sn<7>[89166.955260] eth0: no
> IPv6 routers present
> Mar 19 11:27:47 testbot01 kernel: nen ag<7>[89167.130907] veth6aA6qs:
> no IPv6 routers presnt
> Mar 19 11:27:48 testbot01 kernel: <>[817567 t0nI uset>706 tbls-p nosd
> pi7[96.965 ehvou oIv otr rsn
> Mar 19 11:27:49 testbot01 kernel: >81728e: 6usetbe-gsekmi rcs ne,epng
> mt-sam osee swn<7>[89168.776691] eth0: no IPv6 routers present
> Mar 19 11:27:50 testbot01 kernel: dpn<4>[89169.599320] init:
> buildmaster-g-speak main process ended, respawning
> Mar 19 11:27:51 testbot01 kernel: [89170.594697] init:
> buildmaster-g-speak main process ended, respawning
>
> Rebooting made both problems go away.
>
> Uptime was one day.
>
> Whee!
> - Dan