[lxc-users] Error: /proc must be mounted

[Itamar Gal]

> You should also be able to look at /var/lib/lxcfs and see what's in there.
> There should be cgroup and proc mounts with the corrent content.

Yup - looks like it's all there.

    $ ls /var/lib/lxcfs

    cgroup
    proc

    $ mountpoint /var/lib/lxcfs

    /var/lib/lxcfs is a mountpoint

I did experience a couple of anomalies related to this mountpoint though.
First, the "tree" command produces "error opening dir" messages:

    $ sudo tree /var/lib/lxcfs

    /var/lib/lxcfs
    ├── cgroup
    │   ├── blkio [error opening dir]
    │   ├── cpu [error opening dir]
    │   ├── cpuacct [error opening dir]
    │   ├── cpuset [error opening dir]
    │   ├── devices [error opening dir]
    │   ├── freezer [error opening dir]
    │   ├── memory [error opening dir]
    │   └── perf_event [error opening dir]
    └── proc [error opening dir]

    10 directories, 0 files


Also, when I tried using the "find" command I got the following error message:

    "Transport endpoint is not connected"

I then started getting the same error with the "ls" command. So I checked to
make sure lxcfs was still running and (to my surprise) found that it wasn't.
So apparently lxcfs crashed while I was inspecting the /var/lic/lxcfs mount?
I'm not sure why that would happen. Anyway, I restarted the service:

    sudo service lxcfs start

After restarting "ls" and "find" worked again, although "tree" still does not.
Here is what I get from the "find" command:

    $ find /var/lib/lxcfs/

    /var/lib/lxcfs/
    /var/lib/lxcfs/proc
    /var/lib/lxcfs/proc/cpuinfo
    /var/lib/lxcfs/proc/meminfo
    /var/lib/lxcfs/proc/stat
    /var/lib/lxcfs/proc/uptime
    /var/lib/lxcfs/proc/diskstats
    /var/lib/lxcfs/proc/swaps
    /var/lib/lxcfs/cgroup
    /var/lib/lxcfs/cgroup/perf_event
    find: `/var/lib/lxcfs/cgroup/perf_event': Software caused connection abort
    /var/lib/lxcfs/cgroup/blkio
    /var/lib/lxcfs/cgroup/freezer
    /var/lib/lxcfs/cgroup/devices
    /var/lib/lxcfs/cgroup/memory
    /var/lib/lxcfs/cgroup/cpuacct
    /var/lib/lxcfs/cgroup/cpu
    /var/lib/lxcfs/cgroup/cpuset


> Hmmm ... that seems like a much bigger problem.
>
> Are you sure you have correct version of everything? e.g. lxc userland and
> lxcfs?
> Are there perhaps stray versions somewhere (e.g. have you compiled lxc in
> the past, and install it in the /usr/local)
>
> "which lxc-attach" and "lxc-attach --version" should give more insight.

I'm not sure that I know exactly what you mean by "correct version" in this
context, but I'm fairly certain that the only lxc software on this system has
been installed by the package manager (i.e. apt-get). Here's what I did to
double-check the state of lxc on the system. First, I followed your suggestion
and checked to see which lxc-attach binary was in my path (and check its
version number as well):

    $ which -a lxc-attach
    /usr/bin/lxc-attach

    $ type lxc-attach
    lxc-attach is hashed (/usr/bin/lxc-attach)

    $ lxc-attach --version
    2.0.0


I also checked the file-system for any other instances of lxc-attach:

    $ sudo updatedb
    $ locate lxc-attach

    /usr/bin/lxc-attach
    /usr/share/bash-completion/completions/lxc-attach
    /usr/share/man/ja/man1/lxc-attach.1.gz
    /usr/share/man/ko/man1/lxc-attach.1.gz
    /usr/share/man/man1/lxc-attach.1.gz


Just to be really sure, I also checked the entire file-system for any instances
of other lxc tools:

    $ sudo updatedb
    $ for c in /usr/bin/lxc-*; do locate $(basename $c) | grep "${c}$"; done

    /usr/bin/lxc-attach
    /usr/bin/lxc-autostart
    /usr/bin/lxc-cgroup
    /usr/bin/lxc-checkconfig
    /usr/bin/lxc-checkpoint
    /usr/bin/lxc-clone
    /usr/bin/lxc-config
    /usr/bin/lxc-console
    /usr/bin/lxc-copy
    /usr/bin/lxc-create
    /usr/bin/lxc-destroy
    /usr/bin/lxc-device
    /usr/bin/lxc-execute
    /usr/bin/lxc-freeze
    /usr/bin/lxc-info
    /usr/bin/lxc-ls
    /usr/bin/lxc-monitor
    /usr/bin/lxc-snapshot
    /usr/bin/lxc-start
    /usr/bin/lxc-start-ephemeral
    /usr/bin/lxc-stop
    /usr/bin/lxc-top
    /usr/bin/lxc-unfreeze
    /usr/bin/lxc-unshare
    /usr/bin/lxc-usernsexec
    /usr/bin/lxc-wait

So I think it's probably safe to say that there's only the one set of lxc
tools installed. Here's what dpkg has to say about the version information:

    $ dpkg-query -W "*lxc*"

    liblxc0         1.0.0~alpha1-0ubuntu14~ubuntu12.04.1
    liblxc1         2.0.5+master~20161122-1752-0ubuntu1~precise
    lxc             2.0.5+master~20161122-1752-0ubuntu1~precise
    lxc-common      2.0.5+master~20161122-1752-0ubuntu1~precise
    lxc-dbg
    lxc-dev
    lxc-templates   2.0.5+master~20161122-1752-0ubuntu1~precise
    lxc1            2.0.5+master~20161122-1752-0ubuntu1~precise
    lxcfs           2.0.4-0ubuntu1~ubuntu12.04.1~ppa1
    lxcguest
    lxctl           0.3.1+debian-1
    python3-lxc     2.0.5+master~20161122-1752-0ubuntu1~precise

Are those not the correct versions?


> You can try this command in the container. Here's from mine:
>
> # grep /proc /proc/mounts
> proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
> ...
> lxcfs /proc/cpuinfo fuse.lxcfs
> rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
> lxcfs /proc/diskstats fuse.lxcfs
> rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
> lxcfs /proc/meminfo fuse.lxcfs
> rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
> lxcfs /proc/stat fuse.lxcfs
> rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
> lxcfs /proc/swaps fuse.lxcfs
> rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
> lxcfs /proc/uptime fuse.lxcfs
> rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
>
> If you're missing the first entry, or the lxcfs entries, or don't even have
> /proc/mounts altogether, then I'm not sure what else to check.

I tried grepping in /proc/mounts as you suggest. My output isn't identical,
but I also don't see anything there that's jumping out as a problem:

    $ grep /proc /proc/mounts

    proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
    proc /proc/sys/net proc rw,nosuid,nodev,noexec,relatime 0 0
    proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
    proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
    lxcfs /proc/cpuinfo fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
    lxcfs /proc/diskstats fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
    lxcfs /proc/meminfo fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
    lxcfs /proc/stat fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
    lxcfs /proc/swaps fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
    lxcfs /proc/uptime fuse.lxcfs
rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

I should also mention that the "pgrep" command appears to be functioning
normally and I also appear to be able to access the proc file-system (in the
containers) manually without any issues. For example, this won't work:

    $ pgrep ssh | xargs ps -f -p

    Error: /proc must be mounted
      To mount /proc at boot you need an /etc/fstab line like:
          proc   /proc   proc    defaults
      In the meantime, run "mount proc /proc -t proc"

But this will:

    $ for pid in $(pgrep ssh); do cat /proc/${pid}/cmdline; echo; done

    /usr/sbin/sshd-D
    sshd: user1 [priv]
    sshd: user1 at pts/5
    sshd: user2 [priv]
    sshd: user2 at pts/6
    sshd: user2 [priv]
    sshd: user2 at pts/7

    [...]


> You could PROBABLY try to upgrade the kernel
> (http://packages.ubuntu.com/precise-updates/linux-generic-lts-trusty), to
> see if for some reason the problem is caused by missing namespace support in
> your kernel. If that doesn't work, or you can't afford downtime on this
> server, then my best advice at this point would be to "setup a new trusty
> server as lxc host"

Yeah, that sounds about right. My plan is to setup a temporary lxc host as a
backup and then upgrade the main server to Ubuntu 16.04 as you suggested
previously. Thanks again for your help!