[lxc-users] LXC not responsive after update

Fri Jan 22 03:27:53 UTC 2016

Do not add -eopen argument, as it limits your view.

Do this:
- strace -f -tt -r lxc-ls -f

And while strace is running and your program is hanging, find its pid (of
lxc-ls, not strace) and run lsof -p PID (if it will be hanging on socket,
this will give you a hint what that socket is).

b.

On 21 January 2016 at 23:42, Viktor Trojanovic <viktor at troja.ch> wrote:

> Hi Bostjan,
>
> Here is my strace to "lxc-ls -f". It looks the same with the other
> commands, at least the last line is identical.
>
> # strace -eopen lxc-ls -f
> open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libpython3.5m.so.1.0", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = -1 ENOENT (No
> such file or directory)
> open("/usr/lib/gconv/gconv-modules", O_RDONLY|O_CLOEXEC) = 3
> open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/bin/pyvenv.cfg", O_RDONLY)   = -1 ENOENT (No such file or
> directory)
> open("/usr/pyvenv.cfg", O_RDONLY)       = -1 ENOENT (No such file or
> directory)
> open("/proc/meminfo", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/encodings/__pycache__/__init__.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/codecs.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/encodings",
> O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/encodings/__pycache__/aliases.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/encodings/__pycache__/utf_8.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/encodings/__pycache__/latin_1.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/io.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/abc.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/_weakrefset.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/site.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/os.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/stat.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/posixpath.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/genericpath.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/_collections_abc.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/_sitebuiltins.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/sysconfig.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/_sysconfigdata.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/site-packages",
> O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/plat-linux",
> O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload",
> O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/site-packages",
> O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> open("/usr/bin/lxc-ls", O_RDONLY)       = 3
> open("/usr/bin/lxc-ls", O_RDONLY)       = 3
> open("/usr/bin", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/argparse.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/collections/__pycache__/__init__.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/operator.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/keyword.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/heapq.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/_
> heapq.cpython-35m-x86_64-linux-gnu.so",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/reprlib.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/copy.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/types.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/functools.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/weakref.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/collections",
> O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/collections/__pycache__/abc.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/copyreg.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/re.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/sre_compile.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/sre_parse.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/sre_constants.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/textwrap.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/gettext.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/locale.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/struct.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/_
> struct.cpython-35m-x86_64-linux-gnu.so",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/json/__pycache__/__init__.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/json", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC)
> = 3
> open("/usr/lib/python3.5/json/__pycache__/decoder.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/json/__pycache__/scanner.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/_json.cpython-35m-x86_64-linux-gnu.so
> ",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/json/__pycache__/encoder.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
>
> open("/usr/lib/python3.5/site-packages/lxc/__pycache__/__init__.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/site-packages/_
> lxc.cpython-35m-x86_64-linux-gnu.so",
> O_RDONLY|O_CLOEXEC) = 3
> open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/liblxc.so.1", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libseccomp.so.2", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libcgmanager.so.0", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libnih.so.1", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libnih-dbus.so.1", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libdbus-1.so.3", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libsystemd.so.0", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/liblzma.so.5", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/liblz4.so.1", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libgcrypt.so.20", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libgpg-error.so.0", O_RDONLY|O_CLOEXEC) = 3
> open("/proc/self/cgroup", O_RDONLY|O_CLOEXEC) = 3
> open("/etc/lxc/lxc.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such
> file or directory)
> open("/usr/lib/python3.5/__pycache__/subprocess.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/signal.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/enum.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/warnings.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/_
> posixsubprocess.cpython-35m-x86_64-linux-gnu.so",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/
> select.cpython-35m-x86_64-linux-gnu.so",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/selectors.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/math.cpython-35m-x86_64-linux-gnu.so
> ",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/threading.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/traceback.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/linecache.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/tokenize.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/token.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/etc/lxc/lxc.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such
> file or directory)
> open("/usr/lib/python3.5/__pycache__/shutil.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/fnmatch.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/tarfile.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/grp.cpython-35m-x86_64-linux-gnu.so",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/bz2.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/_compression.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/_bz2.cpython-35m-x86_64-linux-gnu.so
> ",
> O_RDONLY|O_CLOEXEC) = 3
> open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libbz2.so.1.0", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/lzma.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/_lzma.cpython-35m-x86_64-linux-gnu.so
> ",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/tempfile.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/random.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/__pycache__/hashlib.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/_
> hashlib.cpython-35m-x86_64-linux-gnu.so",
> O_RDONLY|O_CLOEXEC) = 3
> open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libcrypto.so.1.0.0", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/lib-dynload/_
> random.cpython-35m-x86_64-linux-gnu.so",
> O_RDONLY|O_CLOEXEC) = 3
> open("/dev/urandom", O_RDONLY|O_CLOEXEC) = 3
> open("/usr/lib/python3.5/encodings/__pycache__/ascii.cpython-35.pyc",
> O_RDONLY|O_CLOEXEC) = 4
> open("/var/lib/lxc", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
> open("/proc/net/unix", O_RDONLY)        = 4
> open("/run/lxc/lock//var/lib/lxc/.fileserver", O_RDWR|O_CREAT, 0600) = 4
> open("/var/lib/lxc/fileserver/config", O_RDONLY) = 5
> open("/usr/share/lxc/config/archlinux.common.conf", O_RDONLY) = 6
> open("/usr/share/lxc/config/common.conf", O_RDONLY) = 7
> open("/usr/share/lxc/config/common.conf.d/",
> O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 8
>
> I can't make much of that. I see that it cannot find a few files but
> in my understanding these are optional anyway.
>
> Maybe you see something I don't?
>
> Viktor
>
> On Thu, Jan 21, 2016 at 6:33 PM, Bostjan Skufca <bostjan at a2o.si> wrote:
> > strace command is your friend.
> >
> > b.
> >
> >
> > On 21 January 2016 at 12:12, Viktor Trojanovic <viktor at troja.ch> wrote:
> >>
> >>
> >>
> >> On 20.01.2016 23:50, Fajar A. Nugraha wrote:
> >>
> >> On Thu, Jan 21, 2016 at 5:49 AM, Fajar A. Nugraha <list at fajar.net>
> wrote:
> >>>
> >>> On Thu, Jan 21, 2016 at 5:23 AM, Viktor Trojanovic <viktor at troja.ch>
> >>> wrote:
> >>>>
> >>>> I just did a system upgrade on my Arch System which included updating
> >>>> the kernel, systemd and lxc to the newest versions. After having done
> so, I
> >>>> cannot interact with my Linux container any longer. The system within
> the
> >>>> container still seems to work fine and can be contacted from outside
> (Samba
> >>>> server) but if I try to use one of the lxc commands to query or
> otherwise
> >>>> interact with the container (e.g. lxc-ls -f, lxc-stop, lxc-console,
> >>>> lxc-attach), the command hangs until I cancel it with CTRL+C.
> >>>>
> >>>> I get the following message when cancelling lxc-ls -f (as root):
> >>>>
> >>>> ^CTraceback (most recent call list):
> >>>>   File "/usr/bin/lxc-ls", line 432, in <module>
> >>>>     containers = get_containers(root=True)
> >>>>   File "user/bin/lxc-ls", line 261, in get_containers
> >>>>     if container.controllable:
> >>>> KeyboardInterrupt
> >>>>
> >>>> Regular lxc-ls works normal, by the way.
> >>>>
> >>>> I can probably just reboot the server but I still wanted to ask around
> >>>> if anyone has an idea why this is happening and what I could do except
> >>>> rebooting to regain control of LXC? I tried already systemctl restart
> lxc
> >>>> but that doesn't help.
> >>>>
> >>>
> >>>
> >>> Do you have lxcfs installed? If yes, this should be a know issue. When
> >>> you restart lxcfs, all existing running containers that use it will be
> >>> unable to access lxcfs-provided resources. AFAIK restarting lxc
> service does
> >>> restart running containers.
> >>>
> >>
> >> That should be: AFAIK restarting lxc service does NOT restart running
> >> containers.
> >>
> >>
> >>> Try killing one of those containers (lxc-stop -k -n ...), start it, and
> >>> then test again. If it works, do the same for other containers.
> >>>
> >>> --
> >>> Fajar
> >>
> >>
> >> Hi Fajar,
> >>
> >> I'm not using lxcfs, just regular directory based containers.
> >>
> >> I cannot do anything with the containers.
> >>
> >> lxc-stop -k
> >> lxc-ls -f
> >> lxc-attach
> >> lxc-console
> >> systemctl stop lxc (or restart lxc)
> >>
> >> All these commands hang forever, I have to interrupt with CTRL+C and no
> >> output is written to the logs. But the container system keeps working
> fine,
> >> so does the host system.
> >>
> >> Viktor
> >>
> >>
> >> _______________________________________________
> >> lxc-users mailing list
> >> lxc-users at lists.linuxcontainers.org
> >> http://lists.linuxcontainers.org/listinfo/lxc-users
> >
> >
> >
> > _______________________________________________
> > lxc-users mailing list
> > lxc-users at lists.linuxcontainers.org
> > http://lists.linuxcontainers.org/listinfo/lxc-users
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20160122/4fbd6be7/attachment.html>