[Lxc-users] cannot start any more any container?!

[Ulli Horlacher]

On Wed 2011-10-19 (19:37), Papp Tamas wrote:

> > Besides my problem with "cannot stop/kill lxc-start" (see other mail), I
> > have now an even more severe problem: I cannot start ANY container anymore!
> >
> > I boot the container with:
> >
> > root at vms1:/lxc# lxc-start -f /data/lxc/vmtest1.cfg -n vmtest1 -d -o /data/lxc/vmtest1.log
> 
> What is in log if you don't use -d but -l DEBUG?

      lxc-start 1319050532.778 DEBUG    lxc_conf - allocated pty '/dev/pts/17' (4/5)
      lxc-start 1319050532.778 DEBUG    lxc_conf - allocated pty '/dev/pts/18' (6/7)
      lxc-start 1319050532.778 DEBUG    lxc_conf - allocated pty '/dev/pts/19' (8/9)
      lxc-start 1319050532.778 DEBUG    lxc_conf - allocated pty '/dev/pts/20' (10/11)
      lxc-start 1319050532.778 INFO     lxc_conf - tty's configured
      lxc-start 1319050532.778 DEBUG    lxc_console - using '/dev/null' as console
      lxc-start 1319050532.778 DEBUG    lxc_start - sigchild handler set
      lxc-start 1319050532.778 INFO     lxc_start - 'vmtest1' is initialized
      lxc-start 1319050532.784 DEBUG    lxc_conf - instanciated veth 'vethPjfwg9/vethYBPlDf', index is '19'

That's all.


> > But nothing happens, there is only a lxc-start process dangling around:
> >
> > root at vms1:/lxc# psg vmtest1
> > USER       PID  PPID %CPU    VSZ COMMAND
> > root     31571     1  0.0  20872 lxc-start -f /data/lxc/vmtest1.cfg -n vmtest1 -d -o /data/lxc/vmtest1.log
> 
> ps axf ?

root at vms1:/lxc# ps axf | grep vmtest1
31571 ?        Ds     0:00 lxc-start -f /data/lxc/vmtest1.cfg -n vmtest1 -d -o /data/lxc/vmtest1.log
 2171 ?        Ds     0:00 lxc-start -f /data/lxc/vmtest1.cfg -n vmtest1 -l DEBUG -d -o /data/lxc/vmtest1.log


> > The logfile is empty:
> >
> > root at vms1:/lxc# l vmtest1.log
> > -RW-               0 2011-10-19 19:09 vmtest1.log
> 
> What about strace -ff -s 10000 ?

root at vms1:/lxc# strace -o /tmp/lxc.log -ff -s 10000 lxc-start -f /data/lxc/vmtest1.cfg -n vmtest1 -l DEBUG -o /data/lxc/vmtest1.log &

Uhh...? Does not write /tmp/lxc.log !

Ok, then with:

root at vms1:/lxc# strace -ff -s 10000 lxc-start -f /data/lxc/vmtest1.cfg -n vmtest1 -l DEBUG -o /data/lxc/vmtest1.log 2>/tmp/lxc.log &

The contents of /tmp/lxc.log does not make much sense to me. It grows
rapidly, after a few seconds it has 177 MB and so I terminated the strace.

execve("/opt/lxc-0.7.4.1/bin/lxc-start", ["lxc-start", "-f", "/data/lxc/vmtest1.cfg", "-n", "vmtest1", "-l", "DEBUG", "-o", "/data/lxc/vmtest1.log"], [/* 31 vars */]) = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc3ed14b000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/opt/lxc-0.7.4.1/lib/tls/x86_64/liblxc.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/lxc-0.7.4.1/lib/tls/x86_64", 0x7fff7b417ba0) = -1 ENOENT (No such file or directory)
open("/opt/lxc-0.7.4.1/lib/tls/liblxc.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/lxc-0.7.4.1/lib/tls", 0x7fff7b417ba0) = -1 ENOENT (No such file or directory)
open("/opt/lxc-0.7.4.1/lib/x86_64/liblxc.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/lxc-0.7.4.1/lib/x86_64", 0x7fff7b417ba0) = -1 ENOENT (No such file or directory)
open("/opt/lxc-0.7.4.1/lib/liblxc.so.0", O_RDONLY) = 3
(...)
open("/dev/tty", O_RDWR|O_CREAT|O_APPEND|O_CLOEXEC, 0600) = 14
getuid()                                = 0
write(3, "      lxc-start 1319051254.870 DEBUG    lxc_console - using '/dev/tty' as console\n", 82) = 82
ioctl(14, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig -icanon -echo ...}) = 0
ioctl(14, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig -icanon -echo ...}) = 0
ioctl(14, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig -icanon -echo ...}) = 0
ioctl(14, SNDCTL_TMR_CONTINUE or TCSETSF, {B38400 opost -isig -icanon -echo ...}) = ? ERESTARTSYS (To be restarted)
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
ioctl(14, SNDCTL_TMR_CONTINUE or TCSETSF, {B38400 opost -isig -icanon -echo ...}) = ? ERESTARTSYS (To be restarted)
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
--- SIGTTOU (Stopped (tty output)) @ 0 (0) ---

The last 3 lines are then repeated endless.

And the load goes up:

root at vms1:/lxc# uptime 
 21:13:28 up 9 days,  3:23,  2 users,  load average: 6.01, 5.95, 5.22

The host machine is still responsive, I cannot see a CPU munching
process with top.



-- 
Ullrich Horlacher              Server- und Arbeitsplatzsysteme
Rechenzentrum                  E-Mail: horlacher at rus.uni-stuttgart.de
Universitaet Stuttgart         Tel:    ++49-711-685-65868
Allmandring 30                 Fax:    ++49-711-682357
70550 Stuttgart (Germany)      WWW:    http://www.rus.uni-stuttgart.de/