[lxc-devel] LXC container fails to start by complaining that it is unable to unmount the old pivot-root

Michael H. Warfield mhw at WittsEnd.com
Mon Feb 1 18:54:15 UTC 2010


On Mon, 2010-02-01 at 19:46 +0200, Ciprian Dorin, Craciun wrote: 
> Hello all!

> I have a quite strange problem: the container fails to start and
> complains about being unable to unmount the old pivot root.
>     (What is strange is that I remember that one moth ago the same
> setup worked (lxc binaries and config file, but maybe 2.6.31 kernel).
> Now neither the old binaries or the latest ones from Git don't work.)

Known problem with 0.6.5 and later.  It's under investigation and there
is a patch which was posted to the -users list which seems to work
around the problem.  It hasn't been committed to git because the nature
and origin of the problem is AFAIK not well understood and the patch is
just an interim workaround.  I'm not the original poster of that patch,
Daniel Lezcano was.  Archive of that message from the users list and the
attachment can be found here:

http://sourceforge.net/mailarchive/message.php?msg_name=4B5B6DA5.6050302%40free.fr

When I first encountered this problem, well before the 0.6.5 release in
the git sources, I could make it come and go just by having tmpfs
mounted on /dev/shm (fail) or not (work).  But I can no longer reproduce
it that way.  I have a couple of unusual configurations which can.  The
patches in 0.6.5 are very important, from a security standpoint, so they
can't be simply backed out of there either.  In some cases, it appears
to be related to having mount points mounted on top of mount points
hiding the underlying mount point.  I think the workaround is what is
addressing that. 

> I also put below the config (lxc.conf) file:

> ~~~~
> lxc.utsname = lxc-test
> lxc.rootfs = /tmp/lxc-test
> lxc.mount = /dev/null
> ~~~~
> 
>     The command used (the `container` command `busybox` is wrong, but
> it doesn't even execute it; and I know that I should use something
> behaving like init as a command):
> 
> ~~~~
> ./lxc-start --name lxc-test --rcfile ./lxc.conf ./busybox
> ~~~~
> 
>     Could some one help me?
>     Thanks,
>     Ciprian.
> 
> 
>     And the `strace -f` output:
> 
> ~~~~
> execve("./lxc-start", ["./lxc-start", "--name", "lxc-test",
> "--rcfile", "./lxc.conf", "./busybox"], [/* 17 vars */]) = 0
> uname({sys="Linux", node="erebus", ...}) = 0
> brk(0)                                  = 0x9ba2000
> brk(0x9ba2cd0)                          = 0x9ba2cd0
> set_thread_area({entry_number:-1 -> 6, base_addr:0x9ba2830,
> limit:1048575, seg_32bit:1, contents:0, read_exec_only:0,
> limit_in_pages:1, seg_not_present:0, useable:1}) = 0
> brk(0x9bc3cd0)                          = 0x9bc3cd0
> brk(0x9bc4000)                          = 0x9bc4000
> open("/proc/self/fd", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 3
> fcntl64(3, F_GETFD)                     = 0x1 (flags FD_CLOEXEC)
> getdents(3, /* 6 entries */, 32768)     = 96
> getdents(3, /* 0 entries */, 32768)     = 0
> close(3)                                = 0
> open("./lxc.conf", O_RDONLY)            = 3
> fstat64(3, {st_mode=S_IFREG|0600, st_size=72, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb78d9000
> read(3, "lxc.utsname = lxc-test\nlxc.rootf"..., 4096) = 72
> read(3, "", 4096)                       = 0
> close(3)                                = 0
> munmap(0xb78d9000, 4096)                = 0
> ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
> ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
> socket(PF_FILE, SOCK_DGRAM, 0)          = 3
> sendto(3, "\0\0\0\0lxc-test\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 4104, 0, {sa_family=AF_FILE, path=@"lxc-monitor"}, 110) = -1
> ECONNREFUSED (Connection refused)
> close(3)                                = 0
> ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
> ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
> readlink("/proc/self/fd/0", "/dev/pts/2", 4095) = 10
> rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
> rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
> signalfd4(-1, [CHLD], 8, 0)             = 3
> fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
> rt_sigaction(SIGINT, {0x8049130, ~[RTMIN RT_1], SA_SIGINFO}, {SIG_DFL,
> [], 0}, 8) = 0
> rt_sigaction(SIGQUIT, {0x8049170, ~[RTMIN RT_1], SA_SIGINFO},
> {SIG_DFL, [], 0}, 8) = 0
> socketpair(PF_FILE, SOCK_STREAM, 0, [4, 5]) = 0
> clone(Process 25320 attached (waiting for parent)
> Process 25320 resumed (parent 25319 ready)
> child_stack=0xbfca71b4, flags=CLONE_NEWNS|0x2c000000|SIGCHLD) = 25320
> [pid 25320] getpid( <unfinished ...>
> [pid 25319] close(4 <unfinished ...>
> [pid 25320] <... getpid resumed> )      = 1
> [pid 25319] <... close resumed> )       = 0
> [pid 25320] rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
> [pid 25319] read(5,  <unfinished ...>
> [pid 25320] <... rt_sigprocmask resumed> NULL, 8) = 0
> [pid 25320] close(5)                    = 0
> [pid 25320] fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
> [pid 25320] write(4, "\310q\312\277", 4) = 4
> [pid 25319] <... read resumed> "\310q\312\277", 4) = 4
> [pid 25320] read(4,  <unfinished ...>
> [pid 25319] open("/etc/mtab", O_RDONLY) = 4
> [pid 25319] fstat64(4, {st_mode=S_IFREG|0644, st_size=1539, ...}) = 0
> [pid 25319] mmap2(NULL, 4096, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78d9000
> [pid 25319] read(4, "/dev/sda4 / jfs rw,relatime,erro"..., 4096) = 1539
> [pid 25319] read(4, "", 4096)           = 0
> [pid 25319] close(4)                    = 0
> [pid 25319] munmap(0xb78d9000, 4096)    = 0
> [pid 25319] access("/mnt/cgroup/lxc-test", F_OK) = -1 ENOENT (No such
> file or directory)
> [pid 25319] rename("/mnt/cgroup/25320", "/mnt/cgroup/lxc-test") = 0
> [pid 25319] write(5, "\310q\312\277", 4) = 4
> [pid 25320] <... read resumed> "\310q\312\277", 4) = 4
> [pid 25319] read(5,  <unfinished ...>
> [pid 25320] sethostname("lxc-test", 8)  = 0
> [pid 25320] open("/dev/null", O_RDONLY) = 5
> [pid 25320] fstat64(5, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
> [pid 25320] ioctl(5, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfca69c8) = -1
> ENOTTY (Inappropriate ioctl for device)
> [pid 25320] mmap2(NULL, 4096, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78d9000
> [pid 25320] read(5, "", 4096)           = 0
> [pid 25320] close(5)                    = 0
> [pid 25320] munmap(0xb78d9000, 4096)    = 0
> [pid 25320] stat64("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777,
> st_size=1160, ...}) = 0
> [pid 25320] gettimeofday({1265046041, 620159}, NULL) = 0
> [pid 25320] open("/tmp/tmpfLf4tiB", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
> [pid 25320] unlink("/tmp/tmpfLf4tiB")   = 0
> [pid 25320] fcntl64(5, F_GETFL)         = 0x2 (flags O_RDWR)
> [pid 25320] fstat64(5, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
> [pid 25320] mmap2(NULL, 4096, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78d9000
> [pid 25320] _llseek(5, 0, [0], SEEK_CUR) = 0
> [pid 25320] _llseek(5, 0, [0], SEEK_SET) = 0
> [pid 25320] read(5, "", 4096)           = 0
> [pid 25320] close(5)                    = 0
> [pid 25320] munmap(0xb78d9000, 4096)    = 0
> [pid 25320] access("/tmp/lxc-test/dev/console", R_OK|W_OK) = 0
> [pid 25320] mount("/dev/pts/2", "/tmp/lxc-test/dev/console",
> 0x80c3ac2, MS_BIND, NULL) = 0
> [pid 25320] stat64("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777,
> st_size=1160, ...}) = 0
> [pid 25320] lstat64("/tmp/lxc-rO81F7k", 0xbfca5fd4) = -1 ENOENT (No
> such file or directory)
> [pid 25320] mkdir("/tmp/lxc-rO81F7k", 0700) = 0
> [pid 25320] mount("/tmp/lxc-test", "/tmp/lxc-rO81F7k", 0x80c3ac2,
> MS_BIND|MS_REC, NULL) = 0
> [pid 25320] chdir("/tmp/lxc-rO81F7k")   = 0
> [pid 25320] mkdir("./lxc-oldrootfs-f0F3W4", 0700) = 0
> [pid 25320] pivot_root(".", "./lxc-oldrootfs-f0F3W4") = 0
> [pid 25320] chdir("/")                  = 0
> [pid 25320] open("//lxc-oldrootfs-f0F3W4/proc/mounts", O_RDONLY) = 5
> [pid 25320] fstat64(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
> [pid 25320] mmap2(NULL, 4096, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78d9000
> [pid 25320] read(5, "rootfs / rootfs rw 0 0\n/dev/root"..., 1024) = 1024
> [pid 25320] read(5, "d,nodev,relatime,size=1048576k,n"..., 1024) = 1024
> [pid 25320] read(5, " devpts rw,relatime,mode=600,ptm"..., 1024) = 182
> [pid 25320] read(5, "", 1024)           = 0
> [pid 25320] close(5)                    = 0
> [pid 25320] munmap(0xb78d9000, 4096)    = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/dev") = -1 EBUSY (Device
> or resource busy)
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/dev/pts") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/dev/shm") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/proc") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/sys") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/mnt/cgroup") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/boot") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/stores/archlinux") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/stores/erebus-1") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/lib/modules") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/var") = -1 EBUSY (Device
> or resource busy)
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/var/run") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/var/tmp") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/usr") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/tmp") = -1 EBUSY (Device
> or resource busy)
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/tmp/pacman-packages/vtun--3.0.1--1/rootfs/.workbench/cache/makepkg-pkg")
> = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/tmp/pacman-packages/vtun--3.0.1--1/rootfs/.workbench/cache/makepkg-src")
> = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/tmp/pacman-packages/vtun--3.0.1--1/rootfs/.workbench/tools/pacman/pacman-pkg")
> = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/tmp/pacman-packages/vtun--3.0.1--1/rootfs/.workbench/cache/pacman-mirrors")
> = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/tmp/lxc-test/dev/console") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/dev") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/var") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4/tmp") = 0
> [pid 25320] oldumount("/lxc-oldrootfs-f0F3W4") = -1 EBUSY (Device or
> resource busy)
> [pid 25320] gettimeofday({1265046041, 629159}, NULL) = 0
> [pid 25320] write(2, "lxc-start: ", 11lxc-start: ) = 11
> [pid 25320] write(2, "Device or resource busy - could "..., 54Device
> or resource busy - could not unmount old rootfs) = 54
> [pid 25320] write(2, "\n", 1
> )           = 1
> [pid 25320] gettimeofday({1265046041, 629763}, NULL) = 0
> [pid 25320] write(2, "lxc-start: ", 11lxc-start: ) = 11
> [pid 25320] write(2, "failed to pivot_root to '/tmp/lx"..., 39failed
> to pivot_root to '/tmp/lxc-test') = 39
> [pid 25320] write(2, "\n", 1
> )           = 1
> [pid 25320] rmdir("/tmp/lxc-rO81F7k")   = -1 ENOENT (No such file or directory)
> [pid 25320] gettimeofday({1265046041, 630506}, NULL) = 0
> [pid 25320] write(2, "lxc-start: ", 11lxc-start: ) = 11
> [pid 25320] write(2, "failed to set rootfs for 'lxc-te"..., 35failed
> to set rootfs for 'lxc-test') = 35
> [pid 25320] write(2, "\n", 1
> )           = 1
> [pid 25320] gettimeofday({1265046041, 631000}, NULL) = 0
> [pid 25320] write(2, "lxc-start: ", 11lxc-start: ) = 11
> [pid 25320] write(2, "failed to setup the container", 29failed to
> setup the container) = 29
> [pid 25320] write(2, "\n", 1
> )           = 1
> [pid 25320] write(4, "\377\377\377\377", 4) = 4
> [pid 25319] <... read resumed> "\377\377\377\377", 4) = 4
> [pid 25320] _exit(-1)                   = ?
> Process 25320 detached
> socket(PF_FILE, SOCK_DGRAM, 0)          = 4
> sendto(4, "\0\0\0\0lxc-test\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 4104, 0, {sa_family=AF_FILE, path=@"lxc-monitor"}, 110) = -1
> ECONNREFUSED (Connection refused)
> close(4)                                = 0
> close(4)                                = -1 EBADF (Bad file descriptor)
> close(5)                                = 0
> ioctl(2, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfca71ac) = -1 ENOTTY
> (Inappropriate ioctl for device)
> close(2)                                = 0
> close(1)                                = 0
> close(0)                                = 0
> epoll_create(2)                         = 0
> epoll_ctl(0, EPOLL_CTL_ADD, 3, {EPOLLIN, {u32=163200968,
> u64=577926083638869960}}) = 0
> socket(PF_FILE, SOCK_STREAM, 0)         = 1
> bind(1, {sa_family=AF_FILE,
> path=@"/usr/local/var/lib/lxc/lxc-test/command"}, 110) = 0
> listen(1, 100)                          = 0
> epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN, {u32=163201000, u64=163201000}}) = 0
> epoll_wait(0, {{EPOLLIN, {u32=163200968, u64=577926083638869960}}}, 10, -1) = 1
> waitpid(25320, [{WIFEXITED(s) && WEXITSTATUS(s) == 255}], 0) = 25320
> socket(PF_FILE, SOCK_DGRAM, 0)          = 2
> sendto(2, "\0\0\0\0lxc-test\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 4104, 0, {sa_family=AF_FILE, path=@"lxc-monitor"}, 110) = -1
> ECONNREFUSED (Connection refused)
> close(2)                                = 0
> socket(PF_FILE, SOCK_DGRAM, 0)          = 2
> sendto(2, "\0\0\0\0lxc-test\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
> 4104, 0, {sa_family=AF_FILE, path=@"lxc-monitor"}, 110) = -1
> ECONNREFUSED (Connection refused)
> close(2)                                = 0
> open("/etc/mtab", O_RDONLY)             = 2
> fstat64(2, {st_mode=S_IFREG|0644, st_size=1539, ...}) = 0
> mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0xb78d9000
> read(2, "/dev/sda4 / jfs rw,relatime,erro"..., 4096) = 1539
> read(2, "", 4096)                       = 0
> close(2)                                = 0
> munmap(0xb78d9000, 4096)                = 0
> rmdir("/mnt/cgroup/lxc-test")           = 0
> rt_sigaction(SIGQUIT, {SIG_DFL, [], 0}, NULL, 8) = 0
> rt_sigaction(SIGINT, {SIG_DFL, [], 0}, NULL, 8) = 0
> ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfca725c) = -1 ENOTTY
> (Inappropriate ioctl for device)
> exit_group(255)                         = ?
> ~~~~
> 
> ------------------------------------------------------------------------------
> The Planet: dedicated and managed hosting, cloud storage, colocation
> Stay online with enterprise data centers and the best network in the business
> Choose flexible plans and management services without long-term contracts
> Personal 24x7 support from experience hosting pros just a phone call away.
> http://p.sf.net/sfu/theplanet-com
> _______________________________________________
> Lxc-devel mailing list
> Lxc-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/lxc-devel
> 

-- 
Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20100201/78f9c4d9/attachment.pgp>


More information about the lxc-devel mailing list