[Lxc-users] Failed to access console after launching LXC through libvirt

unicell unicell at gmail.com
Fri Nov 9 16:14:49 UTC 2012 

On Fri, Nov 9, 2012 at 10:52 PM, Serge Hallyn <serge.hallyn at canonical.com>wrote:

> Quoting unicell (unicell at gmail.com):
> > On Fri, Nov 9, 2012 at 12:09 AM, unicell <unicell at gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I'm trying to manage LXC instances through OpenStack, which use
> libvirt as
> > > a virtualization driver layer. After launching LXC instance, I simply
> could
> > > not attach to the console.
> > >
> > > virsh # list
> > >  Id Name                 State
> > > ----------------------------------
> > > 14366 instance-00000078    running
> > >
> > > virsh # console 14366
> > > Connected to domain instance-00000078
> > > Escape character is ^]
> > >
> > >
> > > And it keeps stuck here.
> > >
> > > Initially I thought it could be a cgroup device config issue, and
> tried to
> > > "echo ‘c 5:1 rwm'
> > > >/cgroup/devices/libvirt/lxc/instance-00000078/devices.allow“, to add
> > > /dev/console access right to the container. But it does not work,
> console
> > > stilll not working.
> > >
> > > Before the cgroup tweaking, devices.list are as follows
> > >
> > > [root at localhost libvirt]# cat
> > > /cgroup/devices/libvirt/lxc/instance-0000007a/devices.list
> > >
> > > c 1:3 rwm
> > > c 1:5 rwm
> > > c 1:7 rwm
> > > c 1:8 rwm
> > > c 1:9 rwm
> > > c 5:0 rwm
> > > c 5:2 rwm
> > > c 136:* rwm
> > >
> > >
> > > I'm not quite sure what kind of issue could possibly cause this console
> > > access issue. Cause I'm now using two different guest OS templates on
> two
>
> Just a few things to help you look around:
>
> Both lxc and libvirt-lxc provide consoles by creating a pty on the host
> and bind-mounting that onto /dev/ttyN in the guest before starting init.
>
> Ubuntu on the host passes 'container=lxc' (or 'container=libvirt') to
> the container, which tells ubuntu in the container to not do some things
> (look for 'container' in /etc/init/*;  also things like mountall accept
> that they should not do some things in containers).
>
> You can investigate /dev/ttyN and /dev/console in your container by
> looking at the pid of init in the container, and looking at
>         /proc/$pid/root/dev/{console,tty*}
>
> That *should* show 136:* devices, but I assume that in the cases where
> it is not working it will show 4:* for ttyN.  If you look at
>         cat /proc/$pid/root/proc/1/mounts
> I expect you'll see a /dev is mounted in the container.  Which hides the
> bind mounts which (libvirt-)lxc has set up.
>
> -serge
>

Serge, you're right. In console not working environment, /dev is mounted
inside container. As you pointed out, it hides what (libvirt-)lxc has set
up and messed up with the conosle devices.

Thank you so much for providing this clue, and I'll dig into upstart
scripts to look at more. I doubt there're some difference in container set
up logic in 'container=lxc' and 'container=lxc-libvirt' situation, which
caused the /dev/ mounting issue.

More information for your reference. As you can see, launching by lxc and
launching by libvirt-lxc makes a big difference here.
--

console working case (started by lxc-start)
--
root at localhost:/var/lib/lxc/CN/rootfs/etc/init# cat
/proc/10018/root/proc/1/mounts
rootfs / rootfs rw 0 0
/dev/mapper/lxcvg-lxcimages / ext3
rw,relatime,errors=remount-ro,user_xattr,acl,barrier=1,data=ordered 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,relatime 0 0
devpts /dev/lxc/console devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
devpts /dev/lxc/tty1 devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
devpts /dev/lxc/tty2 devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
devpts /dev/lxc/tty3 devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
devpts /dev/lxc/tty4 devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
devpts /dev/pts devpts rw,relatime,mode=600,ptmxmode=666 0 0
devpts /dev/ptmx devpts rw,relatime,mode=600,ptmxmode=666 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0
0
none /sys/fs/fuse/connections fusectl rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /run tmpfs rw,nosuid,noexec,relatime,size=378636k,mode=755 0 0
none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0

root at localhost:/var/lib/lxc/CN/rootfs/etc/init# ls -l
/proc/10018/root/dev/{console,tty*}
lrwxrwxrwx 1 root root   11 Nov  9 23:55 /proc/10018/root/dev/console ->
lxc/console
crw-rw-rw- 1 root tty  5, 0 Oct 23 20:40 /proc/10018/root/dev/tty
crw------- 1 root tty  4, 0 Oct 23 20:01 /proc/10018/root/dev/tty0
lrwxrwxrwx 1 root root    8 Nov  9 23:55 /proc/10018/root/dev/tty1 ->
lxc/tty1
lrwxrwxrwx 1 root root    8 Nov  9 23:55 /proc/10018/root/dev/tty2 ->
lxc/tty2
lrwxrwxrwx 1 root root    8 Nov  9 23:55 /proc/10018/root/dev/tty3 ->
lxc/tty3
lrwxrwxrwx 1 root root    8 Nov  9 23:55 /proc/10018/root/dev/tty4 ->
lxc/tty4
crw------- 1 root tty  4, 5 Oct 23 20:01 /proc/10018/root/dev/tty5
crw------- 1 root tty  4, 6 Oct 23 20:01 /proc/10018/root/dev/tty6
crw------- 1 root tty  4, 7 Oct 23 20:01 /proc/10018/root/dev/tty7
crw------- 1 root tty  4, 8 Oct 23 20:01 /proc/10018/root/dev/tty8
crw------- 1 root tty  4, 9 Oct 23 20:01 /proc/10018/root/dev/tty9


console *NOT* working case (started by libvirt)
--
[root at localhost instances]#  cat /proc/2086/root/proc/1/mounts
rootfs / rootfs rw 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0
/dev/nbd14 / ext3
rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0
devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0
devfs /dev tmpfs rw,nosuid,relatime,mode=755 0 0

<-- * /dev is mounted here *

proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys proc ro,relatime 0 0
sysfs /sys sysfs ro,relatime 0 0
/dev/sda2 /selinux ext4 ro,relatime,barrier=1,data=ordered 0 0
devpts /dev/ptmx devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0

[root at localhost instances]# ls -l /proc/2086/root/dev/{console,tty*}
lrwxrwxrwx 1 root root 10 Nov  9 23:53 /proc/2086/root/dev/console ->
/dev/pts/0
lrwxrwxrwx 1 root root 10 Nov  9 23:53 /proc/2086/root/dev/tty1 ->
/dev/pts/0


--
Qiu Yu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121110/3121d9eb/attachment.html>

More information about the lxc-users mailing list