On Fri, Nov 9, 2012 at 10:52 PM, Serge Hallyn <span dir="ltr"><<a href="mailto:serge.hallyn@canonical.com" target="_blank">serge.hallyn@canonical.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">Quoting unicell (<a href="mailto:unicell@gmail.com">unicell@gmail.com</a>):<br>
> On Fri, Nov 9, 2012 at 12:09 AM, unicell <<a href="mailto:unicell@gmail.com">unicell@gmail.com</a>> wrote:<br>
><br>
> > Hi,<br>
> ><br>
> > I'm trying to manage LXC instances through OpenStack, which use libvirt as<br>
> > a virtualization driver layer. After launching LXC instance, I simply could<br>
> > not attach to the console.<br>
> ><br>
> > virsh # list<br>
> > Id Name State<br>
> > ----------------------------------<br>
> > 14366 instance-00000078 running<br>
> ><br>
> > virsh # console 14366<br>
> > Connected to domain instance-00000078<br>
> > Escape character is ^]<br>
> ><br>
> ><br>
> > And it keeps stuck here.<br>
> ><br>
> > Initially I thought it could be a cgroup device config issue, and tried to<br>
> > "echo ‘c 5:1 rwm'<br>
> > >/cgroup/devices/libvirt/lxc/instance-00000078/devices.allow“, to add<br>
> > /dev/console access right to the container. But it does not work, console<br>
> > stilll not working.<br>
> ><br>
> > Before the cgroup tweaking, devices.list are as follows<br>
> ><br>
> > [root@localhost libvirt]# cat<br>
> > /cgroup/devices/libvirt/lxc/instance-0000007a/devices.list<br>
> ><br>
> > c 1:3 rwm<br>
> > c 1:5 rwm<br>
> > c 1:7 rwm<br>
> > c 1:8 rwm<br>
> > c 1:9 rwm<br>
> > c 5:0 rwm<br>
> > c 5:2 rwm<br>
> > c 136:* rwm<br>
> ><br>
> ><br>
> > I'm not quite sure what kind of issue could possibly cause this console<br>
> > access issue. Cause I'm now using two different guest OS templates on two<br>
<br>
</div></div>Just a few things to help you look around:<br>
<br>
Both lxc and libvirt-lxc provide consoles by creating a pty on the host<br>
and bind-mounting that onto /dev/ttyN in the guest before starting init.<br>
<br>
Ubuntu on the host passes 'container=lxc' (or 'container=libvirt') to<br>
the container, which tells ubuntu in the container to not do some things<br>
(look for 'container' in /etc/init/*; also things like mountall accept<br>
that they should not do some things in containers).<br>
<br>
You can investigate /dev/ttyN and /dev/console in your container by<br>
looking at the pid of init in the container, and looking at<br>
/proc/$pid/root/dev/{console,tty*}<br>
<br>
That *should* show 136:* devices, but I assume that in the cases where<br>
it is not working it will show 4:* for ttyN. If you look at<br>
cat /proc/$pid/root/proc/1/mounts<br>
I expect you'll see a /dev is mounted in the container. Which hides the<br>
bind mounts which (libvirt-)lxc has set up.<br>
<span class="HOEnZb"><font color="#888888"><br>
-serge<br></font></span></blockquote><div><br></div><div>Serge, you're right. In console not working environment, /dev is mounted inside container. As you pointed out, it hides what (libvirt-)lxc has set up and messed up with the conosle devices. </div>
<div><br></div><div>Thank you so much for providing this clue, and I'll dig into upstart scripts to look at more. I doubt there're some difference in container set up logic in 'container=lxc' and 'container=lxc-libvirt' situation, which caused the /dev/ mounting issue.</div>
<div><br></div><div>More information for your reference. As you can see, launching by lxc and launching by libvirt-lxc makes a big difference here.</div><div>--</div><div><br></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div class="gmail_quote"><div>console working case (started by lxc-start)</div></div><div class="gmail_quote"><div>--</div></div><div class="gmail_quote"><div><div>root@localhost:/var/lib/lxc/CN/rootfs/etc/init# cat /proc/10018/root/proc/1/mounts</div>
</div></div><div class="gmail_quote"><div><div>rootfs / rootfs rw 0 0</div></div></div><div class="gmail_quote"><div><div>/dev/mapper/lxcvg-lxcimages / ext3 rw,relatime,errors=remount-ro,user_xattr,acl,barrier=1,data=ordered 0 0</div>
</div></div><div class="gmail_quote"><div><div>proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0</div></div></div><div class="gmail_quote"><div><div>sysfs /sys sysfs rw,relatime 0 0</div></div></div><div class="gmail_quote">
<div><div>devpts /dev/lxc/console devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0</div></div></div><div class="gmail_quote"><div><div>devpts /dev/lxc/tty1 devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0</div>
</div></div><div class="gmail_quote"><div><div>devpts /dev/lxc/tty2 devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0</div></div></div><div class="gmail_quote"><div><div>devpts /dev/lxc/tty3 devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0</div>
</div></div><div class="gmail_quote"><div><div>devpts /dev/lxc/tty4 devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0</div></div></div><div class="gmail_quote"><div><div>devpts /dev/pts devpts rw,relatime,mode=600,ptmxmode=666 0 0</div>
</div></div><div class="gmail_quote"><div><div>devpts /dev/ptmx devpts rw,relatime,mode=600,ptmxmode=666 0 0</div></div></div><div class="gmail_quote"><div><div>none /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0</div>
</div></div><div class="gmail_quote"><div><div>none /sys/fs/fuse/connections fusectl rw,relatime 0 0</div></div></div><div class="gmail_quote"><div><div>none /sys/kernel/debug debugfs rw,relatime 0 0</div></div></div><div class="gmail_quote">
<div><div>none /sys/kernel/security securityfs rw,relatime 0 0</div></div></div><div class="gmail_quote"><div><div>none /run tmpfs rw,nosuid,noexec,relatime,size=378636k,mode=755 0 0</div></div></div><div class="gmail_quote">
<div><div>none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0</div></div></div><div class="gmail_quote"><div><div>none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0</div></div></div><div class="gmail_quote">
<div><br></div></div><div class="gmail_quote"><div>root@localhost:/var/lib/lxc/CN/rootfs/etc/init# ls -l /proc/10018/root/dev/{console,tty*}</div></div><div class="gmail_quote"><div>lrwxrwxrwx 1 root root 11 Nov 9 23:55 /proc/10018/root/dev/console -> lxc/console</div>
</div><div class="gmail_quote"><div>crw-rw-rw- 1 root tty 5, 0 Oct 23 20:40 /proc/10018/root/dev/tty</div></div><div class="gmail_quote"><div>crw------- 1 root tty 4, 0 Oct 23 20:01 /proc/10018/root/dev/tty0</div></div>
<div class="gmail_quote"><div>lrwxrwxrwx 1 root root 8 Nov 9 23:55 /proc/10018/root/dev/tty1 -> lxc/tty1</div></div><div class="gmail_quote"><div>lrwxrwxrwx 1 root root 8 Nov 9 23:55 /proc/10018/root/dev/tty2 -> lxc/tty2</div>
</div><div class="gmail_quote"><div>lrwxrwxrwx 1 root root 8 Nov 9 23:55 /proc/10018/root/dev/tty3 -> lxc/tty3</div></div><div class="gmail_quote"><div>lrwxrwxrwx 1 root root 8 Nov 9 23:55 /proc/10018/root/dev/tty4 -> lxc/tty4</div>
</div><div class="gmail_quote"><div>crw------- 1 root tty 4, 5 Oct 23 20:01 /proc/10018/root/dev/tty5</div></div><div class="gmail_quote"><div>crw------- 1 root tty 4, 6 Oct 23 20:01 /proc/10018/root/dev/tty6</div></div>
<div class="gmail_quote"><div>crw------- 1 root tty 4, 7 Oct 23 20:01 /proc/10018/root/dev/tty7</div></div><div class="gmail_quote"><div>crw------- 1 root tty 4, 8 Oct 23 20:01 /proc/10018/root/dev/tty8</div></div><div class="gmail_quote">
<div>crw------- 1 root tty 4, 9 Oct 23 20:01 /proc/10018/root/dev/tty9</div></div><br><div><br></div><div><div>console *NOT* working case (started by libvirt)</div></div><div><div>--</div></div><div><div>[root@localhost instances]# cat /proc/2086/root/proc/1/mounts</div>
</div><div><div>rootfs / rootfs rw 0 0</div></div><div><div>devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0</div></div><div><div>/dev/nbd14 / ext3 rw,relatime,errors=continue,user_xattr,acl,barrier=1,data=ordered 0 0</div>
</div><div><div>devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0</div></div><div><div>devfs /dev tmpfs rw,nosuid,relatime,mode=755 0 0</div></div><div><div><br></div></div><div><div><-- * /dev is mounted here *</div>
</div><div><div><br></div></div><div><div>proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0</div></div><div><div>proc /proc/sys proc ro,relatime 0 0</div></div><div><div>sysfs /sys sysfs ro,relatime 0 0</div></div><div>
<div>/dev/sda2 /selinux ext4 ro,relatime,barrier=1,data=ordered 0 0</div></div><div><div>devpts /dev/ptmx devpts rw,relatime,gid=5,mode=620,ptmxmode=666 0 0</div></div><div><div><br></div></div><div><div>[root@localhost instances]# ls -l /proc/2086/root/dev/{console,tty*}</div>
</div><div><div>lrwxrwxrwx 1 root root 10 Nov 9 23:53 /proc/2086/root/dev/console -> /dev/pts/0</div></div><div><div>lrwxrwxrwx 1 root root 10 Nov 9 23:53 /proc/2086/root/dev/tty1 -> /dev/pts/0</div></div></blockquote>
<div><br></div><div>--</div><div>Qiu Yu</div>