[lxc-devel] lxc-start hangs when starting a new container

[Daniel Lezcano]

Matty wrote:
> On Wed, Jul 1, 2009 at 10:30 AM, Daniel Lezcano<dlezcano at fr.ibm.com> wrote:
>   
>> Matty wrote:
>>     
>>> On Wed, Jul 1, 2009 at 8:59 AM, Daniel Lezcano<dlezcano at fr.ibm.com> wrote:
>>>       
>>>> Matty wrote:
>>>>         
>>>>> On Wed, Jul 1, 2009 at 4:12 AM, Daniel Lezcano<dlezcano at fr.ibm.com>
>>>>> wrote:
>>>>>           
>>>>>> Matty wrote:
>>>>>>             
>>>>>>> I have an lxc container that keeps hanging when I run 'lxc-start -n
>>>>>>> <GUEST NAME>'. When I strace lxc-start, I see the following:
>>>>>>>
>>>>>>> $ strace lxc-start -n test
>>>>>>>  ..............
>>>>>>> open("/usr/local/var/lib/lxc/test/state", O_WRONLY) = 13
>>>>>>> flock(13, LOCK_EX)                      = 0
>>>>>>> ftruncate(13, 0)                        = 0
>>>>>>> write(13, "RUNNING"..., 7)              = 7
>>>>>>> close(13)                               = 0
>>>>>>> socket(PF_FILE, SOCK_DGRAM, 0)          = 13
>>>>>>> sendto(13,
>>>>>>> "\0\0\0\0test\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
>>>>>>> 4104, 0, {sa_family=AF_FILE, path=@"lxc-monitor"...}, 110) = -1
>>>>>>> ECONNREFUSED (Connection refused)
>>>>>>> close(13)                               = 0
>>>>>>> close(13)                               = -1 EBADF (Bad file
>>>>>>> descriptor)
>>>>>>> close(14)                               = 0
>>>>>>> socket(PF_FILE, SOCK_STREAM, 0)         = 13
>>>>>>> bind(13, {sa_family=AF_FILE, path=@"test"...}, 110) = 0
>>>>>>> listen(13, 100)                         = 0
>>>>>>> fcntl(13, F_SETFD, FD_CLOEXEC)          = 0
>>>>>>> epoll_create(1)                         = 14
>>>>>>> epoll_ctl(14, EPOLL_CTL_ADD, 12, {EPOLLIN, {u32=17707440,
>>>>>>> u64=17707440}})
>>>>>>> = 0
>>>>>>> epoll_ctl(14, EPOLL_CTL_ADD, 13, {EPOLLIN, {u32=17707504,
>>>>>>> u64=17707504}})
>>>>>>> = 0
>>>>>>>
>>>>>>> I starting the container with 2.6.[29-31pre] kernels and with the
>>>>>>> latest lxc code, and they all appear to exhibit this behavior. Has
>>>>>>> anyone seen this before? Any idea what the lxc-monitor socket is used
>>>>>>> for?
>>>>>>>               
>>>>>> can you give more information about the container itself ?
>>>>>>             
>>>>> Sure thing. This is a Fedora 11 container I created with lxc-fedora.
>>>>> Are there specific details you're after? If so, I will be glad to send
>>>>> them to you.
>>>>>
>>>>>           
>>>>>> What is the configuration ?
>>>>>>             
>>>>> I used the defaults provided by lxc-fedora, though I disabled udev
>>>>> (commented out the udev line in rc.sysinit) to allow the container to
>>>>> boot.
>>>>>           
>>>> Did you added the tty to the rootfs ?
>>>>
>>>> chroot rootfs.test1 /bin/bash
>>>> cd /dev
>>>> MAKEDEV tty
>>>>         
>>> Hey Daniel,
>>>
>>> I am bind mounting /dev into the container, so I would expect all of
>>> the devices to be there. In addition, I adjusted the cgroup
>>> permissions to allow me to access the most common devices:
>>>
>>> devices.deny=a
>>> devices.allow=c 1:3 rwm
>>> devices.allow=c 1:5 rwm
>>> devices.allow=c 1:7 rwm
>>> devices.allow=c 1:8 rwm
>>> devices.allow=c 1:9 rwm
>>> devices.allow=c 5:0 rwm
>>> devices.allow=c 5:1 rwm
>>> devices.allow=c 5:2 rwm
>>> devices.allow=c 4:* rwm
>>> devices.allow=c 254:0 rwm
>>>
>>> Shouldn't this work? Here is the fstab file I am using:
>>>
>>> $  pwd
>>> /usr/local/var/lib/lxc/test
>>>
>>> $ cat fstab
>>> /var/lxc/rootfs/test.rootfs /usr/local/var/lib/lxc/test/rootfs none rbind
>>> 0 0
>>> /dev /var/lxc/rootfs/test.rootfs/dev none bind 0 0
>>>
>>>       
>>>> After doing that, you can start the container again and check if you are
>>>> able to log to the container via lxc-console -n test1 command.
>>>>         
>>> It still hangs in the same location. :(
>>>       
>> I succeeded with the following:
>>
>> devices.deny=a
>> devices.allow=c 1:* rwm
>> devices.allow=b 1:* rwm
>> devices.allow=b 3:* rwm
>> devices.allow=c 5:0 rwm
>> devices.allow=c 5:1 rwm
>> devices.allow=c 5:2 rwm
>> devices.allow=c 4:* rwm
>> devices.allow=c 254:0 rwm
>> devices.allow=b 254:* rwm
>> devices.allow=c 136:* rwm
>> devices.allow=c 10:62 rwm
>>
>>
>> Be careful, sharing /dev for a system container may lead to an unexpected
>> behaviour :)
>>     
>
> Hmmmm. Several documents describe using a bind mounted /dev, so I
> assumed that protections were in place for this? If that is not the
> case, would you recommend creating the device nodes inside each rootfs
> and avoiding the bind mount?
>   
Not especially, I don't have any example of system vs container 
conflict, I guess bind mounted /dev and using the cgroup dev security 
would be safe but I didn't looked at this aspect so I prefer to warn of 
a potential problem. You might discard my comment :)