[lxc-users] sshd-keygen fails during container boot

[Serge Hallyn]

Quoting Peter Steele (pwsteele at gmail.com):
> On 12/08/2015 08:00 AM, Serge Hallyn wrote:
> >Ok, can you change the launch command in the scripts to
> >
> >lxc-start -n $containername -L /tmp/$containername.cout -l trace -o /tmp/$containername.dout -- /sbin/init log_target=console log_level=debug
> >
> >The console output will go into the .cout file and lxc debug output into .dout.
> >
> I've actually made some progress in reproducing this outside of my
> framework. I originally thought the problem only occurred during the
> first boot of the containers. I've discovered that it can happen any
> time the server is rebooted and the containers are started when the
> server comes up.

What do you mean by "when the server comes up"?  If you bring up the
server, let it set for 5 mins, then start them, they still fail?

> I've only seen this problem when multiple
> containers are starting at the same time.

What lxc version are you using again?

> I incorporated your modified start command into a test as follows:
> 
> # for vm in `lxc-ls`; do lxc-start -n $vm -L /tmp/$vm.cout -l trace
> -o /tmp/$vm.dout -- /sbin/init log_target=console log_level=debug;
> done
> 
> This starts all of my previously created containers at roughly the
> same time, and when I do this some of the containers encounter the
> systemd errors I've been seeing. Which containers hit these errors
> vary from test to test. In looking at the .dout logs, I noticed the
> following:
> 
>       lxc-start 1449591253.647 DEBUG    lxc_conf -
> conf.c:setup_rootfs:1295 - mounted '/hf/cs/vm-00/rootfs' on
> '/usr/lib64/lxc/rootfs'
>       lxc-start 1449591253.647 INFO     lxc_conf -
> conf.c:setup_utsname:928 - 'pws-vm-00' hostname has been setup
>       lxc-start 1449591253.660 DEBUG    lxc_conf -
> conf.c:setup_hw_addr:2368 - mac address 'fe:d6:e8:96:7e:2d' on
> 'eth0' has been setup
>       lxc-start 1449591253.660 DEBUG    lxc_conf -
> conf.c:setup_netdev:2595 - 'eth0' has been setup
>       lxc-start 1449591253.660 INFO     lxc_conf -
> conf.c:setup_network:2616 - network has been setup
>       lxc-start 1449591253.660 INFO     lxc_conf -
> conf.c:mount_autodev:1157 - Mounting container /dev
>       lxc-start 1449591253.661 INFO     lxc_conf -
> conf.c:mount_autodev:1179 - Mounted tmpfs onto
> /usr/lib64/lxc/rootfs/dev
>       lxc-start 1449591253.661 INFO     lxc_conf -
> conf.c:mount_autodev:1197 - Mounted container /dev
>       lxc-start 1449591253.661 ERROR    lxc_utils -
> utils.c:open_without_symlink:1626 - No such file or directory -
> Error examining fuse in
> /usr/lib64/lxc/rootfs/sys/fs/fuse/connections

Ok, so this shows that in the container 'sys/fs' existed,
but fuse did not.  This suggests that the fuse kernel module
was not done loading yet.

Could you add to /lib/systemd/system/lxc.service the line

ExecStartPre=modprobe fuse

and see if that helps?  (I'm not sure if you'd also need to sleep
a short time to give syfs time to catch up, or if the modprobe
would wait...  you could just use a script that waits until
/sys/fs/fuse exists on the host)