[Lxc-users] [lxc-devel] [systemd-devel] Unable to run systemd in an LXC / cgroup container.
Michael H. Warfield
mhw at WittsEnd.com
Sun Oct 28 21:33:55 UTC 2012
On Sun, 2012-10-28 at 18:52 +0100, Serge Hallyn wrote:
> Quoting Michael H. Warfield (mhw at WittsEnd.com):
> > On Sat, 2012-10-27 at 13:51 -0400, Michael H. Warfield wrote:
> > > On Sat, 2012-10-27 at 13:40 -0400, Michael H. Warfield wrote:
> > > > /me erasing everything at this point and taking off the systemd crew,
> > > > since this will have no relevance to them...
> > > >
> > > > Testing the hook feature out using git rev (finally got it built)...
> > > >
> > > > I added this line to my config...
> > > >
> > > > lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
> > > > lxc.hook.mount = /var/lib/lxc/Plover/mount
> > > >
> > > > In /var/lib/lxc/Plover/mount I have this:
> > > >
> > > > --
> > > > rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
> > > > --
> > >
> > > > (This is just testing out the concepts.
> > >
> > > > If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
> > > > mounting takes place and lxc.hook.mount takes place after the mount.
> > >
> > > > Problem is, the result of that rsync is not showing up in the mounted
> > > > tmpfs file system but is showing up in the underlying parent file system
> > > > as if it were run pre-mount. Something not right here...
> >
> > > I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
> > > script in the container) which then works but that then requires the
> > > template and the command to be in the container. Suboptimal to say the
> > > least. But it gives me a way to test this tmpfs thing out.
> >
> > > I also noticed that the .start hook runs, it appears, after caps are
> > > dropped and I see a lot of bitching about mknod on certain devices. I
> > > had to thrown an exit 0 into that script so it would continue in spite
> > > of the errors but, now, I can refine my template based on what it could
> > > create.
> >
> > Crap. I've got a catch-22 here... This is going to take some work.
> Hey,
> I've got a rather minimal patch (appended below) to add the support for
> mounting and populating a minimal /dev working. (A few hours were wasted
> due to my not knowing that upstart was going to issue mounted-dev even though
> /dev was mounted before upstart started - and the mounted-dev hook deletes
> and recreates all consoles. GAH)
I am happy to report that, after manually editing my git head branch to
patch in the failed hunks, I was able to build it and test it and my
Fedora 17 systemd based container fired right up after adding the
lxc.autodev = 1 parameter to the config file. Yeah!!!!
I did run into one gotcha, but one I can live with. I had been bind
mounting the private root file system to another directory and then
using that as the rootfs like this:
--
lxc.rootfs = /srv/lxc/rootfs
lxc.mount.entry=/srv/lxc/private/Alcove /srv/lxc/rootfs none bind.shared 0 0
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
This did not work and I got the startup error that it can not mount
to /dev because it doesn't exist.
I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.
Drop that bind mount for the rootfs and this then worked like a charm:
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.
Regards,
Mike
> > Yes, we can create the /dev directory with tmpfs from a template.
> > Problem is that /dev/pts does not exist at the time we need to mount the
> > devpts on /dev/pts for the pty's so that hurls chunks and dies. We
> > can't create the /dev/ directory contents prior to mounting in the
> > pre-mount hook because we won't have tmpfs in place at the time. We
> > have to get tmpfs mounted on /dev and then create /dev/pts and then
> > mount the ptys in there. There has to be a mkdir in between those two
> > mount actions. Simplest solution would seem to be to add some logic to
> > the mount logic that says "test if directory exists and, if not, create
> > it." I'm not sure of the consequences of that, though.
> >
> > I don't see a way to make this happen with hooks. It's almost like we
> > need and on-mount per mount hook.
>
> Should be moot given my patch, which I intend to push this week, but why
> couldn't a lxc.hook.mount do the whole thing, mount /dev and and populate
> it? I wasn't thinking a lxc.hook.start, for the reasons you encountered,
> but I assume you tried lxc.hook.mount and it failed?
>
> Patch below:
>
> Index: lxc-qp/src/lxc/conf.c
> ===================================================================
> --- lxc-qp.orig/src/lxc/conf.c 2012-10-27 17:24:50.768383000 -0500
> +++ lxc-qp/src/lxc/conf.c 2012-10-28 05:44:07.871228322 -0500
> @@ -619,7 +619,7 @@
> }
>
> if (mount(pty_info->name, lxcpath, "none", MS_BIND, 0)) {
> - WARN("failed to mount '%s'->'%s'",
> + SYSERROR("failed to mount '%s'->'%s'",
> pty_info->name, path);
> continue;
> }
> @@ -636,7 +636,7 @@
> }
> } else {
> if (mount(pty_info->name, path, "none", MS_BIND, 0)) {
> - WARN("failed to mount '%s'->'%s'",
> + SYSERROR("failed to mount '%s'->'%s'",
> pty_info->name, path);
> continue;
> }
> @@ -842,6 +842,67 @@
> return 0;
> }
>
> +struct lxc_devs {
> + char *name;
> + mode_t mode;
> + int maj;
> + int min;
> +};
> +
> +struct lxc_devs lxc_devs[] = {
> + { "null", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 3 },
> + { "zero", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 5 },
> + { "full", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 7 },
> + { "urandom", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 9 },
> + { "random", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 8 },
> + { "tty", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 5, 0 },
> + { "console", S_IFCHR | S_IRUSR | S_IWUSR, 5, 1 },
> +};
> +
> +/*
> + * Do we want to add options for max size of /dev and a file to
> + * specify which devices to create?
> + */
> +static int setup_autodev(char *root)
> +{
> + int ret;
> + struct lxc_devs *d;
> + char path[MAXPATHLEN];
> + int i;
> +
> + INFO("Creating and populating /dev under %s\n", root);
> + ret = snprintf(path, MAXPATHLEN, "%s/dev", root);
> + if (ret < 0 || ret > MAXPATHLEN)
> + return -1;
> + ret = mount("none", path, "tmpfs", 0, "size=100000");
> + if (ret) {
> + SYSERROR("Failed to mount /dev at %s\n", root);
> + return -1;
> + }
> + for (i = 0; i < sizeof(lxc_devs) / sizeof(lxc_devs[0]); i++) {
> + d = &lxc_devs[i];
> + ret = snprintf(path, MAXPATHLEN, "%s/dev/%s", root, d->name);
> + if (ret < 0 || ret >= MAXPATHLEN)
> + return -1;
> + ret = mknod(path, d->mode, makedev(d->maj, d->min));
> + if (ret) {
> + SYSERROR("Error creating %s\n", d->name);
> + return -1;
> + }
> + }
> + ret = snprintf(path, MAXPATHLEN, "%s/dev/pts", root);
> + if (ret < 0 || ret >= MAXPATHLEN)
> + return -1;
> + ret = mkdir(path, S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH);
> + if (ret) {
> + SYSERROR("Failed to create /dev/pts in container");
> + return -1;
> + }
> +
> + INFO("Populated /dev under %s\n", root);
> + return 0;
> +}
> +
> static int setup_rootfs(const struct lxc_rootfs *rootfs)
> {
> if (!rootfs->path)
> @@ -2208,6 +2269,13 @@
> return -1;
> }
>
> + if (lxc_conf->autodev) {
> + if (setup_autodev(lxc_conf->rootfs.mount)) {
> + ERROR("failed to set up /dev in the container");
> + return -1;
> + }
> + }
> +
> if (setup_mount(&lxc_conf->rootfs, lxc_conf->fstab, name)) {
> ERROR("failed to setup the mounts for '%s'", name);
> return -1;
> Index: lxc-qp/src/lxc/conf.h
> ===================================================================
> --- lxc-qp.orig/src/lxc/conf.h 2012-10-27 17:24:50.768383000 -0500
> +++ lxc-qp/src/lxc/conf.h 2012-10-27 17:24:50.768383000 -0500
> @@ -227,6 +227,7 @@
> struct lxc_list hooks[NUM_LXC_HOOKS];
> char *seccomp; // filename with the seccomp rules
> int maincmd_fd;
> + int autodev; // if 1, mount and fill a /dev at start
> };
>
> int run_lxc_hooks(const char *name, char *hook, struct lxc_conf *conf);
> Index: lxc-qp/src/lxc/confile.c
> ===================================================================
> --- lxc-qp.orig/src/lxc/confile.c 2012-10-27 17:24:50.768383000 -0500
> +++ lxc-qp/src/lxc/confile.c 2012-10-27 17:24:50.768383000 -0500
> @@ -77,6 +77,7 @@
> static int config_seccomp(const char *, char *, struct lxc_conf *);
> static int config_includefile(const char *, char *, struct lxc_conf *);
> static int config_network_nic(const char *, char *, struct lxc_conf *);
> +static int config_autodev(const char *, char *, struct lxc_conf *);
>
> typedef int (*config_cb)(const char *, char *, struct lxc_conf *);
>
> @@ -118,6 +119,7 @@
> { "lxc.console", config_console },
> { "lxc.seccomp", config_seccomp },
> { "lxc.include", config_includefile },
> + { "lxc.autodev", config_autodev },
> };
>
> static const size_t config_size = sizeof(config)/sizeof(struct lxc_config_t);
> @@ -853,6 +855,16 @@
>
> return 0;
> }
> +
> +static int config_autodev(const char *key, char *value,
> + struct lxc_conf *lxc_conf)
> +{
> + int v = atoi(value);
> +
> + lxc_conf->autodev = v;
> +
> + return 0;
> +}
>
> static int config_aa_profile(const char *key, char *value,
> struct lxc_conf *lxc_conf)
>
> ------------------------------------------------------------------------------
> WINDOWS 8 is here.
> Millions of people. Your app in 30 days.
> Visit The Windows 8 Center at Sourceforge for all your go to resources.
> http://windows8center.sourceforge.net/
> join-generation-app-and-make-money-coding-fast/
> _______________________________________________
> Lxc-devel mailing list
> Lxc-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/lxc-devel
>
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121028/0e803b9a/attachment.pgp>
More information about the lxc-users
mailing list