[lxc-devel] Last minute template addition - universal image based template
Dwight Engen
dwight.engen at oracle.com
Tue Jan 14 01:04:56 UTC 2014
On Fri, 10 Jan 2014 15:10:14 -0500
Stéphane Graber <stgraber at ubuntu.com> wrote:
> Hey everyone,
>
> First of all, sorry for coming up with that so late in the 1.0
> development cycle. I tried to convince myself for a long time that
> this wasn't necessary but reality is that with unprivileged
> containers, we need to start thinking about new ways to let our users
> create containers.
Interesting idea, and I can see how it might be helpful. I converted
the Oracle template to common.conf style and tried this out a bit. Some
comments inline along with some questions.
> So briefly put, the idea is to introduce a new template called simply
> enough "download" (I'm open to better name suggestions) which instead
> of doing our usual magic of building a rootfs locally will instead
> download a pre-built rootfs and some metadata from a central server.
>
> My goal is to make the dependencies of that template minimal enough
> that it should be able to run on every distribution that ships LXC,
> ideally, including Android.
>
>
> Now as for the implementation, the plan is to bring up a new server on
> the linuxcontainers.org domain which will sit at:
> https://images.linuxcontainers.org
>
> The structure of that server will be something like
> ├── images
> │ ├── debian
> │ │ └── wheezy
> │ │ └── armhf
> │ │ └── default
> │ │ └── 2014-01-10_19:35
> │ │ ├── meta.tar.gz
> │ │ ├── meta.tar.gz.asc
> │ │ ├── rootfs.tar.gz
> │ │ └── rootfs.tar.gz.asc
> │ └── ubuntu
> │ └── 14.04
> │ └── amd64
> │ └── default
> │ └── 2014-01-10_19:30
> │ ├── meta.tar.gz
> │ ├── meta.tar.gz.asc
> │ ├── rootfs.tar.gz
> │ └── rootfs.tar.gz.asc
> └── meta
> └── 1.0
> ├── index-system
> ├── index-system.asc
> ├── index-user
> └── index-user.asc
>
> So the filesystem structure is fairly logical and user friendly. Now
> as for the details of the various files.
>
> rootfs.tar.gz is a tarball of only the root filesystem of the
> container with the tarball starting at what will be the root of the
> container.
>
> meta.tar.gz is a tarball containing a bit of metadata which the
> template script will need, that includes:
> - config (mandatory; text file containing a minimal set of config
> options to include in the container's config)
> - fstab (mandatory; a standard fstab text file)
> - expiry (mandatory; text file containing the Unix Epoch of the
> recommended expiry date for the image)
These two didn't actually seem to be mandatory, at least not yet :)
> - create-message (mandatory; text file which will be displayed to the
> user post-create)
> - *.<compat-level> (optional; any of the above may be overriden by
> appending .<compat-level> to their name which
> will be used instead of the original if the compat-level
> matches that of the currently running LXC).
>
> The index-* files are standard CSV files with the following syntax:
> - <distribution>;<version>;<architecture>;<variant>;<current
> build>;<url>
I grabbed the index-system that is up on images.linuxcontainers.org,
and in that file shouldn't the url have a "variant" "default" path
component in it to be consistent with the above tree? ie:
/images/ubuntu/trusty/amd64/2014-01-10_23:15 ->
/images/ubuntu/trusty/amd64/default/2014-01-10_23:15
> index-system is for system containers, index-user is for unprivileged
> containers. index-{system|user}.<compat-level> is also supported and
> overrides the main index if found.
>
> All of those files are signed by the server's GPG key which will be
> stored in-line in the script.
Patch included at the bottom to get the script to not even attempt to
download the signatures when --no-validate is passed since on my test
http server I didn't bother with any of the signature files.
> The template itself will require the following arguments:
> - distribution
> - version
> - architecture
>
> And the following will be optional:
> - variant [default: default]
> - server [default: https://images.linuxcontainers.org]
> - pubkey [default: inline gpg key]
> - no-validate [default: off]
>
> The goal for the template is to only depend on:
> - POSIX shell
> - wget
> - tar
> - gzip
It also uses recursive chown, which busybox does seem to have but
thought I'd note it.
> Which should be satisfiable even by busybox.
>
> However, the template will require --no-validate if gpg isn't also
> installed on the system.
>
> The initial set of distros to be supported that way will be any distro
> that can currently be built from an Ubuntu system (as that's what
> linuxcontainers.org uses) and that uses the new common-config
> (currently only ubuntu and ubuntu-cloud) setup (as I really don't
> want massive config files that change all the time to be part of
> those images).
Just trying to make sure I understand here: The plan is to
automatically build a rootfs for any distro that lxc can currently
build a rootfs via lxc-create on ubuntu host? Thats probably ~40 or so
just for Oracle Linux if we did all the versions (I have an sh+expect
test scripts that I use to make sure they all work), but maybe we'd
only want the latest of each (ie. 4, 5, 6) which cuts it way down to
only 3.
One question I have on this is that the rootfs (at least that the
Oracle template makes, not sure about others) isn't quite right since
the host name will have already been injected into config files (for
example /etc/hosts), but lxc-download wants to template them itself.
Should we add a switch to the distro template so it knows when its
being called to create a rootfs for lxc-download to put LXC_NAME in
there instead the --name it was passed?
> The compat-level stuff mentioned above is there to allow us to change
> the way the server works without breaking backward compatibility. It
> also allows us to introduce new templates to the system only for
> releases of LXC which support them.
> The initial value will be "1" and I'm going to need a very very good
> reason for bumping it (which is why I want all those distros to use
> lxc.include for their config).
>
>
> With all that said, I'll start working on the implementation of this
> and hope to have the server working with ubuntu and debian images
> published over the next few days.
> If any other distro is interested (I very much hope they all will),
> start by switching to lxc.include for your config, using something
> similar to what ubuntu is doing, then get in touch with me.
I have not tried from a normal user account yet. Is it basically that
you expect the template to be run by a normal user, but be running in a
userns? Can you describe how mapped_uid is supposed to be set? Thanks!
-->8 8<--
From 91e4ac620496f896eacd96a44eed25183267d496 Mon Sep 17 00:00:00 2001
From: Dwight Engen <dwight.engen at oracle.com>
Date: Mon, 13 Jan 2014 19:54:02 -0500
Subject: [PATCH] download template: don't download signatures when
--no-validate given
- show full path to failed download location
- change test to -f in case meta.tar.xz:templates has a blank line it
won't attempt to sed a directory
Signed-off-by: Dwight Engen <dwight.engen at oracle.com>
---
templates/lxc-download.in | 20 ++++++++++++++------
1 file changed, 14 insertions(+), 6 deletions(-)
diff --git a/templates/lxc-download.in b/templates/lxc-download.in
index cab6cbc..fa54686 100644
--- a/templates/lxc-download.in
+++ b/templates/lxc-download.in
@@ -60,7 +60,7 @@ download_file() {
if [ "$3" = "noexit" ]; then
return 1
else
- echo "ERROR: Failed to download $1" 1>&2
+ echo "ERROR: Failed to download http://${DOWNLOAD_SERVER}/$1" 1>&2
exit 1
fi
elif [ "$DOWNLOAD_SHOW_HTTP_WARNING" = "true" ]; then
@@ -72,6 +72,14 @@ download_file() {
fi
}
+download_sig() {
+ download_file $1 $2 noexit
+ if [ "$DOWNLOAD_VALIDATE" = "true" ]; then
+ echo "ERROR: Failed to download http://${DOWNLOAD_SERVER}/$1" 1>&2
+ exit 1
+ fi
+}
+
gpg_setup() {
if [ "$DOWNLOAD_VALIDATE" = "false" ]; then
return
@@ -263,8 +271,8 @@ if [ "$DOWNLOAD_USE_CACHE" = "false" ]; then
! download_file ${DOWNLOAD_INDEX_PATH}.${DOWNLOAD_COMPAT_LEVEL}.asc \
${DOWNLOAD_TEMP}/index.asc noexit; then
download_file ${DOWNLOAD_INDEX_PATH} ${DOWNLOAD_TEMP}/index normal
- download_file ${DOWNLOAD_INDEX_PATH}.asc \
- ${DOWNLOAD_TEMP}/index.asc normal
+ download_sig ${DOWNLOAD_INDEX_PATH}.asc \
+ ${DOWNLOAD_TEMP}/index.asc
fi
gpg_validate ${DOWNLOAD_TEMP}/index.asc
@@ -298,14 +306,14 @@ if [ "$DOWNLOAD_USE_CACHE" = "false" ]; then
echo "Downloading the rootfs"
download_file $DOWNLOAD_URL/rootfs.tar.xz \
${DOWNLOAD_TEMP}/rootfs.tar.xz normal
- download_file $DOWNLOAD_URL/rootfs.tar.xz.asc \
+ download_sig $DOWNLOAD_URL/rootfs.tar.xz.asc \
${DOWNLOAD_TEMP}/rootfs.tar.xz.asc normal
gpg_validate ${DOWNLOAD_TEMP}/rootfs.tar.xz.asc
echo "Downloading the metadata"
download_file $DOWNLOAD_URL/meta.tar.xz \
${DOWNLOAD_TEMP}/meta.tar.xz normal
- download_file $DOWNLOAD_URL/meta.tar.xz.asc \
+ download_sig $DOWNLOAD_URL/meta.tar.xz.asc \
${DOWNLOAD_TEMP}/meta.tar.xz.asc normal
gpg_validate ${DOWNLOAD_TEMP}/meta.tar.xz.asc
@@ -393,7 +401,7 @@ fi
# Replace variables in all templates
for file in $TEMPLATE_FILES; do
- [ ! -e "$file" ] && continue
+ [ ! -f "$file" ] && continue
sed -i "s#LXC_NAME#$LXC_NAME#g" $file
sed -i "s#LXC_PATH#$LXC_PATH#g" $file
--
1.8.3.1
More information about the lxc-devel
mailing list