[lxc-devel] Last minute template addition - universal image based template

S.Çağlar Onur caglar at 10ur.org
Wed Jan 15 03:02:39 UTC 2014 

On Tue, Jan 14, 2014 at 9:50 PM, Stéphane Graber <stgraber at ubuntu.com> wrote:
> On Tue, Jan 14, 2014 at 09:48:13PM -0500, Stéphane Graber wrote:
>> On Tue, Jan 14, 2014 at 09:30:50PM -0500, S.Çağlar Onur wrote:
>> > On Tue, Jan 14, 2014 at 9:24 PM, Stéphane Graber <stgraber at ubuntu.com> wrote:
>> > > On Tue, Jan 14, 2014 at 09:12:31PM -0500, S.Çağlar Onur wrote:
>> > >> Hey Stéphane,
>> > >>
>> > >> On Tue, Jan 14, 2014 at 6:52 PM, Stéphane Graber <stgraber at ubuntu.com> wrote:
>> > >> > On Tue, Jan 14, 2014 at 06:38:43PM -0500, Dwight Engen wrote:
>> > >> >> On Mon, 13 Jan 2014 20:56:23 -0500
>> > >> >> Stéphane Graber <stgraber at ubuntu.com> wrote:
>> > >> >>
>> > >> >> [...]
>> > >> >> >
>> > >> >> > Yeah, I did a few last-minute changes.
>> > >> >> >
>> > >> >> > fstab is no longer required (only copied over if it's there).
>> > >> >> >
>> > >> >> > If the expiry file is messing, it's simply assumed that the container
>> > >> >> > won't expire.
>> > >> >>
>> > >> >> Nice, that makes sense.
>> > >> >>
>> > >> >> [...]
>> > >> >> > Now for Oracle, I guess it depends what's most useful for your users.
>> > >> >> >
>> > >> >> > Is there actually still a demand for 4.x? I thought at least RHEL 4.x
>> > >> >> > went out of support last year.
>> > >> >> >
>> > >> >> > In any case, I suspect that starting with the latest of each supported
>> > >> >> > release on i386 and amd64 is probably a good start.
>> > >> >> > We can then expand that to include popular minor releases, possibly
>> > >> >> > the last few 6.x?
>> > >> >>
>> > >> >> Yeah I think it makes sense to start with just x86_64 6.latest, and
>> > >> >> we'll see how it goes from there.
>> > >> >>
>> > >> >> > > One question I have on this is that the rootfs (at least that the
>> > >> >> > > Oracle template makes, not sure about others) isn't quite right
>> > >> >> > > since the host name will have already been injected into config
>> > >> >> > > files (for example /etc/hosts), but lxc-download wants to template
>> > >> >> > > them itself. Should we add a switch to the distro template so it
>> > >> >> > > knows when its being called to create a rootfs for lxc-download to
>> > >> >> > > put LXC_NAME in there instead the --name it was passed?
>> > >> >> >
>> > >> >> > So currently my build script (Jenkins job) will call the template
>> > >> >> > with:
>> > >> >> >  - --rootfs=/build-lxc/container/LXC_NAME/rootfs
>> > >> >> >  - --path=/build-lxc/container/LXC_NAME
>> > >> >> >  - --name=LXC_NAME
>> > >> >>
>> > >> >> Oh yeah, doh! Clever, that should make it work fine.
>> > >> >>
>> > >> >> [...]
>> > >> >> > > I have not tried from a normal user account yet. Is it basically
>> > >> >> > > that you expect the template to be run by a normal user, but be
>> > >> >> > > running in a userns? Can you describe how mapped_uid is supposed to
>> > >> >> > > be set? Thanks!
>> > >> >> > -d oracle -r 6.5 -a x86_64 --server phocis/lxc-images --no-validate
>> > >> >> > Yeah, lxc-create basically does all the magic we need there.
>> > >> >> > When run as a user, it'll run the template in a userns mapped with the
>> > >> >> > container's range.
>> > >> >> >
>> > >> >> > $mapped_uid appears to be set to the uid in the userns which maps back
>> > >> >> > to the user's own uid outside of it and makes it possible to chown
>> > >> >> > files back to the actual user.
>> > >> >>
>> > >> >> So I had to dig into the source to see how I could get this to work,
>> > >> >> and tried passing a line like:
>> > >> >>
>> > >> >> lxc-create -P /home/dengen/ct -f /home/dengen/ct/dengen_id_map.conf -n ol65 -t download -- -d oracle -r 6.5 -a x86_64 --server lxchttp/lxc-images --no-validate
>> > >> >>
>> > >> >> where /home/dengen/ct/dengen_id_map.conf just has:
>> > >> >>  lxc.id_map = u 0 100000 10000
>> > >> >>  lxc.id_map = g 0 100000 10000
>> > >> >>
>> > >> >> in order to get mapped_uid in lxc-create to kick in. Sadly this didn't
>> > >> >> work because lxc.spec is still installing lxc-create (and a lot more)
>> > >> >> setuid, so the geteuid() check in create_run_template() isn't firing.
>> > >> >> So my question is: which lxc binaries are supposed to be installed
>> > >> >> setuid these days? Is it just lxc-user-nic? Once I know, I can go ahead
>> > >> >> and fixup lxc.spec. Thanks.
>> > >> >
>> > >> > Oh, wow, I didn't realize some distros were still installing anything
>> > >> > setuid these days, especially after we drop lxc-setuid and lxc-setcap
>> > >> > from the upstream tree.
>> > >> >
>> > >> > So anyway, nowadays, the recommendation is for only lxc-user-nic to be
>> > >> > setuid, everything else should always run as the user. When run as root,
>> > >> > you'll end up with system containers, when run as a user, you'll get an
>> > >> > unprivileged container.
>> > >>
>> > >> Should I expect [1] to work on saucy with trusty kernel? Here is what
>> > >> I'm getting
>> > >>
>> > >> [caglar at oOo:~] ./unprivileged.sh
>> > >> Creating unpriv with dir
>> > >> Using image from local cache
>> > >> Unpacking the rootfs
>> > >> chown: invalid user: ‘-f’
>> > >
>> > > What shell is providing /bin/sh on your system?
>> > >
>> > > I confirmed that chown had a -f argument on both busybox, dash and bash,
>> > > but apparently you're using something else.
>> > >
>> > > Anyway, I'll fix that particular failure by using a good old || true then.
>> >
>> > I'm using bash but I think problem is not the -f parameter but wrong
>> > LXC_MAPPED_UID value.
>>
>> Well, turns out -f isn't in POSIX so I sent a patch anyway.
>>
>> >
>> > Printing the LXC_MAPPED_UID via
>> >
>> > if [ -n "$LXC_MAPPED_UID" ] && [ "$LXC_MAPPED_UID" != "-1" ]; then
>> >     echo "***" $LXC_MAPPED_UID "***"
>> >     chown $LXC_MAPPED_UID -f $LXC_PATH/config $LXC_PATH/fstab || true
>> > fi
>>
>> Now that's pretty odd... I just did a quick test here and I don't see
>> where what you're seeing is coming from...
>>
>> stgraber at castiana:~/data/code/lxc/lxc$ lxc-create -t download -n p1 -- -d ubuntu -r trusty -a amd64
>> called with: --path=/home/stgraber/.local/share/lxc/p1 --name=p1 --rootfs=/home/stgraber/.local/share/lxc/p1/rootfs -d ubuntu -r trusty -a amd64 --mapped-uid 65536
>> Setting up the GPG keyring
>> Downloading the image index
>> Downloading the rootfs
>> Downloading the metadata
>> The image cache is now ready
>> Unpacking the rootfs
>>
>> ---
>> You just created an Ubuntu container (release=trusty, arch=amd64).
>> The default username/password is: ubuntu / ubuntu
>> To gain root privileges, please use sudo.
>>
>>
>> stgraber at castiana:~/data/code/lxc/lxc$ sudo lxc-create -t download -n p1 -- -d ubuntu -r trusty -a amd64
>> called with: --path=/var/lib/lxc/p1 --name=p1 --rootfs=/var/lib/lxc/p1/rootfs -d ubuntu -r trusty -a amd64
>> Using image from local cache
>> Unpacking the rootfs
>>
>> ---
>> You just created an Ubuntu 14.04 LTS (amd64) container.
>> The default username/password is: ubuntu / ubuntu
>> To gain root privileges, please use sudo.
>>
>>
>>
>> Can you also patch your lxc-download to print "$*" and post the result here?
>
> Nevermind that, I think I found the problem, patch sent.

Yep "[lxc-devel] [PATCH] lxc-download: Fix wrong option parsing"
solved the issue :) I added

    echo 1 | sudo tee -a /sys/fs/cgroup/memory/memory.use_hierarchy > /dev/null
    echo 1 | sudo tee -a /sys/fs/cgroup/cpuset/cgroup.clone_children > /dev/null

    for controller in /sys/fs/cgroup/*; do
        sudo mkdir -p $controller/$USER
        sudo chown -R $USER $controller/$USER
        echo $$ > $controller/$USER/tasks
    done

to the script and containers starting/stopping (I also updated gist).
lxc-attach is now giving me trouble like this

[caglar at oOo:~] lxc-attach -n unpriv -P /home/caglar/lxcpath
lxc-attach: Operation not permitted - failed to set namespace 'mnt'
lxc-attach: failed to enter the namespace

Do you have any solution to that :)?

>>
>> > shows this
>> >
>> > [caglar at oOo:~] ./unprivileged.sh
>> > Creating unpriv with dir
>> > Using image from local cache
>> > Unpacking the rootfs
>> > *** -- ***
>> > chown: invalid user: ‘-f’
>> >
>> > ---
>> > You just created an Ubuntu container (release=saucy, arch=amd64).
>> > The default username/password is: ubuntu / ubuntu
>> > To gain root privileges, please use sudo.
>> >
>> >
>> >
>> > >>
>> > >> ---
>> > >> You just created an Ubuntu container (release=saucy, arch=amd64).
>> > >> The default username/password is: ubuntu / ubuntu
>> > >> To gain root privileges, please use sudo.
>> > >> Starting/Attaching/Stopping to unpriv
>> > >> lxc-start: command get_cgroup failed to receive response
>> > >> lxc-attach: failed to get the init pid
>> > >> unpriv is not running
>> > >> Destroying unpriv
>> > >
>> > > The rest of the failure is coming from you not having the proper
>> > > cgroupfs setup. Try something like:
>> > >
>> > > for controller in /sys/fs/cgroup; do
>> > >     sudo mkdir -p $controller/$USER/lxc
>> > >     sudo chown -R $USER $controller/$USER
>> > >     echo $$ > $controller/$USER/lxc/tasks
>> > > done
>> > >
>> > > This assumes that the top level clone_children and use_hierarchy are set
>> > > to 1 prior to doing this.
>> > >
>> > > It's unfortunately a bit painful to setup by hand everytime but
>> > > cgmanager should fix that once it lands.
>> >
>> > Ah right, will try with this.
>> >
>> > >>
>> > >> [caglar at oOo:~] grep ERROR unpriv.log
>> > >>       lxc-start 1389751901.034 ERROR    lxc_cgroup - Permission denied
>> > >> - Could not create cgroup /unpriv
>> > >>       lxc-start 1389751901.034 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/hugetlb//lxc
>> > >>       lxc-start 1389751901.035 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/hugetlb/
>> > >>       lxc-start 1389751901.035 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/perf_event//lxc
>> > >>       lxc-start 1389751901.035 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/perf_event/
>> > >>       lxc-start 1389751901.035 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/blkio//lxc
>> > >>       lxc-start 1389751901.036 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/blkio/
>> > >>       lxc-start 1389751901.036 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/freezer//lxc
>> > >>       lxc-start 1389751901.036 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/freezer/
>> > >>       lxc-start 1389751901.036 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/devices//lxc
>> > >>       lxc-start 1389751901.036 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/devices/
>> > >>       lxc-start 1389751901.036 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/memory//lxc
>> > >>       lxc-start 1389751901.037 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/memory/
>> > >>       lxc-start 1389751901.037 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuacct//lxc
>> > >>       lxc-start 1389751901.037 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuacct/
>> > >>       lxc-start 1389751901.037 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpu//lxc
>> > >>       lxc-start 1389751901.037 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpu/
>> > >>       lxc-start 1389751901.038 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuset//lxc
>> > >>       lxc-start 1389751901.038 ERROR    lxc_cgroup - Permission denied
>> > >> - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuset/
>> > >>       lxc-start 1389751901.038 ERROR    lxc_start - failed to create
>> > >> cgroups for 'unpriv'
>> > >>       lxc-start 1389751901.038 ERROR    lxc_start - failed to spawn 'unpriv'
>> > >>       lxc-start 1389751901.039 ERROR    lxc_commands - command
>> > >> get_cgroup failed to receive response
>> > >>
>> > >> [1] https://gist.github.com/caglar10ur/8429502
>> > >>
>> > >> > --
>> > >> > Stéphane Graber
>> > >> > Ubuntu developer
>> > >> > http://www.ubuntu.com
>> > >> >
>> > >> > _______________________________________________
>> > >> > lxc-devel mailing list
>> > >> > lxc-devel at lists.linuxcontainers.org
>> > >> > http://lists.linuxcontainers.org/listinfo/lxc-devel
>> > >> >
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> S.Çağlar Onur <caglar at 10ur.org>
>> > >> _______________________________________________
>> > >> lxc-devel mailing list
>> > >> lxc-devel at lists.linuxcontainers.org
>> > >> http://lists.linuxcontainers.org/listinfo/lxc-devel
>> > >
>> > > --
>> > > Stéphane Graber
>> > > Ubuntu developer
>> > > http://www.ubuntu.com
>> > >
>> > > _______________________________________________
>> > > lxc-devel mailing list
>> > > lxc-devel at lists.linuxcontainers.org
>> > > http://lists.linuxcontainers.org/listinfo/lxc-devel
>> > >
>> >
>> >
>> >
>> > --
>> > S.Çağlar Onur <caglar at 10ur.org>
>> > _______________________________________________
>> > lxc-devel mailing list
>> > lxc-devel at lists.linuxcontainers.org
>> > http://lists.linuxcontainers.org/listinfo/lxc-devel
>>
>> --
>> Stéphane Graber
>> Ubuntu developer
>> http://www.ubuntu.com
>
>
>
>> _______________________________________________
>> lxc-devel mailing list
>> lxc-devel at lists.linuxcontainers.org
>> http://lists.linuxcontainers.org/listinfo/lxc-devel
>
>
> --
> Stéphane Graber
> Ubuntu developer
> http://www.ubuntu.com
>
> _______________________________________________
> lxc-devel mailing list
> lxc-devel at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-devel
>



-- 
S.Çağlar Onur <caglar at 10ur.org>

More information about the lxc-devel mailing list