<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px"><div dir="ltr"><span>Thanks much.. This helps</span></div><div dir="ltr"><span><br></span></div><div dir="ltr"><span>Regards</span></div><div dir="ltr"><span>Mohan</span></div><br> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id="yui_3_16_0_1_1425530844351_96841"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id="yui_3_16_0_1_1425530844351_96840"> <div dir="ltr" id="yui_3_16_0_1_1425530844351_96839"> <hr size="1"> <font size="2" face="Arial" id="yui_3_16_0_1_1425530844351_96838"> <b><span style="font-weight:bold;">From:</span></b> Fajar A. Nugraha <list@fajar.net><br> <b><span style="font-weight: bold;">To:</span></b> LXC users mailing-list <lxc-users@lists.linuxcontainers.org> <br> <b><span style="font-weight: bold;">Sent:</span></b> Friday, March 6, 2015 12:30 PM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [lxc-users] nested containers<br> </font> </div> <div class="y_msg_container" id="yui_3_16_0_1_1425530844351_96843"><br>This is on ubuntu 14.10, lxc and lxcfs from ppa:ubuntu-lxc/daily<br clear="none"><br clear="none">Test memory limit on the parent container, by writing data to /run/shm<br clear="none">(a tmpfs mount):<br clear="none"><br clear="none">@host # lxc-cgroup -n v memory.use_hierarchy<br clear="none">1<br clear="none"><br clear="none">@host # lxc-cgroup -n v memory.limit_in_bytes 1G<br clear="none"><br clear="none">@host # lxc-cgroup -n v memory.limit_in_bytes<br clear="none">1073741824<br clear="none"><br clear="none">@host # lxc-attach -n v -- mount | grep shm<br clear="none">none on /run/shm type tmpfs (rw,nosuid,nodev,relatime)<br clear="none"><br clear="none">@host # lxc-cgroup -n v memory.usage_in_bytes<br clear="none">12881920<br clear="none"><br clear="none">@host # lxc-attach -n v -- dd if=/dev/zero of=/run/shm/testfile<br clear="none">dd: writing to ‘/run/shm/testfile’: Cannot allocate memory<br clear="none">2076473+0 records in<br clear="none">2076472+0 records out<br clear="none">1063153664 bytes (1.1 GB) copied, 2.18223 s, 487 MB/s<br clear="none"><br clear="none">@host # lxc-cgroup -n v memory.usage_in_bytes<br clear="none">1073561600<br clear="none"><br clear="none"><br clear="none">See how it's limited to around 1GB? Now delete the file before running<br clear="none">more test to free up memory.<br clear="none"><br clear="none"><br clear="none">@host # lxc-attach -n v -- rm /run/shm/testfile<br clear="none"><br clear="none">@host # lxc-cgroup -n v memory.usage_in_bytes<br clear="none">10219520<br clear="none"><br clear="none"><br clear="none"><br clear="none">Now test the limit on the nested container. We test with memory limit<br clear="none">2G. It should max out at 1G, since the parent only has that much<br clear="none">limit.<br clear="none"><br clear="none">@host # lxc-attach -n v<br clear="none"><br clear="none"><a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-cgroup -n nv memory.limit_in_bytes 2G<br clear="none"><br clear="none"><a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-cgroup -n nv memory.limit_in_bytes<br clear="none">2147483648<br clear="none"><br clear="none"><a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-cgroup -n nv memory.usage_in_bytes<br clear="none">7045120<br clear="none"><br clear="none"><a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-attach -n nv -- dd if=/dev/zero of=/run/shm/testfile<br clear="none">dd: writing to ‘/run/shm/testfile’: Cannot allocate memory<br clear="none">2080265+0 records in<br clear="none">2080264+0 records out<br clear="none">1065095168 bytes (1.1 GB) copied, 2.96393 s, 359 MB/s<br clear="none"><br clear="none"><a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-cgroup -n nv memory.usage_in_bytes<br clear="none">lxc_container: lxc_cgroup.c: main: 113 failed to retrieve value of<br clear="none">'memory.usage_in_bytes' for '/var/lib/lxc:nv'<br clear="none"><br clear="none"><a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-ls -f<br clear="none">Killed<br clear="none"><br clear="none"><br clear="none">See how the testfile on the nested container is also limited at around<br clear="none">1GB? Note that at this time the container "v" (the parent) can't do<br clear="none">anything (it can't even run "lxc-ls"), because it wants to allocate<br clear="none">more memory, and all available memory has been used by the child<br clear="none">container "nv".<br clear="none"><br clear="none">Since we can't run anything on the parent container due to memory<br clear="none">limit, kill the nested container<br clear="none"><br clear="none"><a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-stop -k -n nv<br clear="none">lxc-stop: commands.c: lxc_cmd_stop: 615 failed to stop 'nv': Operation<br clear="none">not permitted<br clear="none"><br clear="none"><a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-stop -k -n nv<br clear="none">nv is not running<br clear="none"><br clear="none"><a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-ls -f<br clear="none">NAME STATE IPV4 IPV6 GROUPS AUTOSTART<br clear="none">--------------------------------------------<br clear="none">nv STOPPED - - - NO<br clear="none"><br clear="none">So in short, the nested limit works. To make it operable, make sure<br clear="none">that the parent container ALWAYS have some free memory to work with.<br clear="none"><br clear="none">-- <br clear="none">Fajar<br clear="none"><br clear="none">On Fri, Mar 6, 2015 at 1:19 PM, Mohan G <<a shape="rect" ymailto="mailto:mohan_gg@yahoo.com" href="mailto:mohan_gg@yahoo.com">mohan_gg@yahoo.com</a>> wrote:<br clear="none">> Hi Folks,<br clear="none">> Let me explain my problem and then you can suggest me some way of over<br clear="none">> coming this.<br clear="none">> I wan to be able to run different protocol clients to use my file system<br clear="none">> mounted on the host. But i want to be able to limit their memory resource to<br clear="none">> 10Gb in total. Also, be able to set memory.sw limit so that each can have a<br clear="none">> softy limit of 5G. But if only one container is runing, it can use the<br clear="none">> entire 10G. I am able to use memory.use_hierarchy in cgroups to acheive the<br clear="none">> same. But how do i acheive this with container.<br clear="none">> Will nesting of containers help. my plan is to create a parent container and<br clear="none">> set limits to this and expect the nested conatiners to inherit these<br clear="none">> limit, and then i will set soft limits on these children container. But when<br clear="none">> i start nested containers, i don;t see any entrry under /sys/fs/groups for<br clear="none">> the children container.<br clear="none">><br clear="none">> Regards<br clear="none">> Mohan<br clear="none">><br clear="none">><br clear="none">> ________________________________<br clear="none">> From: Fajar A. Nugraha <<a shape="rect" ymailto="mailto:list@fajar.net" href="mailto:list@fajar.net">list@fajar.net</a>><br clear="none">> To: LXC users mailing-list <<a shape="rect" ymailto="mailto:lxc-users@lists.linuxcontainers.org" href="mailto:lxc-users@lists.linuxcontainers.org">lxc-users@lists.linuxcontainers.org</a>><br clear="none">> Sent: Wednesday, March 4, 2015 2:44 PM<br clear="none">> Subject: Re: [lxc-users] nested containers<br clear="none">><br clear="none">> On Wed, Mar 4, 2015 at 12:15 PM, Mohan G <<a shape="rect" ymailto="mailto:mohan_gg@yahoo.com" href="mailto:mohan_gg@yahoo.com">mohan_gg@yahoo.com</a>> wrote:<br clear="none">><br clear="none">><br clear="none">><br clear="none">>> Hi,<br clear="none">>> Is there anyway we can have nested containers/cgroups. One parent<br clear="none">>> container<br clear="none">>> forming the basis for children containers. i.e subset of parent container.<br clear="none">><br clear="none">><br clear="none">> Yes.<br clear="none">><br clear="none">> On parent container config (in ubuntu), add this:<br clear="none">> lxc.aa_profile=lxc-container-default-with-nesting<br clear="none">><br clear="none">> And then on that container, you can create containers<br clear="none">><br clear="none">> utopic ~ # lxc-ls -f --running<br clear="none">> NAME STATE IPV4 IPV6 GROUPS AUTOSTART<br clear="none">> -----------------------------------------------------------------<br clear="none">> v RUNNING 10.0.3.1, 192.168.124.173 - - NO<br clear="none">><br clear="none">> utopic ~ # lxc-attach -n v<br clear="none">><br clear="none">> <a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~#<br clear="none">><br clear="none">> <a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# cat /proc/1/cgroup<br clear="none">> 12:name=systemd:/lxc/v<br clear="none">> 11:perf_event:/lxc/v<br clear="none">> 10:net_prio:/lxc/v<br clear="none">> 9:net_cls:/lxc/v<br clear="none">> 8:memory:/lxc/v<br clear="none">> 7:hugetlb:/lxc/v<br clear="none">> 6:freezer:/lxc/v<br clear="none">> 5:devices:/lxc/v<br clear="none">> 4:cpuset:/lxc/v<br clear="none">> 3:cpuacct:/lxc/v<br clear="none">> 2:cpu:/lxc/v<br clear="none">> 1:blkio:/lxc/v<br clear="none">><br clear="none">> <a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-create -t download -n nv -- -d ubuntu -r vivid -a amd64<br clear="none">> Using image from local cache<br clear="none">> Unpacking the rootfs<br clear="none">><br clear="none">> ---<br clear="none">> You just created an Ubuntu container (release=vivid, arch=amd64,<br clear="none">> variant=default)<br clear="none">><br clear="none">> To enable sshd, run: apt-get install openssh-server<br clear="none">><br clear="none">> For security reason, container images ship without user accounts<br clear="none">> and without a root password.<br clear="none">><br clear="none">> Use lxc-attach or chroot directly into the rootfs to set a root password<br clear="none">> or create user accounts.<br clear="none">><br clear="none">> <a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-start -n nv<br clear="none">><br clear="none">> <a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-ls -f --running<br clear="none">> NAME STATE IPV4 IPV6 GROUPS AUTOSTART<br clear="none">> --------------------------------------------------<br clear="none">> nv RUNNING 10.0.3.249 - - NO<br clear="none">><br clear="none">><br clear="none">><br clear="none">> Now run a process inside the nested container<br clear="none">><br clear="none">> <a shape="rect" ymailto="mailto:root@v" href="mailto:root@v">root@v</a>:~# lxc-attach -n nv -- cat /proc/1/cgroup<br clear="none">> 12:name=systemd:/lxc/v/lxc/nv<br clear="none">> 11:perf_event:/lxc/v/lxc/nv<br clear="none">> 10:net_prio:/lxc/v/lxc/nv<br clear="none">> 9:net_cls:/lxc/v/lxc/nv<br clear="none">> 8:memory:/lxc/v/lxc/nv<br clear="none">> 7:hugetlb:/lxc/v/lxc/nv<br clear="none">> 6:freezer:/lxc/v/lxc/nv<br clear="none">> 5:devices:/lxc/v/lxc/nv<br clear="none">> 4:cpuset:/lxc/v/lxc/nv<br clear="none">> 3:cpuacct:/lxc/v/lxc/nv<br clear="none">> 2:cpu:/lxc/v/lxc/nv<br clear="none">> 1:blkio:/lxc/v/lxc/nv<br clear="none">><br clear="none">> Note how the cgroup is nested<br clear="none">><br clear="none">> --<br clear="none">> Fajar<br clear="none">> _______________________________________________<br clear="none">> lxc-users mailing list<br clear="none">> <a shape="rect" ymailto="mailto:lxc-users@lists.linuxcontainers.org" href="mailto:lxc-users@lists.linuxcontainers.org">lxc-users@lists.linuxcontainers.org</a><br clear="none">> <a shape="rect" href="http://lists.linuxcontainers.org/listinfo/lxc-users" target="_blank">http://lists.linuxcontainers.org/listinfo/lxc-users</a><br clear="none">><br clear="none">><br clear="none">> _______________________________________________<br clear="none">> lxc-users mailing list<br clear="none">> <a shape="rect" ymailto="mailto:lxc-users@lists.linuxcontainers.org" href="mailto:lxc-users@lists.linuxcontainers.org">lxc-users@lists.linuxcontainers.org</a><br clear="none">> <a shape="rect" href="http://lists.linuxcontainers.org/listinfo/lxc-users" target="_blank">http://lists.linuxcontainers.org/listinfo/lxc-users</a><div class="qtdSeparateBR"><br><br></div><div class="yqt2933482887" id="yqtfd01835"><br clear="none">_______________________________________________<br clear="none">lxc-users mailing list<br clear="none"><a shape="rect" ymailto="mailto:lxc-users@lists.linuxcontainers.org" href="mailto:lxc-users@lists.linuxcontainers.org">lxc-users@lists.linuxcontainers.org</a><br clear="none"><a shape="rect" href="http://lists.linuxcontainers.org/listinfo/lxc-users" target="_blank">http://lists.linuxcontainers.org/listinfo/lxc-users</a></div><br><br></div> </div> </div> </div></body></html>