[Lxc-users] Many containers and too many open files

[Andre Nathan]

On Wed, 2011-03-02 at 14:24 +0100, Daniel Lezcano wrote:
> > I could paste my configuration files if you think it'd help you
> > reproducing the issue.
> 
> Yes, please :)

Ok. The test host has a br0 interface which is not attached to any
physical interface:

  auto br0
  iface br0 inet static
    address 192.168.0.1
    netmask 255.255.0.0
    broadcast 192.168.255.255
    bridge_stp off
    bridge_maxwait 5
    pre-up /usr/sbin/brctl addbr br0
    post-up /usr/sbin/brctl setfd br0 0
    post-down /usr/sbin/brctl delbr br0

I use NAT for container access, translating to the host's eth0 address.
There is also a MARK rule that I use for bandwidth limiting. These
commands are run on the host startup:

iptables -t mangle -A PREROUTING -i br0 -j MARK --set-mark 2
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source $ETH0_IP
iptables -P FORWARD DROP
iptables -A FORWARD -i br0 -o eth0 -j ACCEPT
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
tc qdisc add dev eth0 root handle 1: htb

I'm using a custom container creation script based on the ubuntu
templace that you can find here: 

  http://andre.people.digirati.com.br/lxc-create.sh

It sets up the bandwidth limit for each container and populates the
container's rootfs (there is a usage message :). It creates
configuration files like this:

  lxc.utsname = c2

  lxc.network.type = veth
  lxc.network.link = br0
  lxc.network.flags = up
  lxc.network.ipv4 = 192.168.0.2/16 192.168.255.255
  lxc.network.name = eth0
  lxc.network.veth.pair = veth0.2

  lxc.tty = 4
  lxc.pts = 1024
  lxc.rootfs = /var/lib/lxc/c2/rootfs
  lxc.mount  = /var/lib/lxc/c2/fstab

  lxc.cgroup.devices.deny = a
  # /dev/null and zero
  lxc.cgroup.devices.allow = c 1:3 rwm
  lxc.cgroup.devices.allow = c 1:5 rwm
  # consoles
  lxc.cgroup.devices.allow = c 5:1 rwm
  lxc.cgroup.devices.allow = c 5:0 rwm
  #lxc.cgroup.devices.allow = c 4:0 rwm
  #lxc.cgroup.devices.allow = c 4:1 rwm
  # /dev/{,u}random
  lxc.cgroup.devices.allow = c 1:9 rwm
  lxc.cgroup.devices.allow = c 1:8 rwm
  lxc.cgroup.devices.allow = c 136:* rwm
  lxc.cgroup.devices.allow = c 5:2 rwm
  # rtc
  lxc.cgroup.devices.allow = c 254:0 rwm

  # capabilities
  lxc.cap.drop = audit_control audit_write fsetid kill ipc_lock
ipc_owner lease linux_immutable mac_admin mac_override net_bind_service
mknod setfcap setpcap sys_admin sys_boot sys_module sys_nice sys_pacct
sys_ptrace sys_rawio sys_resource sys_time sys_tty_config

and fstab like this:

  /bin /var/lib/lxc/c2/rootfs/bin ext4 bind,ro 0 0
  /lib /var/lib/lxc/c2/rootfs/lib ext4 bind,ro 0 0
  /lib64 /var/lib/lxc/c2/rootfs/lib64 ext4 bind,ro 0 0
  /sbin /var/lib/lxc/c2/rootfs/sbin ext4 bind,ro 0 0
  /usr /var/lib/lxc/c2/rootfs/usr ext4 bind,ro 0 0
  /etc/environment /var/lib/lxc/c2/rootfs/etc/environment none bind,ro 0
0
  /etc/resolv.conf /var/lib/lxc/c2/rootfs/etc/resolv.conf none bind,ro 0
0
  /etc/localtime /var/lib/lxc/c2/rootfs/etc/localtime none bind,ro 0 0
  /etc/network/if-down.d /var/lib/lxc/c2/rootfs/etc/network/if-down.d
none bind,ro 0 0
  /etc/network/if-post-down.d /var/lib/lxc/c2/rootfs/etc/network/if-post-down.d none bind,ro 0 0
  /etc/network/if-pre-up.d /var/lib/lxc/c2/rootfs/etc/network/if-pre-up.d none bind,ro 0 0
  /etc/network/if-up.d /var/lib/lxc/c2/rootfs/etc/network/if-up.d none
bind,ro 0 0
  /etc/login.defs /var/lib/lxc/c2/rootfs/etc/login.defs none bind,ro 0 0
  /etc/securetty /var/lib/lxc/c2/rootfs/etc/securetty none bind,ro 0 0
  /etc/pam.conf /var/lib/lxc/c2/rootfs/etc/pam.conf none bind,ro 0 0
  /etc/pam.d /var/lib/lxc/c2/rootfs/etc/pam.d none bind,ro 0 0 
  /etc/security /var/lib/lxc/c2/rootfs/etc/security none bind,ro 0 0
  /etc/alternatives /var/lib/lxc/c2/rootfs/etc/alternatives none bind,ro
0 0
  proc /var/lib/lxc/c2/rootfs/proc proc ro,nodev,noexec,nosuid 0 0
  devpts /var/lib/lxc/c2/rootfs/dev/pts devpts defaults 0 0
  sysfs /var/lib/lxc/c2/rootfs/sys sysfs defaults 0 0


I think that's all. If you need any more info feel free to ask :)

Thanks
Andre