[lxc-devel] [lxd/master] Add txqueuelen tweak.

cryptofuture on Github lxc-bot at linuxcontainers.org
Mon Sep 5 08:57:25 UTC 2016


A non-text attachment was scrubbed...
Name: not available
Type: text/x-mailbox
Size: 338 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20160905/1af26b2d/attachment.bin>
-------------- next part --------------
From 3b92631e81b8b76cd9001d812bf4e7305d5dfb2e Mon Sep 17 00:00:00 2001
From: cryptofuture <cryptofuture at users.noreply.github.com>
Date: Mon, 5 Sep 2016 08:50:44 +0000
Subject: [PATCH] Add txqueuelen tweak.

---
 doc/production-setup.md | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/doc/production-setup.md b/doc/production-setup.md
index b68b2bd..eb71a2a 100644
--- a/doc/production-setup.md
+++ b/doc/production-setup.md
@@ -44,3 +44,43 @@ Then, reboot the server.
 
 
 [1]: http://man7.org/linux/man-pages/man7/inotify.7.html
+
+## Network Bandwidth Tweaking 
+If you have at least 1GbE NIC on your lxd host with a lot of local activity (container - container connections, or host - container connections), or you have 1GbE or better internet connection on your lxd host it worth play with txqueuelen. These settings work even better with 10GbE NIC.
+
+### Server Changes
+
+#### txqueuelen 
+
+You need to change `txqueuelen` of your real NIC to 10000 (not sure about the best possible value for you), and change and change lxdbr0 interface `txqueuelen` to 10000.  
+In Debian-based distros you can change `txqueuelen` permanently in `/etc/network/interfaces`  
+You can add for ex.: `up ip link set eth0 txqueuelen 10000` to your interface configuration to set txqueuelen value on boot.  
+For permanent lxdbr0 txqueuelen value change I prefer edit `/usr/lib/lxd/lxd-bridge`. You can add `ifconfig lxdbr0 txqueuelen 10000` in start section, just after iptables rules. For ex.:
+```bash
+iptables "${use_iptables_lock}" -I FORWARD -o "${LXD_BRIDGE}" -j ACCEPT -m comment --comment "managed by lxd-bridge"
+iptables "${use_iptables_lock}" -t mangle -A POSTROUTING -o "${LXD_BRIDGE}" -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill -m comment --comment "managed by lxd-bridge"
+ifconfig lxdbr0 txqueuelen 10000
+```
+If you use lxd master in production or find this inappropriate you can set in `rc.local` or in another way you like.
+You could set it txqueuelen temporary (for test purpose) with `ifconfig interfacename# txqueuelen 10000`
+
+#### /etc/sysctl.conf
+
+You also need to increase `net.core.netdev_max_backlog` value.  
+You can add `net.core.netdev_max_backlog = 182757` to `/etc/sysctl.conf` to set it permanently (after reboot)
+You set `netdev_max_backlog` temporary (for test purpose) with `echo 182757 > /proc/sys/net/core/netdev_max_backlog`
+Note: You can find this value too high, most people prefer set `netdev_max_backlog` = `net.ipv4.tcp_mem` min. value.
+For example I use this values `net.ipv4.tcp_mem = 182757 243679 365514`
+
+### Containers changes
+
+You also need to change txqueuelen value for all you ethernet interfaces in containers.  
+In Debian-based distros you can change txqueuelen permanently in `/etc/network/interfaces`  
+You can add for ex.: `up ip link set eth0 txqueuelen 10000` to your interface configuration to set txqueuelen value on boot.
+
+### Notes regarding this change
+
+10000 txqueuelen value commonly used with 10GbE NICs. Basically small txqueuelen values used with slow devices with a high latency, and higher with devices with low latency. I personally have like 3-5% improvement with these settings for local (host with container, container vs container) and internet connections. Good thing about txqueuelen value tweak, the more containers you use, the more you can be can benefit from this tweak. And you can always temporary set this values and check this tweak in your environment without lxd host reboot.
+
+
+


More information about the lxc-devel mailing list