[lxc-devel] [lxd/master] "neighbour: ndisc_cache: neighbor table overflow"

[simos on Github]

A non-text attachment was scrubbed...
Name: not available
Type: text/x-mailbox
Size: 540 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20180717/0403a334/attachment.bin>
-------------- next part --------------
From fd6bf81d98f65625c8406bd816ae88baa46b9455 Mon Sep 17 00:00:00 2001
From: Simos Xenitellis <simos at users.noreply.github.com>
Date: Tue, 17 Jul 2018 18:24:26 +0300
Subject: [PATCH] "neighbour: ndisc_cache: neighbor table overflow"

The default ARP table size in the kernel has a limit of 1024 entries.
If you launch more than a thousand containers, you will fill up the time
and no new entries get accepted because each container takes up an entry in the ARP table.
---
 doc/production-setup.md | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/doc/production-setup.md b/doc/production-setup.md
index cfd75ce06..2a15e4fd3 100644
--- a/doc/production-setup.md
+++ b/doc/production-setup.md
@@ -16,6 +16,7 @@ using containers that require tens of thousands of file operations.
 
 `failed to open stream: Too many open files in...`
 
+`neighbour: ndisc_cache: neighbor table overflow!`
 
 # Server Changes
 ## /etc/security/limits.conf
@@ -39,12 +40,14 @@ fs.inotify.max\_user\_instances | 1048576   | 128     | This specifies an upper
 fs.inotify.max\_user\_watches   | 1048576   | 8192    | This specifies an upper limit on the number of watches that can be created per real user ID. [1]
 vm.max\_map\_count              | 262144    | 65530   | This file contains the maximum number of memory map areas a process may have. Memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries.
 kernel.dmesg\_restrict          | 1         | 0       | This denies container access to the messages in the kernel ring buffer. Please note that this also will deny access to non-root users on the host system.
-
+net.ipv4.neigh.default.gc_thresh3| 8192     | 1024    | This is the maximum number of entries in ARP table (IPv4). You should increase this if you create over 1024 containers. Otherwise, you will get the error `neighbour: ndisc_cache: neighbor table overflow!` when the ARP table gets full and those containers will not be able to get a network configuration. [2]
+net.ipv6.neigh.default.gc_thresh3| 8192     | 1024    | This is the maximum number of entries in ARP table (IPv6). You should increase this if you plan to create over 1024 containers. Otherwise, you will get the error `neighbour: ndisc_cache: neighbor table overflow!` when the ARP table gets full and those containers will not be able to get a network configuration. [2]
 
 Then, reboot the server.
 
 
 [1]: http://man7.org/linux/man-pages/man7/inotify.7.html
+[2]: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
 
 ## Network Bandwidth Tweaking 
 If you have at least 1GbE NIC on your lxd host with a lot of local activity (container - container connections, or host - container connections), or you have 1GbE or better internet connection on your lxd host it worth play with txqueuelen. These settings work even better with 10GbE NIC.