[Lxc-users] Packet forwarding performance drop with 1000 containers

[Guido Jäkel]

Dear Benoit,

there's a lot of local matching and translation between layer2 and layer3 in your case. I wounder if it is related to the apr cache size and garbage parameters. I found [http://linux.die.net/man/7/arp]:

  gc_interval (since Linux 2.2)
     How frequently the garbage collector for neighbor entries should attempt to run. Defaults to 30 seconds. 
  gc_stale_time (since Linux 2.2)
     Determines how often to check for stale neighbor entries. When a neighbor entry is considered stale, it is resolved again before sending data to it. Defaults to 60 seconds. 
  gc_thresh1 (since Linux 2.2)
     The minimum number of entries to keep in the ARP cache. The garbage collector will not run if there are fewer than this number of entries in the cache. Defaults to 128. 
  gc_thresh2 (since Linux 2.2)
     The soft maximum number of entries to keep in the ARP cache. The garbage collector will allow the number of entries to exceed this for 5 seconds before collection will be performed. Defaults to 512. 
  gc_thresh3 (since Linux 2.2)
     The hard maximum number of entries to keep in the ARP cache. The garbage collector will always run if there are more than this number of entries in the cache. Defaults to 1024. 

This still seems to be the default on recent kernels, on a box with 3.3.5 I found

   root at bladerunner9 ~ # cat /proc/sys/net/ipv4/neigh/default/gc*
   30
   60
   128
   512
   1024

If the ARP cache get's exhausted, there must be continous additional ARP resolution traffic and latency. May you check this theory?


Greetings

Guido


On 2013-04-23 23:34, Benoit Lourdelet wrote:
> Hello,
> 
> Forwarding throughput is decreasing gradually as I add containers. I don't
> see any sudden drop.
> 
> I we consider aggregated forwarding performance with 100 containers to be
> 1, here are the measurements for
> 
> # containers 	Aggregated throughput
> ------------------------------------
> 100 			1
> 500 			.71
> 1000 			.27
> 1100 			.23