[Lxc-users] Packet forwarding performance drop with 1000 containers

[Benoit Lourdelet]

Hello,

Working with 1000 containers I had already modified gc_thresh* to fit my
needs.

By mistake I had set gc_interval to a too high value (past 2^32) , forcing
linux to set gc_interval to the default value (30) with is not suitable in
my case.

Setting gc_interval to 3600000 solved my problem.

Thanks for pointing me in the right direction

Benoit

On 24/04/2013 07:21, "Guido Jäkel" <G.Jaekel at DNB.DE> wrote:

>Dear Benoit,
>
>there's a lot of local matching and translation between layer2 and layer3
>in your case. I wounder if it is related to the apr cache size and
>garbage parameters. I found [http://linux.die.net/man/7/arp]:
>
>  gc_interval (since Linux 2.2)
>     How frequently the garbage collector for neighbor entries should
>attempt to run. Defaults to 30 seconds.
>  gc_stale_time (since Linux 2.2)
>     Determines how often to check for stale neighbor entries. When a
>neighbor entry is considered stale, it is resolved again before sending
>data to it. Defaults to 60 seconds.
>  gc_thresh1 (since Linux 2.2)
>     The minimum number of entries to keep in the ARP cache. The garbage
>collector will not run if there are fewer than this number of entries in
>the cache. Defaults to 128.
>  gc_thresh2 (since Linux 2.2)
>     The soft maximum number of entries to keep in the ARP cache. The
>garbage collector will allow the number of entries to exceed this for 5
>seconds before collection will be performed. Defaults to 512.
>  gc_thresh3 (since Linux 2.2)
>     The hard maximum number of entries to keep in the ARP cache. The
>garbage collector will always run if there are more than this number of
>entries in the cache. Defaults to 1024.
>
>This still seems to be the default on recent kernels, on a box with 3.3.5
>I found
>
>   root at bladerunner9 ~ # cat /proc/sys/net/ipv4/neigh/default/gc*
>   30
>   60
>   128
>   512
>   1024
>
>If the ARP cache get's exhausted, there must be continous additional ARP
>resolution traffic and latency. May you check this theory?
>
>
>Greetings
>
>Guido
>
>
>On 2013-04-23 23:34, Benoit Lourdelet wrote:
>> Hello,
>> 
>> Forwarding throughput is decreasing gradually as I add containers. I
>>don't
>> see any sudden drop.
>> 
>> I we consider aggregated forwarding performance with 100 containers to
>>be
>> 1, here are the measurements for
>> 
>> # containers 	Aggregated throughput
>> ------------------------------------
>> 100 			1
>> 500 			.71
>> 1000 			.27
>> 1100 			.23
>