[Lxc-users] 转发: lxc scalability problem

Thu Jun 13 01:07:45 UTC 2013

-----邮件原件-----
发件人: yaozhicheng [mailto:yaozhicheng at emails.bjut.edu.cn] 
发送时间: 2013年6月8日 23:32
收件人: 'Serge Hallyn'
主题: 答复: [Lxc-users] lxc scalability problem

Dear serge,
Sorry for my delayed reply :)
Problem Still persists with selinux disabled. Maybe it is not the memory
problem because that my machine has 2TB memory and 8 x E7-8830 cpus.
The OS in the containers is debian6.0. I have run filebench and lmbench in 8
containers simultaneously, they performed very well.
100 more containers can be started when physical eth2 removed out from br0.
May be the veth network problem?

This is part of the vmcore-dmesg.txt(selinux disabled):

<6>br0: port 5(vethOaLJ1U) entering forwarding state
<6>br0: port 4(vethZZ0xuP) entering forwarding state
<6>br0: port 8(vethjAE1Pb) entering forwarding state
<6>br0: port 9(veth0SqVmc) entering forwarding state
<6>br0: port 16(vethtVuHLb) entering forwarding state
<6>br0: port 7(vethtTiLzY) entering forwarding state
<6>br0: port 15(vethxWoVMd) entering forwarding state
<6>br0: port 12(vethvSfzsf) entering forwarding state
<6>br0: port 13(vethw78ho7) entering forwarding state
<6>br0: port 11(vethrf5HMe) entering forwarding state
<6>br0: port 6(veth4Vm0m0) entering forwarding state
<6>br0: port 10(vethho7Oae) entering forwarding state
<6>br0: port 20(vethIsYsGj) entering forwarding state
<6>br0: port 14(vethXaQOHa) entering forwarding state
<6>br0: port 17(veth8rD1je) entering forwarding state <4>------------[ cut
here ]------------ <2>kernel BUG at mm/slab.c:3069!
<4>invalid opcode: 0000 [#1] SMP
<4>last sysfs file: /sys/devices/virtual/net/vetha5mnvF/flags
<4>CPU 70
<4>Modules linked in: veth autofs4 sunrpc cpufreq_ondemand acpi_cpufreq
freq_table mperf bridge stp llc ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6
xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode mlx4_core
igb dca ptp pps_core i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support sg
i7core_edac edac_core ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif
pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log
dm_mod [last unloaded: scsi_wait_scan] <4>
<4>Pid: 0, comm: swapper Not tainted 2.6.32-358.6.2.el6.x86_64 #1 Supermicro
X8OBN/X8OBN
<4>RIP: 0010:[<ffffffff81167354>]  [<ffffffff81167354>]
cache_alloc_refill+0x1e4/0x240
<4>RSP: 0018:ffff8800282c3ad0  EFLAGS: 00010096
<4>RAX: 000000000000003c RBX: ffff8940274f0140 RCX: 00000000ffffffcc
<4>RDX: 000000000000003c RSI: 0000000000000000 RDI: ffff884026fa7800
<4>RBP: ffff8800282c3b30 R08: 0000000000000000 R09: 0000000000000000
<4>R10: 0000000000000000 R11: 0000000000000000 R12: ffff884026fa7800
<4>R13: ffff884026fa0440 R14: 000000000000003c R15: ffff88402455c000
<4>FS:  0000000000000000(0000) GS:ffff8800282c0000(0000)
knlGS:0000000000000000
<4>CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 00007f5599065000 CR3: 0000000001a85000 CR4: 00000000000007e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process swapper (pid: 0, threadinfo ffff88c027c06000, task
ffff8880283d1500)
<4>Stack:
<4> ffff8800282c3b10 000000048144caec ffff884026fa0480 000412201096c080
<4><d> ffff884026fa0460 ffff884026fa0450 ffff8940259fa6e8 0000000000000000
<4><d> 0000000000000020 ffff8940274f0140 0000000000000020 0000000000000246
<4>Call Trace:
<4> <IRQ>
<4> [<ffffffff8116840f>] kmem_cache_alloc+0x15f/0x190 <4>
[<ffffffffa02726e0>] ? __br_forward+0x0/0xd0 [bridge] <4>
[<ffffffff8143f15f>] skb_clone+0x6f/0xb0 <4> [<ffffffffa02726e0>] ?
__br_forward+0x0/0xd0 [bridge] <4> [<ffffffffa0272320>]
deliver_clone+0x30/0x60 [bridge] <4> [<ffffffffa0272549>] br_flood+0x79/0xd0
[bridge] <4> [<ffffffffa02725b5>] br_flood_forward+0x15/0x20 [bridge] <4>
[<ffffffffa02736ee>] br_handle_frame_finish+0x27e/0x2a0 [bridge] <4>
[<ffffffffa02738ba>] br_handle_frame+0x1aa/0x250 [bridge] <4>
[<ffffffff81448599>] __netif_receive_skb+0x529/0x750 <4>
[<ffffffff8143da41>] ? __alloc_skb+0x81/0x190 <4> [<ffffffff8144a8f8>]
netif_receive_skb+0x58/0x60 <4> [<ffffffff8143da41>] ?
__alloc_skb+0x81/0x190 <4> [<ffffffff8144a8f8>] netif_receive_skb+0x58/0x60
<4> [<ffffffff8144aa00>] napi_skb_finish+0x50/0x70 <4> [<ffffffff8144cfa9>]
napi_gro_receive+0x39/0x50 <4> [<ffffffffa015045c>] igb_poll+0x7ec/0xc70
[igb] <4> [<ffffffff81033ef7>] ? native_apic_msr_write+0x37/0x40 <4>
[<ffffffff8144851b>] ? __netif_receive_skb+0x4ab/0x750 <4>
[<ffffffff810a7b05>] ? tick_dev_program_event+0x65/0xc0 <4>
[<ffffffff81012bb9>] ? read_tsc+0x9/0x20 <4> [<ffffffff8144d0c3>]
net_rx_action+0x103/0x2f0 <4> [<ffffffff81076fb1>] __do_softirq+0xc1/0x1e0
<4> [<ffffffff810e1720>] ? handle_IRQ_event+0x60/0x170 <4>
[<ffffffff8100c1cc>] call_softirq+0x1c/0x30 <4> [<ffffffff8100de05>]
do_softirq+0x65/0xa0 <4> [<ffffffff81076d95>] irq_exit+0x85/0x90 <4>
[<ffffffff81517145>] do_IRQ+0x75/0xf0 <4> [<ffffffff8100b9d3>]
ret_from_intr+0x0/0x11 <4> <EOI> <4> [<ffffffff812d39ae>] ?
intel_idle+0xde/0x170 <4> [<ffffffff812d3991>] ? intel_idle+0xc1/0x170 <4>
[<ffffffff81415277>] cpuidle_idle_call+0xa7/0x140 <4> [<ffffffff81009fc6>]
cpu_idle+0xb6/0x110 <4> [<ffffffff81506fcc>] start_secondary+0x2ac/0x2ef
<4>Code: 89 ff e8 70 1c 12 00 eb 99 66 0f 1f 44 00 00 41 c7 45 60 01 00 00
00 4d 8b 7d 20 4c 39 7d c0 0f 85 f2 fe ff ff eb 84 0f 0b eb fe <0f> 0b 66 2e
0f 1f 84 00 00 00 00 00 eb f4 8b 55 ac 8b 75 bc 31 <1>RIP
[<ffffffff81167354>] cache_alloc_refill+0x1e4/0x240 <4> RSP
<ffff8800282c3ad0>

Sorry for my poor English :)

Sincere
-yao

-----邮件原件-----
发件人: Serge Hallyn [mailto:serge.hallyn at ubuntu.com]
发送时间: 2013年6月7日 4:16
收件人: yaozhicheng
抄送: lxc-users at lists.sourceforge.net
主题: Re: [Lxc-users] lxc scalability problem

Quoting yaozhicheng (yaozhicheng at emails.bjut.edu.cn):
> Hi all
> 
> My OS get crashed when I start more then 20 containers. 

Seems like an selinux labeling related problem.  If you boot without selinux
enabled do you still have this problem?  (I'm not suggesting that as a
workaround - only to verify that the problem is with selinux using too much
memory for the dev tmpfs labeling, so ppl know where to look)

-serge